Navigation road map to perceptual lane map matching

Abstract

The invention discloses navigational road map to perceived lane map matching, and particularly discloses, in various examples, embodiments relate to navigational road and perceived lane matching for autonomous and semi-autonomous systems and applications. In this regard, geometric similarities are used to perform lane-road matching to generate efficient lane-road mappings for lane planning and making decisions, among others. In some embodiments, road data representing at least one road section and lane data representing a lane associated with a location of an own machine are received. Thereafter, it is determined that the one or more consecutive road sections match the lane based at least on a geometric similarity between the lane and the one or more consecutive road sections. Based at least on determining that the one or more consecutive road segments match the lane, a representation of the lane mapped to the one or more consecutive road segments is generated.

Description

BACKGROUND OF THE INVENTION

[0001] Designing a system for autonomously driving a vehicle safely without supervision or semi - autonomously driving a vehicle safely with limited supervision is extremely difficult. For example, some designs attempt to enable autonomous vehicles or other self - machines to function like a focused driver - which utilizes a perception and action system that has an amazing ability to identify and respond to dynamic and static obstacles in complex environments - to avoid other objects or structures along the vehicle's path. When designing such a system, a lane map can be used, for example, to facilitate localization (e.g., localizing the self - machine in a map), path planning (e.g., determining candidate routes for the self - machine), object detection and tracking (e.g., for performing object - in - path analysis (OIPA), etc.) and / or making decisions (e.g., determining the best route based on candidate paths, vehicle state, and the environment). For example, a lane map can be used for lane planning to make decisions regarding lane changes, turns, and lane keeping. In the context of autonomous vehicles, lane planning is valuable for many reasons, including following a planned route and turning in an organized manner, performing efficient maneuvers, and facilitating lane - keeping assistance. Thus, an accurate lane map is valuable for effectively facilitating localization, path planning, and / or making decisions.

[0002] However, in some cases, the lane map may not have enough data reflected in the lane map, which affects the effectiveness of the lane map (e.g., for performing lane planning). For example, the lane map may have inaccurate or incomplete lane widths, markings, or other attributes, which may result in sub - optimal lane planning and navigation decisions. Thus, using such a lane map for planning and decision - making can lead to inaccurate information or inefficiencies, which limits the ability of an autonomous vehicle to navigate effectively in an environment. SUMMARY OF THE INVENTION

[0003] Embodiments of the present disclosure relate to navigating roads and sensing lane matching for autonomous and semi - autonomous systems and applications. Systems and methods are disclosed for using geometric similarity to identify or detect instances of lanes that match a road or a portion thereof for use by autonomous vehicles, semi - autonomous vehicles, robots, and / or other object or machine types in localization, navigation, and / or other uses.

[0004] Compared to conventional systems such as those described above, in some embodiments, lane-road matching is performed using geometric similarity to generate valid lane-road mappings that can be used for lane planning and decision-making, as well as other uses in autonomous or semi-autonomous systems and applications. For example, using the techniques provided herein, lane-road mappings can be generated for use in generating valid lane-level instructions and for use by a behavior planning module to perform efficient and effective navigation. Additionally, lanes can be annotated with road-level attributes from a road map, including road curvature values, traffic control signs, and signals, resulting in more detailed and effective information.

[0005] At a high level, the embodiments described herein are directed to matching lanes to roads or portions thereof and generating lane-road mappings representative of such matches. In an embodiment, lanes and roads or portions thereof can be identified as a match based on geometric similarity analysis. Specifically, a lane can be identified as matching a road or portion thereof when certain matching conditions of specified geometric similarity are met. For example, matching conditions indicating that a lane matches a road or portion thereof can include that the lane is within a distance threshold and / or a direction threshold of the road. When it is determined that the geometric similarity between a lane and a road or portion thereof indicates a match, a match score can be generated to indicate the degree of such geometric similarity. Since a lane can be determined to match multiple roads (e.g., based on meeting the matching conditions), the match score can be used to select a particular road to map to that lane. In this way, a particular lane associated with the maximum match score that matches a road or portion thereof can be selected for the lane-road mapping. Thus, a lane-road mapping generally maps one or more particular lane sections to one or more road sections that maximize geometric similarity (e.g., based on one or more match scores). A lane-road mapping can be represented in any number of ways. In some cases, a lane-road mapping can be represented using lane section identifiers and road section identifiers corresponding to the locations where the lanes were determined to match the roads.

[0006] In operation, to identify whether a lane matches a road or a portion thereof, various points along the lane and the road can be analyzed to determine whether a set of matching conditions are met, indicating geometric similarity between the lane and the road. An example of a matching condition includes the distance between the lane and the road being within a distance threshold. For example, when the distance between the lane and the road is within a predetermined distance, the lane can be considered to match the road. In some cases, to identify whether the distance condition is met, the vertical spans extending from the lane and / or the road (e.g., via a lane map and / or a road map) can be analyzed. For example, a vertical line can be extended from the road being analyzed until it intersects the lane being analyzed. The distance from the road to the intersection can be determined and compared with the distance threshold. In the case where the distance from the road to the intersection is equal to or less than the distance threshold, the distance condition can be considered to be met at that particular location. As another example, a vertical span having the length of the distance threshold can be extended from the road. In the case where there is an intersection between the vertical span and the lane, the distance condition can be considered to be met at that particular location.

[0007] Another example of a matching condition includes the lane being within a direction threshold of the road. For example, when the lane is within a predetermined direction relative to the road, the lane can be considered to match the road. In some cases, to identify whether the direction condition is met, the dot product function can be used to determine the similarity between the direction of the lane at a particular location and the direction of the road at a particular location, as described herein.

[0008] Thus, the techniques described herein can be used to match lanes with roads or portions thereof and generate a lane-road mapping representing the lane-road match. The generated lane-road mapping can be provided to an autonomous or semi-autonomous vehicle driving stack to assist in performing one or more operations related to vehicle positioning, safety planning, and / or control. Thus, the lane-road mapping can assist autonomous or semi-autonomous vehicles or other machine types (e.g., robots, construction vehicles, drones, etc.) in navigating in a physical environment. Specifically, it can assist in lane reasoning and planning for more accurate and reliable navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The present systems and methods for navigation-road and perception-lane matching for autonomous and semi-autonomous systems and applications are described in detail below with reference to the drawings, in which:

[0010] Figure 1 is a data flow diagram showing an example process of a lane-road matching system according to some embodiments of the present disclosure;

[0011] Figure 2 is a data flow diagram showing an example process for facilitating lane and road matching according to some embodiments of the present disclosure;

[0012] Figure 3 is an illustration of an example of a coverage frustum relative to a vehicle position in accordance with some embodiments of the present disclosure;

[0013] Figures 4A - 4D provides an illustration including the effect of prologue intersection modification in accordance with some embodiments of the present disclosure;

[0014] Figure 5 provides an example of generating a matching score in accordance with some embodiments of the present disclosure;

[0015] Figures 6 - 8 provides an example method for generating a lane - road map in accordance with some embodiments of the present disclosure;

[0016] Figure 9A is an illustration of an example autonomous vehicle in accordance with some embodiments of the present disclosure;

[0017] Figure 9B is in accordance with some embodiments of the present disclosure Figure 9A example of camera positions and fields of view of an example autonomous vehicle;

[0018] Figure 9C is in accordance with some embodiments of the present disclosure Figure 9A block diagram of an example system architecture of an example autonomous vehicle;

[0019] Figure 9D is a system diagram of communication between a cloud - based server and Figure 9A an example autonomous vehicle in accordance with some embodiments of the present disclosure;

[0020] Figure 10 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0021] Figure 11 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] Systems and methods related to navigation - road and perception - lane matching for autonomous and semi - autonomous systems and applications are disclosed. For example, systems and methods are disclosed that use geometric similarity to identify or detect instances of lanes that match a road or a portion thereof for use by autonomous vehicles, semi - autonomous vehicles, robots, and / or other object or machine types in positioning, navigation, and / or other uses. In this regard, the present technology can be used to enhance the use of lane maps. Specifically, the present technology for matching lanes to a road or a portion thereof enables more efficient lane planning and decision - making. For example, using the techniques provided herein, effective lane - level instructions can be generated and used by a behavior planning module to perform efficient and effective navigation. Additionally, lanes can be annotated with road - level attributes from a road map, including road curvature values, traffic control signs, and signals, resulting in more detailed and effective information.

[0023] Although the present disclosure may be described with respect to an exemplary autonomous or semi - autonomous vehicle or machine 900 (alternatively referred to herein as "vehicle 900", "ego - vehicle 900", "machine 900", or "ego - machine 900", examples of which are described with respect to Figures 9A - 9D ), this is not intended to be limiting. For example, the systems and methods described herein can be used by, but are not limited to: non - autonomous vehicles or machines, semi - autonomous vehicles or machines (e.g., in one or more advanced driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off - road vehicles, vehicles coupled to one or more trailers, airships, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, airplanes, engineering vehicles, trains, submarines, remotely controlled vehicles (e.g., drones), and / or other types of vehicles. Additionally, although the present disclosure may be described with respect to navigation - road and perception - lane matching in autonomous or semi - autonomous systems and applications, this is not intended to be restrictive, and the systems and methods described herein can be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi - autonomous machine applications, and / or any other technical space where lane detection, positioning, and / or navigation can be used.

[0024] At a high level, the embodiments described herein are directed to matching lanes to a road (or another type of delineated area, such as a path, corridor, or a portion of a sidewalk, a portion of an open space, etc.) or a portion thereof (matching to a navigable surface), and generating a lane-road mapping representing such a match. In an embodiment, lanes and a road or a portion thereof may be identified as a match based on geometric similarity analysis. Specifically, when certain matching conditions of a specified geometric similarity are met, a lane may be identified as matching a road or a portion thereof. For example, the matching conditions indicating that a lane matches a road or a portion thereof may include that the lane is within a distance threshold of the road and the lane is within a direction threshold of the road. When the geometric similarity between a lane and a road or a portion thereof indicates a match, a match score may be generated to indicate the degree of such geometric similarity. Since a lane may be determined to match multiple roads (e.g., according to meeting the matching conditions), the match score may be used to select a particular road to be mapped to that lane. In this way, a particular lane associated with the maximum match score that matches a road or a portion thereof may be selected for lane-road mapping. Thus, lane-road mapping generally maps a particular lane segment to a road segment that maximizes geometric similarity (e.g., based on the match score). The lane-road mapping may be represented in any number of ways. In some cases, the lane-road mapping may be represented using a lane segment identifier and a road segment identifier corresponding to the location where the lane is determined to match the road.

[0025] In operation, to identify whether a lane matches a road or a portion thereof, lanes and various points along the road may be analyzed to determine whether a set of matching conditions are met, thereby indicating geometric similarity between the lane and the road. An example of a matching condition includes that the distance between the lane and the road is within a distance threshold. For example, when the distance between the lane and the road is within a predetermined distance, the lane may be considered to match the road. In some cases, to identify whether the distance condition is met, a vertical span extending from the lane and / or the road may be analyzed (e.g., via a lane map and / or a road map). For example, a vertical line may be extended from the road being analyzed until it intersects the lane being analyzed. The distance from the road to the intersection point may be determined and compared to the distance threshold. In the case where the distance from the road to the intersection point is equal to or less than the distance threshold, the distance condition may be considered to be met at that particular location. As another example, a vertical span having the length of the distance threshold may be extended from the road. In the case where there is an intersection point between the vertical span and the lane, the distance condition may be considered to be met at that particular location.

[0026] Another example of a matching condition includes the lane being within a directional threshold of the road. For example, when the lane is within a predetermined direction relative to the road, the lane can be considered to match the road. In some cases, to identify whether the directional condition is met, a dot product function can be used to determine the similarity of the direction of the lane at a particular location to the direction of the road at the particular location, as described herein.

[0027] Lane-road matching analysis can be performed in association with respective points or positions along the lane and / or road being analyzed. As an example, the points being analyzed can be a particular distance apart (e.g., increments of five meters apart from each other). In an embodiment, lane-road matching can be performed in an iterative manner. In this regard, the analysis can proceed along the length of the lane and / or road to identify whether a set of matching conditions are met at a particular location. Thus, at each incremental location along the road and / or lane, a set of matching conditions can be analyzed in association with the lane-road pair being analyzed.

[0028] Since there can be various road segments, depending on the progress made in completing a road segment associated therewith, it can be determined whether there are any unvisited successor road segments of the road. Specifically, any unvisited successor road segments along the current traversal path of the road map can be identified. If there are successor road segments, then the progress can continue along the successor road segments. This iterative process enables the matching of any number of road segments.

[0029] In an embodiment, the iterative process of performing matching analysis on the lane and the road or portions thereof can continue until a termination event is identified. In this regard, the matching analysis can proceed along successive points of the road and / or lane until a termination event is identified. An example of a termination event is the identification of the end of the lane and / or road being analyzed. Another example of a termination event is the identification of a failed match. For example, assume that the matching analysis performed in association with a particular point along the road results in a failure to meet the matching conditions, such as a distance condition and / or a direction condition. In such a case, the failure to meet the matching conditions can be considered a termination event, for which the matching analysis of the current lane and road pair being analyzed can be terminated. In some cases, the matching does not terminate until a series of failures to meet the matching conditions (e.g., in association with successive points along the road) are identified.

[0030] In addition to identifying matches associated with lanes and roads or portions thereof, a match score indicating the degree of match can be determined. In an embodiment, the match score can be based on geometric similarity. Since area calculations can be computationally intensive and require real-time performance in a self-machine, an approximation of area-based geometric similarity can be used, which is based on the distance between the lane and the road at positions regularly sampled along the road length. For example, given a lane and a road, the geometric similarity between a given section of the lane and a given section of the road can correspond to the sum of the reciprocals of the distances between the lane and the road at the sampled positions. Thus, the more closely the lane section fits or is close to the road section, the higher the geometric similarity and thus the higher the quality of the corresponding match.

[0031] Based on the match score, a lane-road map can be generated and / or provided for a particular lane-road match having the maximum match score. Lane-road mapping generally maps one or more lanes to one or more particular roads or portions thereof that attempt to maximize geometric similarity (e.g., based on the match score). In an embodiment, the lane-road map is represented by one or more entries indicating the road sections of the match. In some cases, the lane-road mapping can be represented using lane section identifiers and road section identifiers corresponding to the positions where the lane-to-road match is determined.

[0032] Thus, the techniques described herein can be used to match lanes to roads or portions thereof and generate a lane-road mapping representing the lane-road match. The generated lane-road mapping can be provided to an autonomous or semi-autonomous vehicle drive stack to assist in performing one or more operations related to vehicle positioning, safety planning, and / or control. Thus, the lane-road mapping can assist an autonomous or semi-autonomous vehicle or machine in navigating in a physical environment and, in particular, can assist in lane reasoning and planning to achieve more accurate and reliable navigation. Different from conventional methods, various embodiments provide a method for implementing effective lane-level instructions that can be executed by a behavior planning module, thereby allowing more precise lane selection and navigation instructions than conventional methods. For example, embodiments enable the identification of lanes that follow the road that is farthest along the navigation road map. In addition, various embodiments can be used to enable the annotation of lanes with road-level attributes from the navigation road map, thereby enhancing the lane map and its use.

[0033] Referring to Figure 1 , Figure 1is a data flow diagram showing an example process 100 of a lane-road matching system according to some embodiments of the present disclosure. It should be understood that such arrangements and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, function groupings, etc.) may be used in addition to or in place of the shown arrangements and elements, and some elements may be entirely omitted. Moreover, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components, and may be implemented in any suitable combination and location. The various functions performed by the entities described herein may be executed by hardware, firmware, and / or software. For example, the various functions may be executed by a processor executing instructions stored in a memory. In some embodiments, the systems, methods, and processes described herein may use components, features, and / or functions similar to those of an example autonomous vehicle 900, Figures 9A - 9D an example computing device 1000, and / or Figure 10 an example data center 1100 that are similar to the components, features, and / or functions of Figure 11 to perform.

[0034] At a high level, process 100 uses a lane-road matching manager 118, which may be configured to match or align a lane or a portion thereof (also referred to as a lane segment) with a road or a portion thereof (also referred to as a road segment) and generate a lane-road map 120. In the embodiments described herein, a lane and a road or a portion thereof may be identified as matching based on a geometric similarity analysis. Specifically, a lane may be identified as matching a road or a portion thereof when certain matching conditions of specified geometric similarity are met. For example, the matching conditions indicating that a lane matches a road or a portion thereof may include that the lane is within a threshold distance of the road and the lane is within a threshold direction of the road. When it is determined that the geometric similarity between a lane and a road or a portion thereof indicates a match, a matching score may be generated to indicate the degree of such geometric similarity. Since a lane may be determined to match multiple roads (e.g., according to the attainment of the matching conditions), the matching score may be used to select a particular road to be mapped to the lane. In this way, a particular lane associated with the maximum matching score that matches a road or a portion thereof may be selected for the lane-road mapping. Thus, the lane-road mapping generally maps a particular lane segment to a road segment that maximizes the geometric similarity (e.g., based on the matching score) of the road segment. The lane-road mapping may be represented in any number of ways. In some cases, the lane-road mapping may be represented using lane segment identifiers and road segment identifiers corresponding to the locations where the lane and the road are determined to match.

[0035] In an embodiment, the lane-road matching manager 118 may perform lane-road matching using the road map data 108 and / or the lane map data 116. In this regard, the road map generator 106 may generate a road map, and the data associated therewith may be input into the lane-road matching manager 118 to perform lane-road matching. The road map or the navigation road map may be a directed graph representing a road network. The road map data (or road data), such as the road map data 108, may refer to any data for representing a road map (e.g., the road map generated by the road map generator 106). In some cases, the road map generator 106 is located in the vehicle. In this way, the road map generator 106 may generate a road map in real time based on the road network data 104 (e.g., streamed from the road network data source 102).

[0036] To generate the road map and / or the road map data, the road map generator 106 may obtain and use the road network data 104. The road network data 104 generally refers to any data representing a road network or a portion thereof. The road network data may include, for example, information about the road network infrastructure, such as the topology and connectivity of the road network, the locations of traffic signals, the speed limits of different road segments, etc.

[0037] The road network data 104 may be obtained in any manner. In some embodiments, the road network data 104 is obtained from the road network data source 102. In some cases, the road network data source 102 may be a navigation head unit (e.g., a third-party navigation head unit) that provides or streams road network data encoded in a specific format, such as according to the ADASISv2 specification. The ADASISv2 specification refers to a standard for encoding road network data in a format that can be used by the road map generator 106.

[0038] According to the obtained road network data 104, the road map generator 106 may use such data to generate a road map. As described above, the road map may be a directed graph representing a road network. In the road map, the edges in the graph may represent road segments linking a pair of junctions. In this regard, except for the linked junctions, the road segments may have no entry or exit points. The direction of an edge may correspond to the direction of travel along the road segment represented by the edge. An indivisible road segment or a road segment with a reversible direction of travel may be represented by a pair of edges. The predecessor of a particular edge may represent the road segment that continues onto the road segment represented by the particular edge, while the successor of a particular edge represents the road segment that the road segment represented by the particular edge may continue onto.

[0039] One or more edges of a road map may have an associated geometry in the form of a sequence of line segments that trace the center of the road segment represented by the edge. In some cases, the sequence may be a single line segment having endpoints corresponding to the endpoints of the segment. In other cases, the sequence may have multiple line segments, e.g., the line segments trace curvature variations along the length of the segment. The points of the line segments may be represented in any number of ways. In one embodiment, the line segment points may be represented in a World Geodetic System (e.g., WGS84) coordinate space.

[0040] A road map may be generated for any region or area. For example, in some cases, a road map is generated for an area including an immediate or nearby area of a vehicle and an immediate or nearby portion of a path considered to be a likely or most likely path. An example of a likely or most likely path includes the current segment of a route input to a head unit of a vehicle. Another example of a likely or most likely path includes a path inferred based on one or more historical routes and / or vehicle telemetry data, which is interpreted to determine driver intent.

[0041] Based on the generated road map, the road map generator 106 may provide road map data 108 to the lane-road matching manager 118. As described herein, the lane-road matching component 118 may use such road map data to perform lane-road matching, as described herein. Alternatively or additionally, the road map data 108 may be stored in a data store for subsequent use (e.g., by the lane-road matching manager 118).

[0042] In addition, as described above, the lane-road matching manager 118 may perform lane-road matching using lane map data 116. In this regard, the lane map generator 114 may generate a lane map, and the data associated therewith may be input to the lane-road matching manager 118 to perform lane-road matching. A lane map or a perceived lane map may be a directed graph representing a lane network. Lane map data (or lane data), such as lane map data 116, may refer to any data used to represent a lane map (e.g., a lane map generated by the lane map generator 114).

[0043] In some cases, the lane map generator 114 is located in a self machine (e.g., an autonomous vehicle). In this regard, the lane map generator 114 may be or include a perception component or module that generates a lane map in real time based on sensor data 112 (e.g., streamed from one or more sensors 110 or other components). Generally, a perception component may collect information from the environment and extract data (e.g., via one or more sensors).

[0044] To generate a lane map and / or lane map data, the lane map generator 114 may obtain and use sensor data 112. In some cases, the sensor data 112 may be pre-processed such that the data has a format that can be accepted and processed. The sensor data 112 may be obtained from any number and any type of one or more sensors 110, such as but not limited to LiDAR sensors, RADAR sensors, cameras, ultrasonic sensors, and / or other sensor types, such as the sensor types described below with respect to the autonomous vehicle 900. For example, one or more sensors 110 may include one or more sensors 110 of the ego machine (e.g., one or more RADAR sensors 960 of the autonomous vehicle 900), and the one or more sensors 110 may be used to generate sensor data 112 that represents the perception in the 3D environment associated with one or more ego machines and the objects in the 3D environment around one or more ego machines. According to embodiments described herein, the sensor data 112 is collected in association with any number of ego machines.

[0045] Based on the obtained sensor data 112, the lane map generator 114 may use such data to generate a lane map. As described, the lane map may be a directed graph representing a lane network. In the lane map, the edges in the graph may represent a sequence of one or more lane segments of one or more linked road segments. The direction of the edge may correspond to the driving direction along the lane segment represented by the edge, but it need not be so. The lane segment represented by a given edge may overlap with the lane segment represented by another edge. The lane representation of the last lane segment of the predecessor of the edge may or may not continue to the lane of the first lane segment represented by the edge, and the lane representation of the last lane segment of the successor of the edge may or may not continue to the lane of the first lane segment represented by the edge.

[0046] Edges can have associated geometric shapes presented in the form of a sequence of line segments that trace the center of one or more lane segments represented by the edge. In some cases, the sequence can be a single line segment having endpoints corresponding to the endpoints of the segment sequence. In other cases, the sequence can have multiple line segments, e.g., the line segments tracing curvature variations along the segment length. The points of the line segments can be represented in any number of ways. In one embodiment, the line segment points are represented in a three-dimensional coordinate space local to the vehicle's position and its orientation. Specifically, the vehicle's position can form the origin; the x-axis can be located on the vehicle's longitudinal axis, where positive values indicate positions in front of the vehicle; the y-axis can be located on the vehicle's transverse axis, where positive values indicate positions to the left of the vehicle; and the z-axis can be located on the vehicle's vertical axis, where positive values are an indication of positions above the vehicle. In some examples, the magnitude of the coordinate values can correspond to meters (or any other unit of measurement). For example, the coordinate values (10.5, -5.3, 1.1) are interpreted as 10.5 meters in front of the vehicle, 5.3 meters to the right of the vehicle, and 1.1 meters above the vehicle.

[0047] A lane map can be generated for any region or area. As an example, the lane map can correspond to a region that includes a frustum having a near plane behind the vehicle and a far plane in front of the vehicle, where the vehicle position is towards the rear of the frustum and centered along the longitudinal axis of the frustum. Briefly refer to Figure 3 , Figure 3 for an illustration of an example frustum covering relative to the vehicle position. In this example, region 302 includes a near plane 304 behind vehicle 306 and a far plane 308 in front of vehicle 306. The vehicle position is towards the rear of the frustum and centered along the longitudinal axis 310 of the frustum.

[0048] Based on the generated lane map, the lane map generator 114 can provide lane map data 116 to the lane-road matching manager 118. As described herein, the lane-road matching manager 118 can use such lane map data to perform lane-road matching. Alternatively or additionally, the lane map data 116 can be stored in a data store for subsequent use (e.g., by the lane-road matching manager 118).

[0049] After obtaining the road map data 108 and the lane map data 116, the lane-road matching component 118 can perform lane-road matching. In this regard, the lane-road matching manager 118 can match or align a lane or a portion thereof (also referred to as a lane segment) with a road or a portion thereof (also referred to as a road segment). In an embodiment, the lane-road matching is or includes a sequence of successively adjacent segments of a given lane that match or align with a sequence of roads or road segments that form a simple path in the road map. Based on geometric similarity analysis, a lane and a road or a portion thereof can be identified as matching. Specifically, when certain conditions are met, such as the lane being within a threshold distance of the road and within a threshold direction of the road, the lane can be identified as matching the road or a portion thereof. When determining that a lane matches a road, a matching score can be generated to indicate the degree of geometric similarity. Since a lane can be determined to match multiple roads (e.g., based on meeting the matching conditions), the matching score can be used to select a particular lane to be mapped to that road. In this way, a particular lane associated with the maximum matching score that matches the road can be selected to produce the lane-road mapping 120.

[0050] A lane-road mapping, such as the lane-road mapping 120, generally refers to mapping one or more lanes to one or more specific roads or portions thereof that maximize geometric similarity (e.g., based on the matching score). The lane-road mapping 120 can be represented in any number of ways. In some cases, the lane-road mapping 120 can be represented using one or more lane segment identifiers and one or more road segment identifiers corresponding to the locations where the lanes are determined to match the roads. As an example, the lane-road mapping can include a sequence of entries corresponding to the matching lane-road portions. The entries can include a lane segment field indicating the start and end positions of the segment along the lane, a road identification field indicating the road, and a road segment field indicating the start and end positions of the segment along the road.

[0051] The lane-road mapping 120 output by the lane-road matching manager 118 can be provided for Figures 9A - 9D use by the vehicle 900 in to perform one or more operations, such as positioning, navigation, and / or other operations. For example, the representation of the lane-road mapping 120 can be used by one or more control components of the vehicle 900, such as in Figures 9A - 9DAn autonomous or semi-autonomous driving software stack 122 that executes on one or more components of vehicle 900 (e.g., one or more SoCs 904, one or more CPUs 918, one or more GPUs 920, etc.). For example, vehicle 900 can use this information (e.g., an obstacle instance) to locate its position in the map, navigate, plan, or otherwise perform one or more operations in the environment (e.g., avoid obstacles or bumps, stay in a lane, change lanes, merge, split, adjust the suspension system of the ego-machine to match the current road surface, apply early acceleration or deceleration based on the approaching road grade, map the environment, etc.).

[0052] In some embodiments, the lane-road mapping 120 can be used by one or more layers of the autonomous driving software stack 122 (alternatively referred to herein as the "driving stack 122"). The driving stack 122 can include a sensor manager (not shown), one or more perception components (e.g., corresponding to the perception layer of the driving stack 122), a world model manager 126, one or more planning components 128 (e.g., corresponding to the planning layer of the driving stack 122), one or more control components 130 (e.g., corresponding to the control layer of the driving stack 122), one or more obstacle avoidance components 132 (e.g., corresponding to the obstacle or collision avoidance layer of the driving stack 122), one or more actuation components 134 (e.g., corresponding to the actuation layer of the driving stack 122), and / or other components corresponding to additional and / or alternative layers of the driving stack 122. In some examples, process 100 can be performed at least in part by one or more perception components or in association with one or more perception components, which can feed the layers of the driving stack 122 to the world model manager, as described in more detail herein.

[0053] The sensor manager can manage and / or extract sensor data from the sensors of vehicle 900. For example, with reference to Figure 9C, sensor data can be generated by one or more LIDAR sensors 964, one or more RADAR sensors 960, one or more ultrasonic sensors 962, one or more stereo cameras 968, other cameras, and / or other sensors (e.g., permanently, intermittently, based on certain conditions). The sensor manager can receive sensor data in different formats from the sensors (e.g., the same type of sensor can output sensor data in different formats), and can be configured to convert the different formats into a unified format (e.g., for each sensor of the same type). Thus, other components, features, and / or functions of the autonomous vehicle 900 can use the unified format, thereby simplifying the processing of sensor data. In some examples, the sensor manager can use the unified format to apply control back to the sensors of the vehicle 900, such as setting the frame rate or performing gain control. The sensor manager can also update the sensor data packets or communications corresponding to the sensor data with timestamps to help inform the various components, features, and functions of the autonomous vehicle control system of the processing of the sensor data.

[0054] The world model manager 126 can be used to generate, update, and / or define a world model. The world model manager 126 can use the information generated and received from one or more perception components of the driving stack 122 (e.g., the locations of detected obstacles). One or more perception components can include an obstacle sensor, a path sensor, a waiting sensor, a map sensor, and / or one or more other perception components. For example, the world model can be defined at least in part based on the affordances of obstacles, paths, and waiting conditions, which can be sensed in real time or near real time by the obstacle sensor, the path sensor, the waiting sensor, and / or the map sensor. The world model manager 126 can continuously update the world model based on newly generated and / or received inputs (e.g., data) from the obstacle sensor, the path sensor, the waiting sensor, the map sensor, and / or other components of the autonomous vehicle control system.

[0055] The world model can be used to help inform one or more planning components 128, one or more control components 130, one or more obstacle avoidance components 132, and / or one or more actuation components 134 of the driving stack 122. The obstacle sensor can perform obstacle sensing, which can be based on where the vehicle 900 is allowed or able to drive (e.g., based on the location of a drivable path or other navigable path defined by avoiding detected obstacles in the environment and / or bumps detected in the road surface), and how fast the vehicle 900 can drive without colliding with an obstacle (e.g., an object such as a structure, an entity, a vehicle, etc.) sensed by the sensors of the vehicle 900.

[0056] The path sensor can perform path perception, such as by perceiving the nominal paths available in a particular situation. In some examples, the path sensor can also consider lane changes for path perception. The lane map can represent one or more paths available to vehicle 900 and can be as simple as a single path on a highway on-ramp. In some examples, the lane map can include paths to the desired lanes and / or can indicate available changes in a highway (or other road type), or can include nearby lanes, lane changes, diverges, turns, cloverleaf interchanges, merges, and / or other information. In some embodiments, the path sensor can consider one or more lane maps and / or one or more lane-road mappings. For example, the path sensor can evaluate the reconstructed 3D road surface to identify lane changes and lane merges.

[0057] The wait sensor can be responsible for determining constraints on vehicle 900 based on rules, conventions, and / or practical considerations. For example, the rules, conventions, and / or practical considerations can be related to the 3D road surface, traffic signals, multi-way stops, yield signs, lane changes, toll booths, gates, police or other emergency personnel, road workers, parked buses or other vehicles, one-way bridge arbitration, ferry entrances, etc. Thus, the wait sensor can be utilized to identify potential obstacles and implement one or more controls (e.g., deceleration, stopping, etc.) that may not be achievable relying solely on the obstacle sensor. In some embodiments, the wait sensor can consider the lane map and / or the lane-road mapping. For example, the wait sensor can evaluate the reconstructed 3D road surface to identify an approaching lane merge and determine to apply and / or apply early acceleration or deceleration to accommodate the approaching lane merge.

[0058] The map sensor can include a mechanism for discerning behavior, which in certain examples is used to determine specific examples of which conventions apply in a particular area. For example, the map sensor can determine based on data representing previous drives or trips that there is no U-turn between certain hours at an intersection, that electronic signs showing lane directionality change based on the time of day, that two traffic signals that are very close (e.g., have little to no offset from each other) are associated with different roads, that the first vehicle waiting to turn left at a traffic signal in Rhode Island turns before oncoming vehicles when the signal turns green and is in violation of the law, and / or other information. The map sensor can inform vehicle 900 of static or fixed infrastructure objects and obstacles. The map sensor can also generate information for the wait sensor and / or the path sensor, for example, to determine which signal at an intersection must be green for vehicle 900 to take a particular path.

[0059] In some examples, information from the map sensor can be sent, transmitted, and / or provided to one or more servers (e.g., the map manager of one or more of the servers 978 in Figure 9D ), and information from one or more servers can be sent, transmitted, and / or provided to the map sensor and / or localization manager of vehicle 900. The map manager can include a cloud map application that is located at a remote location from vehicle 900, and vehicle 900 can access the cloud map application via one or more networks. For example, the map sensor and / or localization manager of vehicle 900 can communicate with one or more other components or features of the map manager and / or one or more servers to inform the map sensor and / or localization manager of the past and present driving or trips of vehicle 900, as well as the past and present driving or trips of other vehicles. The map manager can provide a map output (e.g., map data), which can be localized by the localization manager based on the specific location of vehicle 900, and the localized map output can be used by the world model manager 126 to generate and / or update the world model.

[0060] One or more planning components 128 can include a route planner, a lane planner, a behavior planner, and a behavior selector, as well as other components, features, and / or functions. The route planner can use information from the map sensor, the map manager, and / or the localization manager, as well as other information, to generate a planned path, which can consist of GNSS waypoints (e.g., GPS waypoints), 3D world coordinates (e.g., Cartesian coordinates, polar coordinates, etc.) indicating coordinates relative to an origin on vehicle 900, and so on. A waypoint can represent a specific distance into the future of vehicle 900, such as the number of city blocks, kilometers, feet, inches, miles, etc., and can be used as a target for the lane planner.

[0061] The lane planner can use the lane map and / or lane-road mapping, the object poses and / or distances in the lane map or lane-road mapping, and a target point and direction a certain distance into the future from the route planner as inputs. The target point and direction can be mapped to the best-matching drivable point and direction in the lane map (e.g., based on GNSS and / or compass direction). Then a graph search algorithm can be executed on the lane map and / or lane-road mapping starting from the current edge to find the shortest path to the target point.

[0062] The behavior planner can determine the feasibility of a basic behavior of vehicle 900 (e.g., staying in a lane or changing lanes left or right) so that the feasible behavior can match the most desired behavior output from the lane planner. For example, if the desired behavior is determined to be unsafe and / or unavailable, a default behavior can be selected instead (e.g., when the desired behavior or lane change is unsafe, the default behavior can be to stay in the lane). The behavior planner can use a lane-road map to facilitate such planning.

[0063] One or more control components 130 can follow a trajectory or path (lateral and longitudinal) received from the behavior selector of one or more planning components 128 as closely as possible within the capabilities of vehicle 900. One or more control components 130 can use strict feedback to handle unmodeled unplanned events or behaviors and / or anything that causes a difference from the ideal value (e.g., unexpected delays). In some examples, one or more control components 130 can use a forward prediction model that takes control as an input variable and produces a prediction that can be compared to a desired state (e.g., compared to the desired lateral and longitudinal paths requested by one or more planning components 128). One or more controls that minimize the difference can be determined.

[0064] Although one or more planning components 128 and one or more control components 130 are shown separately, this is not intended to be restrictive. For example, in some embodiments, the boundary between one or more planning components 128 and one or more control components 130 may not be precisely defined. Thus, at least some of the components, features, and / or functions attributed to one or more planning components 128 can be associated with one or more control components 130 and vice versa. This can also apply to any separately shown component of the driving stack 122.

[0065] One or more obstacle avoidance components 132 may assist an autonomous vehicle 900 in avoiding collisions with objects (e.g., moving and stationary objects). One or more obstacle avoidance components 132 may include computing mechanisms at an obstacle avoidance “primitive level” and may act as the “survival brain” or “reptile brain” of the vehicle 900. In some examples, one or more obstacle avoidance components 132 may be used independently of the components, features, and / or functions of the vehicle 900 that are required to comply with traffic rules and drive courteously. In such examples, one or more obstacle avoidance components may ignore traffic laws, road rules, and norms of courteous driving to ensure that no collision occurs between the vehicle 900 and any object. Thus, the obstacle avoidance layer may be a layer separate from the road rules layer, and the obstacle avoidance layer may ensure that the vehicle 900 performs safe actions only from the perspective of obstacle avoidance. On the other hand, the road rules layer may ensure that the vehicle complies with traffic laws and practices and adheres to legal and conventional rights of way (as described herein).

[0066] In some examples, one or more obstacle avoidance components 132 may use a drivable path or other navigable path and / or one or more lane maps or one or more lane map mappings to determine a control or action to take. For example, the drivable path may provide an indication to one or more obstacle avoidance components 132 of where the vehicle 900 can be maneuvered without hitting any objects, bumps, structures, etc., or at least an indication of where there are no static structures.

[0067] In a non-limiting embodiment, one or more obstacle avoidance components 132 may be implemented as a separate, discrete feature of the vehicle 900. For example, one or more obstacle avoidance components 132 may operate independently (e.g., in parallel with, before, and / or after) of the planning layer, control layer, actuation layer, and / or other layers of the driving stack 122.

[0068] Thus, the vehicle 900 may use this information (e.g., as the edges or tracks of a path) to navigate, plan, or otherwise perform one or more operations (e.g., lane keeping, lane changing, merging, splitting, etc.) in the environment. Although the driving stack 122 is shown as obtaining a lane-road mapping, it can be understood that the various components associated with the driving stack 122 may be used to facilitate the generation of the lane-road mapping.

[0069] Now turning to Figure 2 , Figure 2 is a data flow diagram illustrating an example process 200 for facilitating lane and road matching according to some embodiments of the present disclosure. In some embodiments, process 200 represents Figure 1Possible ways in which the lane - road matching manager 118 matches lanes with roads or portions thereof and / or generates a lane - road map therefrom.

[0070] Figure 2 Illustrated is a road map generator 206 that processes road network data 202 and a lane map generator 214 that processes sensor data 212 into a format (input data 205) that can be used by a lane - road matching manager 218 and feeds the input data 205 to the lane - road matching manager 218, which can generate one or more lane - road maps 220. In some embodiments, Figure 2 the components illustrated in can be implemented in a self - machine. For example, the road map generator 206, the lane map generator 214, and the lane - road matching manager 218 can run in a self - machine to provide in real - time a lane - road map that can be used to perform localization (e.g., locate the self - machine in a map), path planning (e.g., determine candidate routes for the self - machine), and / or make decisions. In other embodiments, Figure 2 one or more of the components shown in can be implemented remotely from the self - machine. For example, the lane - road matching manager can run in a server that communicates with the self - machine.

[0071] The road map generator 206 is generally configured to obtain road network data 202. The road network data 202 can include various types of data associated with the road network. By way of example and not limitation, the road network data can include information about the road network infrastructure, such as the topology and connectivity of the road network, the locations of traffic signals, speed limits for different road segments, etc.

[0072] As described above, the road map generator 206 generates a road map, and the data associated therewith is provided as input data 205 to the lane - road matching manager 218. The road map generator 206 can generate the road map and / or road map data in a streaming or real - time manner. For example, as road network data is streamed from a road network data source, the road map generator 206 can generate a road map associated with the obtained road network data. In some cases, the road map data provided to the lane - road matching manager 218 can be or include the generated road map or a representation thereof.

[0073] The lane map generator 214 is generally configured to obtain sensor data 212. The sensor data 212 can include various types of data that provide perception associated with the self - machine or the environment in which the self - machine is located.

[0074] As described above, the lane map generator 214 generates a lane map, and the data associated therewith is provided as input data 205 to the lane-road matching manager 218. The lane map generator 214 can generate the lane map and / or lane map data in a streaming or real-time manner. For example, as sensor data is streamed from one or more sensors (e.g., sensors associated with a ego machine or a non-ego machine), the lane map generator 214 can generate a lane map associated with the acquired sensor data. In some cases, the lane map data provided to the lane-road matching manager 218 can be or include the generated lane map or its representation.

[0075] Based on the lane-road matching manager 218 obtaining the input data 205 (including road map data and lane map data), the lane-road matching manager 218 is configured to perform lane-road matching. As shown, the lane-road matching manager 218 can include a matching starter 240, a matching manager 242, and a map manager 244. As can be understood, these components are provided for illustrative purposes, and any number of components can be used to implement the functions described herein.

[0076] The matching starter 240 is generally configured to initiate the execution of lane-road matching. In this regard, the matching starter 240 can identify one or more root lanes and / or root roads for lane-road matching analysis. In an embodiment, the matching starter 240 analyzes the input data 205 (e.g., lane map data and / or road map data) to identify one or more root lanes and / or root roads associated therewith. A lane can be identified as a root lane when there is a position along the geometry of the lane that is within a predetermined distance of the position of the ego machine. Based on identifying a lane as a root lane, a root lane position can also be identified. The root lane position can be the position closest to the position of the ego machine. In some cases, the same or similar positions can be identified. In such cases, the root lane position can be the position closest to the starting point (or other predetermined point) of the geometry of the lane. Initiating lane-road matching using one or more root lanes will limit one or more lane-road matches and / or lane-road mappings to one or more lanes reachable from the one or more root lanes.

[0077] When there is a location within a predetermined distance from the position of the ego machine along the geometry of the road, the road can be recognized as a root road. In some cases, the predetermined distance is the same as the distance used to recognize root lanes. Based on recognizing the road as a root road, the root road position can also be recognized. In some cases, the root road position can be the position closest to the position of the ego machine. In the case where multiple positions are recognized as the position closest to the ego machine position, the root lane position can be the position closest to the starting point (or other predetermined point) of the lane geometry. Using the root road to initiate lane-road matching restricts one or more lane-road matches and / or lane-road mappings to one or more roads reachable from one or more root roads.

[0078] In an embodiment, the match initiator 240 can initialize values associated with lane-road matching. As described herein, match scores associated with road-lane matching can be generated during the matching process. In this regard, the match initiator 240 can initialize the match score value, which can be adjusted during the matching process. For example, the match score value for a particular lane-road pair can be initialized to 0 or negative infinity. A match indicator can also be initialized, which indicates whether the lane-road pair or its parts match.

[0079] In some cases, the match initiator 240 can also initialize the values of the lane-road mapping. For example, for a lane-road pair or its parts, the mapping can be initialized to empty, indicating that no match of one or more lane segments to one or more road segments has been recognized yet.

[0080] In an embodiment, to perform lane-road matching, the lane map data and / or the road map data can be transformed or reformatted to align with each other. As an example, matching can be performed using geometric calculations corresponding to the coordinate space of the lane map. As described above, in one embodiment, the lane map can be represented or expressed in a three-dimensional coordinate space local to the ego machine position and its orientation. Thus, the road map or the data associated therewith can be transformed into this three-dimensional coordinate space. For example, the WGS84 coordinates of the points of the road geometry associated with the road map can be transformed into coordinates local to the ego machine position and orientation. Advantageously, in addition to eliminating the need for non-trivial spherical geometric calculations, this transformation also facilitates the execution of lane map matching due to the vehicle position and orientation.

[0081] The matching manager 242 is generally configured to perform lane - road matching. As described above, lane - road matching identifies a match between a lane and a road or a portion thereof. In this regard, any number of lane segments can be identified to match any number of road segments. Identifying lane segments that match road segments enables enhanced execution of various autonomous vehicle functions, such as lane - level directives (e.g., used by the behavior planning module), annotating lanes with road - level attributes from a road map (e.g., road curvature values and traffic control signs and signals), and so on.

[0082] Lane - road matching can be performed in any of a variety of ways. In one embodiment, a lane is identified as matching a road when a set of matching conditions are met. Matching conditions can generally refer to any conditions that can be used to identify a match between a lane and a road or a portion thereof. As described herein, lane - road matching can be performed based on geometric similarity. In this way, a lane - road match or a portion thereof can be identified when the lane is geometrically similar to the road. Thus, the matching conditions can correspond to geometric similarity parameters.

[0083] An example of a matching condition includes the distance between the lane and the road being within a distance threshold. For example, when the distance between the lane and the road is within a predetermined distance, the lane can be considered to match the road. The distance threshold can be established in any number of ways. For example, the distance threshold can be based on a default setting, a preset setting, or a user - selected setting. Additionally, the distance threshold can be expressed in any number of ways. For example, a specific metric (e.g., meters) can be used to specify the distance to use the threshold to determine a match. In an embodiment, the distance threshold is large enough to account for differences in position, orientation, and granularity between the geometries of the lane map and the road map, but not so large as to reduce the reasonableness of the match.

[0084] In some cases, to identify whether the distance condition is met, the vertical spans extending from the lane and / or the road can be analyzed (e.g., via the lane map and / or the road map). For example, a vertical line can be extended from the road being analyzed until it intersects the lane being analyzed. The distance from the road to the intersection point (also referred to as the intersection distance) can be determined and compared to the distance threshold. In the case where the intersection distance from the road to the intersection point is equal to or less than the distance threshold, the distance condition can be considered to be met at that particular location. As another example, a vertical span of the distance threshold length can be extended from the road. The vertical span can have any length and / or metric. As an example, the vertical span can be ten meters. In the case where there is an intersection between the vertical span and the lane, the distance condition can be considered to be met at that particular location. Although these examples extend the vertical span from the road to identify an intersection with the lane, the vertical span can alternatively be extended from the lane to identify whether there is an intersection with the road and / or to identify the distance to achieve an intersection with the road.

[0085] Another example of a matching condition includes the lane being within a directional threshold direction of the road. For example, when a lane is within a predetermined direction relative to the road, the lane can be considered to match the road. The directional threshold can be established in any number of ways. For example, the directional threshold can be based on a default setting, a preset setting, or a user-selected setting. Additionally, the directional threshold can be expressed in any number of ways. For example, a specific metric (e.g., degrees) can be used to specify the directional threshold for determining a match. In an embodiment, the distance threshold is large enough to account for differences in position, orientation, and granularity between the lane map and the road map, but not so large as to reduce the reasonableness of the match.

[0086] In some cases, to identify whether the direction condition is met, a dot product function can be used to determine the similarity between the lane direction at a specific location and the road direction at a specific location. Generally, when two vectors (e.g., a vector associated with the lane and a vector associated with the road) point in the same direction, the dot product is positive. The more the two vectors point in the same direction, the larger the result of the dot product between the two vectors. In this way, vectors associated with the lane and points or positions along the road can be identified and used to determine whether the direction condition is met. Other methods (e.g., transcendental functions) can be employed to determine direction similarity.

[0087] As an example implementation of using the dot product function, the dot product can be used to determine directionality given a road R and a position r along R, a perceived lane L, and a position l along L. When there is a viable vertical span intersection between R and L (e.g., from r to l), a dot product directionality test can be employed, thereby calculating a normalized vector representing the direction of R at r and a normalized vector representing the direction of L at l. Subsequently, the dot product of the two normalized vectors is determined and the dot product result is compared to a given threshold, e.g., a threshold of approximately 0.77, which corresponds to an angular similarity of approximately 45 degrees.

[0088] It can be understood that in some cases, the calculation of the direction for a lane and / or a road (e.g., associated with a vector) can include the calculation of a square root, and direction data associated with the geometry of the lane and / or the road (such as a normalized direction) can be stored (e.g., cached) for subsequent use. For example, since the direction associated with a lane and / or a road can be used for future matching condition analysis, caching the normalized direction on demand can facilitate a more efficient lane-road matching process.

[0089] In some implementations, it may be necessary to satisfy or meet each matching condition in a set of matching conditions to identify a match associated with a lane and a road or a portion thereof. For example, assume a set of matching conditions includes a distance condition and a direction condition. In such a case, it may be necessary to satisfy both the distance condition and the direction condition to determine a lane-road match.

[0090] The matching manager 242 can perform a matching condition analysis in association with the lane being analyzed and / or various points or positions along the road. In this regard, the lane being analyzed and / or the various points or positions along the road can be identified and used to perform a matching analysis in association with these points or positions. These points or positions can be selected in any number of ways. As an example, the points being analyzed can be a specific distance apart from each other (e.g., increments of five meters apart from each other). For example, assume that a lane is being analyzed at a first position, and an intersection of a vertical segment extending from the lane to the road is identified at the first position. Then the first position associated with the lane and the road can be analyzed in association with one or more matching conditions. Thereafter, a second position of the lane five meters in front of the first position of the lane can be identified. Then an intersection of a vertical segment extending from the lane at the second position to the road at the second position can be identified. Then the second position associated with the lane and the road can be analyzed in association with one or more conditions.

[0091] In some cases, in order to identify a match between a lane and the road or a portion thereof, a threshold number of points or points within a threshold distance can be analyzed and identified as a match. For example, assume that only two consecutive points between a lane and the road are identified as a match. In this case, even though two points on the lane are identified as matching the road (e.g., satisfying the matching conditions), a lane-road match cannot be determined.

[0092] In an embodiment, the matching manager 242 performs lane-road matching in an iterative manner. In this regard, the matching manager 242 can advance along the length of the lane and / or the road to identify whether a set of matching conditions is satisfied at a particular position. Thus, at each incremental position along the road and / or the lane, a set of matching conditions can be analyzed in association with the lane-road pair being analyzed.

[0093] As an example, at each point along a road and / or lane, an intersection between the vertical span of the road and the lane is identified at a location. The vertical span at a location on the road can be a line segment centered at that location along the road and extending perpendicular to the direction of the road at that location. In some cases, the vertical span can extend in two directions. For example, the length of the vertical span can be twice the radius of the vertical span. The intersection result (e.g., the intersection result between the lane and the vertical span extending from the road) is evaluated according to a set of matching conditions to determine if there is a match. One example condition is that there is a unique intersection location on the lane that is greater than or equal to the previous lane location. Another example matching condition is that, according to a direction similarity threshold, the direction of the lane at the intersection location is similar enough to the direction of the road at that location. In the case where a match is identified, various parameters can be updated. After identifying a match and updating any parameters, an increment is used to advance along the road. In some cases, the increment reflects the walk rate. A higher walk rate indicates a smaller distance between consecutive vertical span positions along the length of the road map. A higher walk rate can result in a higher accuracy and reasonableness of the match determination.

[0094] As described herein, individual road segments can be used to represent individual roads. Thus, depending on the completion of the advancement or walk associated with a road segment, it can be determined if there are any unvisited successor road segments for that road. Specifically, any unvisited successor road segments along the current path traversing the road map can be identified. In the presence of successor road segments, the advancement can continue along the successor road segments. This iterative advancement enables the matching of any number of road segments.

[0095] In an embodiment, an iterative process of performing a matching analysis on a lane and a road or a portion thereof can continue until a termination event is identified. A termination event can be any event indicating the termination of the matching analysis of the lane and / or the road. In this regard, the matching analysis can be advanced along consecutive points on the road and / or lane until a termination event is identified. One example of a termination event is the identification of the end of the lane and / or road being analyzed. For example, assume the end of a lane segment and / or a road segment is identified and no subsequent or successor lane segment and / or road segment is identified. In this case, the matching analysis for the current lane and road pair being analyzed can be terminated.

[0096] Another example of a termination event is the identification of a failed match. For example, assume the matching analysis performed associated with a particular point along the road results in a failure to meet the matching conditions (e.g., distance condition and / or direction condition). In this case, the failure to meet the matching conditions can be considered a termination event for which the matching analysis for the current lane and road pair being analyzed can be terminated.

[0097] In some cases, the termination condition may not be recognized until a threshold number or distance of match conditions fails (such as, consecutive match conditions fail). For example, assume that the threshold number of failed match conditions is five. In this case, after recognizing that the match conditions associated with five consecutive points on the current path of the road map fail, a termination event can be recognized. Implementing the failure of the match condition of the threshold number or distance enables a more efficient process. For example, without limitation, the iterative matching process can proceed along the entire length of the road map path (e.g., continue for hundreds of meters) without observing a viable intersection or match. In operation, the lower the limit, the better the performance of the algorithm, however, using too low a limit will result in a lower rationality of the calculated match results. In addition, when matching standard-defined road map data with lane geometries based on fine-grained perception, this method enables tolerance of the coarse-grained geometric road representation of the standard-defined road map data (e.g., especially at a confluence where the road geometry is arranged at a right angle, while the sensed lane can capture the smooth curvature of the confluent lane).

[0098] In some embodiments, the match manager 242 may be configured to perform intersection modification in association with identifying lane-road matches. By way of example, and with reference to Figures 4A - 4B , Figures 4A - 4B a diagram illustrating the effect of prologue intersection modification is provided. Figure 4A An example match calculation between root lane l and root road r without applying the modification is provided, while Figure 4B an example match pair calculation between root lane l and root road r with the modification applied is provided. In Figure 4A and Figure 4B , the position of the ego machine is represented by a five-pointed star, the root lane position is represented by a triangle, and the root road position is represented by a circle. The line segments emanating from road r represent the vertical span of the threshold distance along the length of r, where the dashed line segments correspond to non-match results (e.g., one or more match conditions are not satisfied), and the solid line segments correspond to match results (e.g., match conditions are satisfied). In Figure 4A , line l starts in the direction of r after road r. Thus, after advancing the position along r, the initial intersection result is not viable before transitioning to a series of viable results. Therefore, the calculated match between l and r exhibits a misalignment at the start of the match. By adding prologue intersection modification, when the initial intersection result between l and r is not viable, a fallback intersection calculation is performed using an instance of the vertical span translated from the root position of r to the root position of l, as shown in Figure 4BAs shown. If the translated vertical span intersects r at a unique location, and the intersection location is between the root location of r and the next location along r, the intersection result is considered feasible, and the effect is to eliminate the above misalignment.

[0099] Now turn to Figures 4C - 4D , Figures 4C - 4D A diagram showing the effect of including epilogue intersection modification is provided. Figure 4C An example matching calculation between the terminal lane l and the road r without applying the modification is provided, while Figure 4D an example of applying the modification is provided. The line segments emanating from r represent the vertical spans for intersection result calculation along the length of r, where the dashed line segments correspond to infeasible matching results and the solid line segments correspond to feasible matching effects. In this example, when the matching calculation terminates with a series of infeasible results, a fallback intersection calculation can be performed using an instance of the vertical span of r that is the road with the most recent feasible matching result, which is translated to the end of l. If the translated vertical span intersects r at a unique location and the intersection location follows the position of the most recent feasible result, i.e., along r, it is considered feasible, and the effect is to eliminate the above misalignment.

[0100] In some implementations, a linear scan can be used to perform the matching analysis. In other implementations, a spatial index can be used to perform the matching analysis. The spatial index can limit the number of geometries to be evaluated. The spatial index is initialized at the start of the Match method with the geometries of the lanes reachable from one or more root lanes. In practice, the upfront cost associated with the construction of the spatial index is expected to be amortized during the execution of the algorithm and ultimately reduce the number of intersection calculations performed by the algorithm by several orders of magnitude.

[0101] In addition to identifying matches associated with lanes and roads or portions thereof, the match manager 242 can also be configured to generate a match score to indicate the degree of the match. In an embodiment, the match score can be based on geometric similarity. Since area calculation would be computationally intensive and needs to be performed in real time on the self machine, an approximation of the area-based geometric similarity can be used, which is based on the distances between the lane and the road at positions regularly sampled along the road length. For example, given a lane and a road, the geometric similarity between a given section of the lane and a given section of the road can correspond to the sum of the reciprocals of the distances between the lane and the road at the sampled positions. In other words, the higher the degree of fit or proximity between the lane section and the road section, the higher the geometric similarity, and thus the higher the quality of the corresponding match.

[0102] To determine the reciprocal of the distance between a lane position and a road, in some cases, a vertical span extending from the road and / or the lane can be used. As an example, the vertical span can extend from a road point until the vertical span intersects the lane or reaches a distance threshold. The intersection distance can be subtracted from the distance threshold to generate the reciprocal of the distance between the lane position and the road. For example, assume the vertical span extends from the road to intersect the lane, and such an intersection distance is identified as 0.2 units. Further assume the distance threshold is 1.0 unit. In this case, the reciprocal of the distance is 1.0 – 0.2, which results in a match score of 0.8 for that particular position. A relatively high match score of 0.8 indicates a high degree of similarity at that particular position.

[0103] In some cases, a match score can be determined at a sampled location (e.g., each sampled location). For example, by advancing along the road to the next point and performing a matching analysis, the associated match score can be determined. In some cases, the match score determined for a sampled location can be specific to that location (e.g., the reciprocal of the distance between the lane point being analyzed and the road point). In other cases, the match score determined for a sampled location can be a running total or sum (e.g., the sum of the reciprocals of the distances associated with the current sampled location and previous sampled locations along the path). Additionally or alternatively, the match score can be determined based on identifying a terminal event (e.g., the end of the lane or road being analyzed). For example, as described above, when no more matches are identified (e.g., within a threshold distance or number of consecutive points) or when the end of the lane or road is identified, the match score can be determined. In this example, the match score can be determined when the end is identified along the matching process.

[0104] The map manager 244 is generally configured to manage the generation of a lane - road map. As described herein, the lane - road map provides an indication of lane - road matching or alignment. In an embodiment, the lane - road map is represented by one or more entries indicating alignment segments. The lane - road mapping (e.g., lane - road mapping 220) can be a mapping of one or more lanes to one or more specific roads or portions thereof that maximizes geometric similarity (e.g., based on a matching score). The lane - road mapping 220 can be represented in any number of ways. In some cases, the lane - road mapping 120 can be represented using lane segment identifiers and road segment identifiers corresponding to the locations where lanes are determined to match roads. As an example, the lane - road mapping can include a sequence of entries corresponding to matched lane - road segments. An entry can include a lane identification field indicating the lane, a lane segment field indicating the start and / or end positions of the segment along the lane, a road identification field indicating the road, and a road segment field indicating the end and / or start positions of the segment along the road. Thus, different entries can correspond to different lane segment and road segment matches. For example, assume a lane is identified as matching a first road segment. In this case, the first entry can indicate the match between the lane and the first road segment. Assume the iterative matching process continues and identifies the lane as matching a second road segment after the first road segment. In this case, the second entry can indicate the match between the lane and the second road segment.

[0105] As described above, in some cases, multiple lane - road matches can be identified. For example, a lane can be identified as matching one road path and another road path (e.g., based on meeting matching criteria). In this case, the map manager 244 can identify or select the most fitting lane - road match for generating the lane - road mapping. In this regard, the matching score can be used to select which lane - road match to use for generating the lane - road mapping. For example, as described above, the larger the matching score, the greater the geometric similarity can be represented. Thus, to maximize the geometric similarity of the lane - road mapping, the lane - road match with the largest matching score can be selected and used for generating the lane - road mapping. In other embodiments, lane - road mappings associated with various lane - road matches can be generated, and the mapping associated with the largest matching score can be selected for output or for performing various autonomous vehicle functions, such as lane - level instructions (e.g., used by a behavior planning module), annotating lanes with road - level attributes from a road map (e.g., road curvature values and traffic control signs and signals), and so on.

[0106] In some embodiments, the map manager 244 may perform various post - processing operations. For example, consider a lane - road match that includes a segment of road r followed by a segment of road r', where road r' is a successor of road r. Since the entries of the computed match can be based on specific positions or points of the lane - road match results, it is possible for the entry associated with r to define a segment of r that does not end at the end of r, and furthermore, it is possible for the entry associated with r' to define a segment of r' that does not start at the beginning of r'. Similarly, these entries can define segments of lane l that are not continuous along the length of lane l. In this way, post - processing steps can be utilized to ensure, for example, that the mapped segments of lane l are continuous along the length of lane l. As an example, the map manager 244 can apply heuristics to resolve cases where there is ambiguity in the positions of the start and end points of the matched segments.

[0107] By way of example only, Figure 5 a diagram of a lane - road match is provided. In Figure 5 , lane l is analyzed to match segments of a road map that includes r a , r b , r c and r d . The line segments extending along the road lengths represent vertical spans with a specific distance threshold. In the case where the vertical span intersects the lane, the road can be identified as matching the lane. In Figure 5 , the solid line segments indicate matches, while the dashed line segments correspond to non - match results. During execution, the algorithm analyzes the geometry of the lane relative to the geometry associated with the path of the road map (e.g., a sequence of roads). In this example, the first road path includes r a , r b , r d , and the second road path includes r a , r c . An iterative approach can be employed that first progresses along the road path to perform the geometric similarity analysis. In this way, the lane - road match analysis can be performed at positions along the road corresponding to the vertical line segments.

[0108] Referring to Figure 5 , Figure 5 an example of generating a match score is provided. For example, assume that the analysis of the first road r a results in a match score of 1. Now assume that road r dTo obtain a matching score. First, referring to line segment 502, assume that the radial distance threshold to which the line segment extends 504 is 1. In some cases, the threshold 1 can represent a distance or a normalized value. Now assume that it is determined that line segment 502 intersects the lane at 0.7. In this case, the matching score can be determined to be 0.3 (1 - 0.7). The new total matching score for this path is now 1.3. Next, line segment 506 can be analyzed, resulting in a corresponding matching score of 0.3 (1 - 0.7) again. In this case, the new total matching score for the path is now 1.6. Assume that this iterative process continues until line segment 508, resulting in a total matching score of 1.6 because the remaining line segments intersect at the radial distance threshold, resulting in a corresponding matching score of 0. In this example, since the matching score of path r a 、r b 、r d is 1.6, which is greater than the matching score of r a 、r c which is 1, path r a 、r b 、r d can be selected to generate the lane - road mapping.

[0109] Various example algorithms are provided below to illustrate ways of implementing the techniques described herein. In these examples, a basic matching algorithm and a general matching algorithm are provided. The basic matching algorithm is typically limited to lane graphs where lanes have no successors, while the general matching algorithm is an extension of the basic matching algorithm that eliminates the limitations of the basic matching algorithm. In this regard, when lanes do have successors, the general matching algorithm can be used. It can be understood that the general matching algorithm is applicable even when lanes have no successors.

[0110] First, a basic matching algorithm for implementing the basic matching algorithm is provided:

[0111]

[0112] In this implementation, the method first initializes the mapping and associated matching scores for each lane. In this example, the mapping is initialized to be empty and the matching score is initialized to negative infinity. For each root lane l and each root road r, the method calls a basic matching recursive method as described below. Each call to MatchRecursive results in an acyclic depth - first traversal of the road graph starting from r. During the traversal, the mapping and the corresponding matching scores are calculated, which can be applied to the sequence of roads in the road graph that includes the current path for l. At the start of the traversal, the calculated mapping is empty and the calculated matching score is equal to negative infinity. Each time the traversal reaches the end road, if the calculated matching score is greater than the matching score of l, the mapping of l and the matching score of l are updated to the calculated mapping and the calculated matching score respectively.

[0113] An implementation of a basic matching algorithm for performing recursive or iterative matching methods can be as follows:

[0114]

[0115] In this algorithm, the method includes an identity parameter, denoted by r; a road position parameter, denoted by ρ; a mapping parameter, denoted by M; and a matching score parameter, denoted by μ. The method begins by initializing a mapping sequence entry e by setting the start and end positions of the lane segment field to λ, the road identification field to r, and the start and end positions of the road segment field to ρ. After initialization, e is pushed onto M. Next, while ρ is less than the length of r, the method advances along the length of r at a regular rhythm, called the walking rate. At each step of the walk, the intersection result between l and the vertical span of r at position ρ is calculated. The vertical span of r at position ρ can be a line segment that extends along r centered at position ρ and continues perpendicular to the direction of r at position ρ. The length of the vertical span can be twice the vertical span radius. The intersection result can be evaluated according to a pair of conditions to determine whether it represents a match.

[0116] In this example, the first condition is that there is a unique intersection position along l that is greater than or equal to λ. The second condition is that, according to the direction similarity threshold, the direction of l at the intersection position is similar enough to the direction of r at position ρ. If the intersection result is recognized as a match, it is processed. The match result can be processed by setting λ to the intersection position along l; setting the end position of the lane segment field of e to the intersection position along l; setting the end position of the road segment field of e to ρ; and μ is incremented by the difference between the vertical span radius and the distance between the intersection position along l and position ρ along r. After calculating the match result and processing it if applicable, the walk along r is advanced by ρ, and ρ is incremented by an amount reflecting the walking rate. After completing the walk, if r has any unvisited successors, e.g., the current path along the traversal of the road map, the method recursively calls itself to obtain each unvisited successor r' of r, using l as the lane identification parameter, λ as the lane position parameter, r' as the road identification parameter, ρ - length(r) as the road position parameter, M as the mapping sequence parameter, and μ as the matching sequence score parameter. If r has no unvisited successors, the method processes the calculated match result by checking μ and the matching score of l. If μ is greater than the matching score of l, the match of l is updated to M, and the matching score of l is updated to μ. At the end of the method, e is popped from M.

[0117] Turning to the general matching algorithm, the following example provides a matching method for implementing the general matching algorithm:

[0118]

[0119] This method is generally similar to the matching method of the basic matching algorithm. In this method, a version of the general matching recursive method is called, which results in performing an acyclic depth-first traversal of the road graph starting from r and an acyclic depth-first traversal of the lane graph starting from l. During the traversal, the mapping and the corresponding matching scores can be calculated, which are applicable to the lanes in the lane sequence including the current path in the lane graph, and this lane sequence corresponds to the road sequence including the current path in the road graph. Each time the traversal reaches the end road, for each lane l in the current path in the lanes, if the calculated matching score is greater than the matching score of l, the mapping of l is updated to the subsequence of the entries specific to l in the calculated mapping, while the matching score of l is updated to the calculated matching score.

[0120] An example of the general matching algorithm for performing a recursive or iterative matching method is as follows:

[0121]

[0122] This method is generally similar to the corresponding method of the basic matching algorithm. In this method, the entries of the current mapping can be extended by adding a lane identification field. The lane identification field indicates that the entry is specific to a given lane. At the start of this method, when initializing e, the lane identification field of e can be set to l. After completing the walk of r, when r has no unvisited successors, this method processes the calculated mapping results by checking the matching scores of each lane l'. For each lane l', there is one or more entries in the calculated matching whose lane field is l'. For each such lane l', if μ is greater than the matching score of l', the matching of l' is updated to the subsequence of the entries in the calculated matching whose lane field is equal to l', and at the same time the matching score of l' is updated to μ.

[0123] In addition, in this method, when the calculated intersection result is not recognized as a match, this method will continue to recursively call MatchRecursive for each lane that is an unvisited successor of l. In other words, if there is no matching intersection result at the current lane, this method will move to attempt to generate a matching intersection with the lanes in the unvisited sub-paths of the lane graph rooted at that lane.

[0124] Now refer to Figures 6 - 8, each block of the methods 600, 700, and 800 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. These methods can also be embodied as computer-usable instructions stored on a computer storage medium. These methods can be provided by a stand-alone application, service, or hosted service (independently or in combination with another hosted service), or a plug-in of another product, to name a few. Additionally, by way of example, the methods 600, 700, and 800 will be described with respect to Figure 1 's lane-road matching system. However, these methods can alternatively or additionally be performed by any one system or any combination of systems, including but not limited to the systems described herein.

[0125] Figure 6 is a flowchart showing a method 600 for generating a lane-road map according to some embodiments of the present disclosure. Method 600 includes, at block B602: receiving road data representing at least one road segment and lane data representing a lane associated with the position of the ego machine. The road data representing the road segment can be road map data, and the lane data representing the lane can be lane map data.

[0126] Method 600 includes, at block B604: determining that the one or more consecutive road segments match the lane based at least on the geometric similarity between the lane and the one or more consecutive road segments. Determining the match between a road segment and a lane based on geometric similarity can be performed in any number of ways. As an example, such determination can be based on the lane being within a distance threshold of one or more points on the consecutive road segment. In some cases, determining whether the lane is within the distance threshold can include extending a perpendicular line segment from a point on the consecutive road segment to intersect the lane. Such determination can be performed iteratively, for example, by advancing along consecutive points (e.g., positioned a predetermined distance apart) on the consecutive road segment. Alternatively or additionally, such determination can be based on the lane being within a direction threshold of one or more points on the consecutive road segment. In some embodiments, the degree of match between the consecutive road segment and the lane is determined. In some cases, such degree can be generated based on the sum of one or more reciprocals of the distances between the lane and one or more sampled positions along the consecutive road segment.

[0127] Method 600 includes, at block B606: generating a representation of lanes mapped to one or more consecutive road segments based at least on determining that the one or more consecutive road segments match the lanes. In some embodiments, the representation of lanes mapped to consecutive road segments may be based on a determination that generates the mapping. For example, mapping may be desired when the degree to which consecutive road segments match the lanes is greater than other degrees of match associated with other road segments. In one embodiment, the representation of lanes mapped to consecutive road segments includes representations of the starting and ending positions of segments along the lanes, a representation of an identification of at least a portion of the consecutive road segments, and representations of the starting and ending positions of at least one road segment.

[0128] Figure 7 A flowchart of a method 700 for generating a lane-road mapping according to some embodiments of the present disclosure is shown. Method 700 includes, at block B702: determining that a road segment of a road map associated with a self-vehicle matches a lane of a lane map associated with the self-vehicle based at least on a geometric similarity between the lane and the road segment. Determining that the road segment matches the lane based on the geographic similarity may be performed in any number of ways. As an example, such determination may be based on the lane being within a distance threshold of one or more points on the consecutive road segment. In some cases, determining whether the lane is within the distance threshold may include extending a perpendicular line segment from a point on the consecutive road segment to intersect the lane. Alternatively or additionally, such determination may be based on the lane being within an orientation threshold of one or more points on the consecutive road segment. In one embodiment, the matching determination may be performed iteratively, for example, by advancing along consecutive points (e.g., located a predetermined distance apart) of the consecutive road segment until a termination event is detected. For example, a termination event may be detected when the road or lane ends. As another example, a termination event may be detected when a threshold number (e.g., five) of consecutive matches along the points of the road are identified as failures.

[0129] Method 700 includes, at block B704: generating a representation of the degree to which the road segment matches the lane based at least on the geometric similarity between the lane and the road segment. In one embodiment, the representation of the degree to which the road segment matches the lane is generated based at least on the sum of one or more reciprocals of one or more distances between the lane and one or more sampled positions on the road segment.

[0130] Method 700 includes, at block B706: generating a representation of lanes mapped to a road segment based at least on a representation of the degree of match of the road segment with the lanes. In some embodiments, the representation of lanes mapped to contiguous road segments can be based on a determination to generate the mapping. For example, when the degree of match of contiguous road segments with lanes is greater than other degrees of match associated with other road segments, the mapping may be desired. In one embodiment, the representation of lanes mapped to contiguous road segments includes a representation of the start and end positions of segments along the lanes, a representation of the identification of at least a portion of the contiguous road segment, and a representation of the start and end positions of at least one road segment.

[0131] Figure 8 is a flow diagram showing a method 800 for generating a lane-road mapping according to some embodiments of the present disclosure. Method 800 includes, at block B802: identifying road segments of a road map associated with a self-machine and lanes of a lane map associated with the self-machine to perform a matching analysis. In some cases, the road segments and lanes are identified based on proximity to the self-machine.

[0132] Method 800 includes, at block B804: determining that a lane is within a predetermined distance of a particular point along a road segment. In some cases, to identify whether the distance condition is met, a vertical span extending from the lane and / or the road segment can be analyzed. For example, a vertical line can be extended from the road segment being analyzed until it intersects the lane being analyzed. The distance from the road to the intersection point can be determined and compared to the predetermined distance. As another example, a vertical span having a distance threshold length can be extended from the road. The vertical span can have any length and / or measure. As an example, the vertical span can be ten meters. If there is an intersection point between the vertical span and the lane, the distance condition can be considered to be met at that particular location.

[0133] Method 800 includes, at block B806: determining that a lane is within a direction threshold of a particular point along a road segment. The direction threshold can be established in any number of ways. For example, the direction threshold can be based on a default setting, a preset setting, or a user-selected setting. Additionally, the direction threshold can be represented in any number of ways. For example, a particular measure (e.g., degrees) can be used to specify the direction threshold for determining a match. In some cases, to identify whether the direction condition is met, a dot product function can be used to determine the similarity of the lane direction at a particular location to the road direction at that particular location.

[0134] Method 800 includes, at block B808: determining whether to analyze subsequent points along a road segment. For example, when a termination event is identified, it can be determined to end the analysis of subsequent points along the road segment. The termination event can be the identification of the end of a lane and / or a road. Another example of a termination event can be the failure to meet a matching condition or the failure to meet a matching condition associated with a predetermined number of consecutive points (e.g., five consecutive matching failures). In the case where another subsequent point along the road segment can be analyzed, the method returns to blocks B804 and B806 to evaluate the matching condition of the next point.

[0135] On the other hand, when it is determined that there are no further subsequent points on the road segment to be analyzed, method 800 proceeds to block B810, where a matching score is generated to indicate the degree of match between the lane and the road segment. The matching score can be based on geometric similarity. For example, given a lane and a road, the geometric similarity between a given segment of the lane and a given segment of the road can correspond to the sum of the reciprocals of the distances between the lane and the road at the sampled locations. To determine the reciprocal of the distance between the lane position and the road, in some cases, a vertical span extending from the road and / or the lane can be used. As an example, the vertical span can extend from a road point until the vertical span intersects the lane or reaches a distance threshold. The intersection distance can be subtracted from the distance threshold to generate the reciprocal of the distance between the lane position and the road.

[0136] Method 800 generates a lane-road mapping based on the matching score at block B812. For example, in some cases, when the matching score is greater than other matching scores associated with other lane-road segment pairs, a lane-road mapping can be generated.

[0137] The systems and methods described herein can be used by, but are not limited to, the following: non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles connected to one or more trailers, airships, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, airplanes, engineering vehicles, trains, submarines, remotely operated vehicles such as drones, and / or other vehicle types. Additionally, the systems and methods described herein can be used for various purposes, such as, but not limited to, for machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, safety and surveillance, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation, and / or digital twins, data center processing, conversational AI, optical transmission simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, and / or any other suitable application.

[0138] The disclosed embodiments can be included in various different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, sensing systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, boating systems, intelligent area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems incorporating one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation for 3D assets, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0139] Example autonomous vehicle

[0140] Figure 9A is an illustration of an example autonomous vehicle 900 in accordance with some embodiments of the present disclosure. The autonomous vehicle 900 (also referred to herein as "vehicle 900") can include, but is not limited to, passenger vehicles such as cars, trucks, buses, ambulances, shuttles, electric or motorized bicycles, motorcycles, fire trucks, police cars, ambulances, boats, construction vehicles, submarines, robotic vehicles, airplanes, drones, vehicles connected to a trailer (e.g., semi-trailer trucks for transporting goods), and / or other types of vehicles (e.g., driverless and / or accommodating one or more passengers). Autonomous vehicles are generally described according to the levels of automation defined by the "Classification and Definitions of Terms Related to Driving Automation Systems for Road Motor Vehicles" of the National Highway Traffic Safety Administration (NHTSA) under the U.S. Department of Transportation and the Society of Automotive Engineers (SAE) (Standard No. J3016-201806 released on June 15, 2018, Standard No. J3016-201609 released on September 30, 2016, and previous and future versions of this standard). The vehicle 900 may be capable of having functions according to one or more of Levels 3-5 of the autonomous driving level. The vehicle 900 may be capable of functioning according to one or more of Levels 1-5 of the autonomous driving level. For example, the vehicle 900 may be capable of providing driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the embodiment. As used herein, the term "autonomous" can include any and / or all types of autonomy of the vehicle 900 or other machines, such as fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, providing assisted autonomy, semi-autonomous, primarily autonomous, or other designations.

[0141] Vehicle 900 may include components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of the vehicle. Vehicle 900 may include a propulsion system 950, such as an internal combustion engine, a hybrid power plant, a fully electric motor, and / or another type of propulsion system. Propulsion system 950 may be connected to the driveline of vehicle 900, which may include a transmission, to effect the propulsion of vehicle 900. Propulsion system 950 may be controlled in response to receiving a signal from throttle / accelerator 952.

[0142] A steering system 954, which may include a steering wheel, may be used to steer vehicle 900 (e.g., along a desired path or route) while the propulsion system 950 is operating (e.g., while the vehicle is in motion). Steering system 954 may receive a signal from a steering actuator 956. For fully automated (Level 5) functionality, the steering wheel may be optional.

[0143] A brake sensor system 946 may be used to operate vehicle brakes in response to receiving a signal from a brake actuator 948 and / or a brake sensor.

[0144] One or more controllers 936, which may include one or more system-on-chips (SoCs) 904 ( Figure 9C ) and / or one or more GPUs, may provide signals (e.g., representing commands) to one or more components and / or systems of vehicle 900. For example, one or more controllers may send signals to operate vehicle brakes via one or more brake actuators 948, operate steering system 954 via one or more steering actuators 956, and operate propulsion system 950 via one or more throttle / accelerators 952. One or more controllers 936 may include one or more on-board (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to effect autonomous driving and / or assist a human driver in driving vehicle 900. One or more controllers 936 may include a first controller 936 for autonomous driving functions, a second controller 936 for functional safety functions, a third controller 936 for artificial intelligence functions (e.g., computer vision), a fourth controller 936 for infotainment functions, a fifth controller 936 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 936 may handle two or more of the above functions, two or more controllers 936 may handle a single function, and / or any combination thereof.

[0145] One or more controllers 936 may provide signals for controlling one or more components and / or systems of vehicle 900 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example and without limitation, a global navigation satellite system sensor 958 (e.g., a global positioning system sensor), a RADAR sensor 960, an ultrasonic sensor 962, a LIDAR sensor 964, an inertial measurement unit (IMU) sensor 966 (e.g., an accelerometer, a gyroscope, a magnetic compass, a magnetometer, etc.), a microphone 996, a stereo camera 968, a wide-angle camera 970 (e.g., a fish-eye camera), an infrared camera 972, a surround camera 974 (e.g., a 360-degree camera), a remote and / or mid-range camera 998, a speed sensor 944 (e.g., for measuring the rate of vehicle 900), a vibration sensor 942, a steering sensor 940, a brake sensor (e.g., as part of a brake sensor system 946), and / or other sensor types.

[0146] One or more of the controllers 936 may receive inputs (e.g., represented by input data) from the instrument cluster 932 of vehicle 900 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 934, an audible annunciator, a speaker, and / or via other components of vehicle 900. These outputs may include information such as vehicle speed, rate, time, map data (e.g., Figure 9C a high-definition (“HD”) map 922), position data (e.g., the position of vehicle 900, e.g., on a map), direction, the positions of other vehicles (e.g., occupancy grids), information about objects and object states as perceived by the controller 936, and so on. For example, the HMI display 934 may display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, exiting 34B in two miles, etc.).

[0147] Vehicle 900 further includes a network interface 924 that may communicate over one or more networks using one or more wireless antennas 926 and / or a modem. For example, network interface 924 may be capable of communicating via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multi-Carrier (“CDMA2000”), etc. One or more wireless antennas 926 may also be used to enable communication between objects (such as vehicles, mobile devices, etc.) in an environment using one or more local area networks such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc. and / or one or more low power wide area networks (LPWAN) such as LoRaWAN, SigFox, etc.

[0148] Figure 9B For an example autonomous vehicle 900 in accordance with some embodiments of the present disclosure for Figure 9A Example camera positions and fields of view of an example autonomous vehicle 900. The cameras and respective fields of view are one example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included, and / or these cameras may be located at different positions on vehicle 900.

[0149] The type of camera used for the cameras may include, but is not limited to, digital cameras that may be adapted to be used with components and / or systems of vehicle 900. The cameras may operate under an Automotive Safety Integrity Level (ASIL) B and / or under another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The cameras may be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a Red Clear Clear Clear (RCCC) color filter array, a Red Clear Clear Blue (RCCB) color filter array, a Red Blue Green Clear (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras such as cameras having RCCC, RCCB, and / or RBGC color filter arrays may be used in an effort to increase light sensitivity.

[0150] In some examples, one or more of the cameras may be used to perform Advanced Driver Assistance System (ADAS) functions (such as as part of a redundant or fail-safe design). For example, a multi-functional monocular camera may be installed to provide functions including lane departure warning, traffic sign assistance, and smart headlight control. One or more of the cameras (such as all of the cameras) may record and provide image data (such as video) simultaneously.

[0151] One or more of the cameras can be mounted in a mounting assembly such as a custom-designed (3D printed) component to cut off stray light and reflections from inside the vehicle (such as reflections from the dashboard reflected in the windshield mirror) that may interfere with the image data capture ability of the camera. Regarding the wing mirror mounting assembly, the wing mirror assembly can be custom 3D printed such that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0152] A camera (e.g., a front camera) having a field of view that includes an environmental portion in front of the vehicle 900 can be used for surround view to help identify the forward path and obstacles and to assist in providing information crucial for generating an occupancy grid and / or determining a preferred vehicle path with the help of one or more controllers 936 and / or a control SoC. The front camera can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. The front camera can also be used for ADAS functions and systems, including lane departure warning ("LDW"), adaptive cruise control ("ACC"), and / or other functions such as traffic sign recognition.

[0153] A variety of cameras can be used in a front-facing configuration, including, for example, a monocular camera platform that includes a complementary metal oxide semiconductor ("CMOS") color imager. Another example can be a wide-angle camera 970, which can be used to sense objects (such as pedestrians, intersection traffic, or bicycles) entering the field of view from the periphery. Although Figure 9B only one wide-angle camera is illustrated, any number (including zero) of wide-angle cameras 970 can be present on the vehicle 900. Additionally, any number of long-range cameras 998 (such as a long-range stereo camera pair) can be used for depth-based object detection, especially for objects for which a neural network has not been trained. The long-range cameras 998 can also be used for object detection and classification and basic object tracking.

[0154] Any number of stereo cameras 968 may also be included in a front-facing configuration. In at least one embodiment, one or more of the stereo cameras 968 may include an integrated control unit that includes a scalable processing unit that may provide a multi-core microprocessor and programmable logic (FPGA) with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit may be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. Alternatively, the stereo camera 968 may include a compact stereo vision sensor that may include two camera lenses (one on the left and one on the right) and an image processing chip that may measure the distance from the vehicle to a target object and activate autonomous emergency braking and lane departure warning functions using the generated information (e.g., metadata). Other types of stereo cameras 968 may be used in addition to or alternatively to those described herein.

[0155] Cameras having a field of view of an environmental portion including the side of the vehicle 900 (e.g., side-view cameras) may be used for surround view, providing information used to create and update an occupancy grid and generate side-impact collision warnings. For example, surround cameras 974 (e.g., the four surround cameras 974 as shown in Figure 9B may be disposed on the vehicle 900. The surround cameras 974 may include wide-angle cameras 970, fish-eye cameras, 360-degree cameras, and / or the like. By way of example, four fish-eye cameras may be disposed on the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle may use three surround cameras 974 (e.g., left, right, and rear) and may utilize one or more other cameras (e.g., a front-facing camera) as a fourth surround camera.

[0156] Cameras having a field of view of an environmental portion including the rear of the vehicle 900 (e.g., rear-view cameras) may be used for assisting with parking, surround view, rear collision warnings, and creating and updating an occupancy grid. A wide variety of cameras may be used, including but not limited to cameras that are also suitable as front-facing cameras as described herein (e.g., long-range and / or mid-range cameras 998, stereo cameras 968, infrared cameras 972, etc.).

[0157] Figure 9C For use in accordance with some embodiments of the present disclosure Figure 9ABlock diagram of an example system architecture of an example autonomous vehicle 900. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, function groupings, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components, and implemented in any suitable combination and location. The various functions described herein as being performed by entities may be implemented by hardware, firmware, and / or software. For example, the various functions may be implemented by a processor executing instructions stored in a memory.

[0158] Figure 9C Each of the components, features, and systems in vehicle 900 is illustrated as being connected via a bus 902. The bus 902 may include a Controller Area Network (CAN) data interface (alternatively referred to herein as the "CAN bus"). The CAN may be a network within vehicle 900 that is used to assist in controlling various features and functions of vehicle 900, such as driving of brakes, acceleration, braking, steering, windshield wipers, and the like. The CAN bus may be configured to have dozens or even hundreds of nodes, each node having its own unique identifier (e.g., CAN ID). The CAN bus may be read to find the steering wheel angle, ground speed, engine revolutions per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0159] Although bus 902 is described herein as a CAN bus, this is not intended to be restrictive. For example, in addition to or instead of the CAN bus, FlexRay and / or Ethernet may be used. Further, although a single line is used to represent bus 902, this is not intended to be restrictive. For example, any number of buses 902 may be present, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 902 may be used to perform different functions, and / or may be used for redundancy. For example, a first bus 902 may be used for collision avoidance functions, and a second bus 902 may be used for drive control. In any example, each bus 902 may communicate with any component of vehicle 900, and two or more buses 902 may communicate with the same component. In some examples, each SoC 904, each controller 936, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors of vehicle 900), and may be connected to a common bus such as the CAN bus.

[0160] Vehicle 900 may include one or more controllers 936, such as those described herein with respect to Figure 9A the controllers described. The controller 936 may be used for a variety of functions. The controller 936 may be coupled to any other different components and systems of the vehicle 900 and may be used for the control of the vehicle 900, the artificial intelligence of the vehicle 900, the infotainment for the vehicle 900, and / or the like.

[0161] Vehicle 900 may include one or more system-on-chips (SoCs) 904. The SoC 904 may include a CPU 906, a GPU 908, a processor 910, a cache 912, an accelerator 914, a data store 916, and / or other components and features not shown. In a variety of platforms and systems, the SoC 904 may be used to control the vehicle 900. For example, one or more SoCs 904 may be combined with an HD map 922 in a system (such as the system of the vehicle 900), and the HD map may obtain map refreshes and / or updates from one or more servers (such as Figure 9D one or more servers 978) via a network interface 924.

[0162] The CPU 906 may include a CPU cluster or a CPU complex (alternatively, referred to herein as "CCPLEX"). The CPU 906 may include multiple cores and / or an L2 cache. For example, in some embodiments, the CPU 906 may include eight cores in a coherent multi-processor configuration. In some embodiments, the CPU 906 may include four dual-core clusters, each of which has a dedicated L2 cache (e.g., a 2MB L2 cache). The CPU 906 (e.g., CCPLEX) may be configured to support simultaneous cluster operation such that any combination of the clusters of the CPU 906 can be active at any given time.

[0163] The CPU 906 may implement power management capabilities including one or more of the following features: each hardware block may automatically perform clock gating when idle to save dynamic power; each core clock may be gated when the core is not actively executing instructions due to the execution of WFI / WFE instructions; each core may be independently power gated; when all cores are clock gated or power gated, each core cluster may be independently clock gated; and / or when all cores are power gated, each core cluster may be independently power gated. The CPU 906 may further implement enhanced algorithms for managing power states, where allowed power states and desired wake-up times are specified, and the hardware / microcode determines the best power state for the cores, clusters, and CCPLEX to enter. The processing cores may support a simplified power state entry sequence in software, and this work is offloaded to the microcode.

[0164] The GPU 908 may include an integrated GPU (alternatively referred to herein as an “iGPU”). The GPU 908 may be programmable and efficient for parallel workloads. In some examples, the GPU 908 may use enhanced tensor instruction sets. The GPU 908 may include one or more streaming microprocessors, where each streaming microprocessor may include an L1 cache (e.g., an L1 cache with at least 96 KB of storage capacity), and two or more of these streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512 KB of storage capacity). In some embodiments, the GPU 908 may include at least eight streaming microprocessors. The GPU 908 may use a compute application programming interface (API). Additionally, the GPU 908 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0165] In automotive and embedded use cases, the GPU 908 may be power optimized for optimal performance. For example, the GPU 908 may be fabricated on fin field-effect transistors (FinFETs). However, this is not intended to be limiting, and the GPU 908 may be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor may incorporate a number of mixed-precision processing cores partitioned into multiple blocks. For example and without limitation, 64 PF32 cores and 32 PF64 cores may be partitioned into four processing blocks. In such an example, each processing block may be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA tensor cores for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64 KB register file. Additionally, the streaming microprocessor may include independent parallel integer and floating-point data paths to provide efficient execution of workloads leveraging a mix of computation and addressing computations. The streaming microprocessor may include independent thread scheduling capabilities to allow for finer-grained synchronization and cooperation between parallel threads. The streaming microprocessor may include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

[0166] The GPU 908 may include a high-bandwidth memory (HBM) and / or a 16 GB HBM2 memory subsystem that provides a peak memory bandwidth of approximately 900 GB / s in some examples. In some examples, in addition to or alternatively to HBM memory, synchronous graphics random access memory (SGRAM), such as fifth-generation graphics double data rate synchronous random access memory (GDDR5), may be used.

[0167] The GPU 908 may include unified memory technology that includes access counters to allow memory pages to be migrated more precisely to the processors that most frequently access them, thereby improving the efficiency of the memory ranges shared among the processors. In some examples, address translation service (ATS) support may be used to allow the GPU 908 to directly access the CPU 906 page tables. In such examples, when the GPU 908 memory management unit (MMU) experiences a miss, an address translation request may be transmitted to the CPU 906. In response, the CPU 906 may look up the virtual-physical mapping for the address in its page table and transmit the translation back to the GPU 908. In this way, the unified memory technology may allow a single unified virtual address space for the memory of both the CPU 906 and the GPU 908, thereby simplifying GPU 908 programming and porting applications to the GPU 908.

[0168] In addition, the GPU 908 may include access counters that may track how frequently the GPU 908 accesses the memory of other processors. The access counters may help ensure that memory pages are moved to the physical memory of the processor that most frequently accesses those pages.

[0169] The SoC 904 may include any number of caches 912, including those described herein. For example, the cache 912 may include an L3 cache available to both the CPU 906 and the GPU 908 (e.g., that is connected to both the CPU 906 and the GPU 908). The cache 912 may include a write-back cache that may track the state of lines, for example, by using a cache coherence protocol (such as MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache may include 4MB or more, but smaller cache sizes may also be used.

[0170] The SoC 904 may include an arithmetic logic unit (ALU) that may be used to perform processing for any of the various tasks or operations regarding the vehicle 900 - such as processing a DNN. Additionally, the SoC 904 may include a floating-point unit (FPU) - or other math co-processor or digital co-processor type - for performing mathematical operations within the system. For example, the SoC 904 may include one or more FPUs integrated within the execution units of the CPU 906 and / or the GPU 908.

[0171] The SoC 904 may include one or more accelerators 914 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC 904 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memories. The large on-chip memory (e.g., 4MB SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster may be used to supplement the GPU 908 and offload some of the tasks of the GPU 908 (e.g., freeing up more cycles of the GPU 908 for performing other tasks). As an example, the accelerator 914 may be used for targeted workloads that are stable enough to be easily accelerated (e.g., perception, convolutional neural network (CNN), etc.). When used herein, the term "CNN" may include all types of CNNs, including region-based or region convolutional neural networks (RCNN) and fast RCNN (e.g., for object detection).

[0172] The accelerator 914 (e.g., the hardware acceleration cluster) may include a deep learning accelerator (DLA). The DLA may include one or more tensor processing units (TPUs) that may be configured to provide an additional one trillion operations per second for deep learning applications and inference. The TPU may be an accelerator configured to perform image processing functions (e.g., for CNN, RCNN, etc.) and optimized for performing image processing functions. The DLA may be further optimized for a specific set of neural network types and floating-point operations and inference. The design of the DLA may provide higher performance per millimeter than a general-purpose GPU and far exceed the performance of the CPU. The TPU may perform several functions, including single-instance convolution functions, supporting INT8, INT16, and FP16 data types for both features and weights, and post-processor functions.

[0173] The DLA may quickly and efficiently execute neural networks, especially CNNs, for any of a variety of functions on processed or unprocessed data, such as, but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and identification and detection using data from microphones; CNNs for face recognition and vehicle owner recognition using data from camera sensors; and / or CNNs for security and / or safety-related events.

[0174] The DLA may perform any function of the GPU 908, and by using inference accelerators, for example, the designer may target the DLA or the GPU 908 for any function. For example, the designer may focus the processing and floating-point operations of the CNN on the DLA and leave other functions to the GPU 908 and / or other accelerators 914.

[0175] The accelerator 914 (e.g., a hardware acceleration cluster) may include a programmable vision accelerator (PVA), which may alternatively be referred to herein as a computer vision accelerator. The PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA may provide a balance between performance and flexibility. For example, each PVA may include, for example and without limitation, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0176] The RISC cores may interact with an image sensor (e.g., the image sensor of any of the cameras described herein), an image signal processor, and / or the like. Each of these RISC cores may include any number of memories. Depending on the embodiment, the RISC cores may use any of several protocols. In some examples, the RISC cores may execute a real-time operating system (RTOS). The RISC cores may be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and / or storage devices. For example, the RISC cores may include an instruction cache and / or tightly coupled RAM.

[0177] The DMA may enable the components of the PVA to access system memory independently of the CPU 906. The DMA may support any number of features used to provide optimizations to the PVA, including but not limited to supporting multi-dimensional addressing and / or circular addressing. In some examples, the DMA may support addressing up to six or more dimensions, which may include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0178] The vector processor may be a programmable processor that may be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as, for example, a single instruction multiple data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW may enhance throughput and rate.

[0179] Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. As a result, in some examples, each of the vector processors may be configured to execute independently of the other vector processors. In other examples, the vector processors included in a particular PVA may be configured to employ data parallelization. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA may simultaneously execute different computer vision algorithms on the same image, or even execute different algorithms on sequential images or portions of an image. Among other things, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each of these PVAs. Additionally, a PVA may include additional error correction code (ECC) memory to enhance overall system security.

[0180] The accelerator 914 (e.g., a hardware acceleration cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for the accelerator 914. In some examples, the on-chip memory may include at least 4MB SRAM composed of, for example and without limitation, eight field-configurable memory blocks, which may be accessed by both the PVA and the DLA. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and the DLA may access the memory via a backbone that provides high-speed memory access to the PVA and the DLA. The backbone may include an on-chip computer vision network (e.g., using APB) that interconnects the PVA and the DLA to the memory.

[0181] The on-chip computer vision network may include an interface that determines that both the PVA and the DLA provide ready and valid signals before transmitting any control signals / address / data. Such an interface may provide separate phases and separate channels for transmitting control signals / address / data, as well as burst communication for continuous data transfer. This type of interface may comply with the ISO 26262 or IEC 61508 standards, but other standards and protocols may also be used.

[0182] In some examples, the SoC 904 can include a real-time ray tracing hardware accelerator as described, for example, in U.S. Patent Application No. 16 / 101,232, filed on August 10, 2018. The real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the position and extent of objects (e.g., within a world model) in order to generate a real-time visualization simulation for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for SONAR system simulation, for general wave propagation simulation, for comparison with LIDAR data for positioning and / or other functional purposes, and / or for other uses. In some embodiments, one or more tree traversal units (TTUs) can be used to perform one or more ray tracing-related operations.

[0183] The accelerator 914 (e.g., a hardware accelerator cluster) has a wide range of autonomous driving applications. The PVA can be a programmable vision accelerator that can be used in key processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are a good match for algorithm domains that require predictable processing, low power, and low latency. In other words, the PVA performs well in semi-dense or dense regular computations, even on small data sets that require predictable runtimes with low latency and low power. Thus, in the context of a platform for autonomous vehicles, the PVA is designed to run classical computer vision algorithms because they are effective in object detection and integer math operations.

[0184] For example, according to one embodiment of the technology, the PVA is used to perform computer stereo vision. In some examples, an algorithm based on semi-global matching can be used, but this is not intended to be limiting. Many applications for level 3-5 autonomous driving require instantaneous motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.). The PVA can perform computer stereo vision functions on inputs from two monocular cameras.

[0185] In some examples, the PVA can be used to perform dense optical flow. Process raw RADAR data (e.g., using a 4D fast Fourier transform) to provide processed RADAR. In other examples, the PVA is used for time-of-flight depth processing, which, for example, processes raw time-of-flight data to provide processed time-of-flight data.

[0186] DLA can be used to run any type of network to enhance control and driving safety, including, for example, a neural network that outputs a confidence metric for each object detection. Such confidence values can be interpreted as probabilities or as providing a relative "weight" for each detection compared to other detections. The confidence value enables the system to make further decisions regarding which detections should be considered true positive detections rather than false positive detections. For example, the system can set a threshold for the confidence and consider only detections that exceed the threshold as true positive detections. In an automatic emergency braking (AEB) system, false positive detections can cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. DLA can run a neural network for regressing confidence values. The neural network can take as its input at least some subset of parameters, such as bounding box dimensions, a ground plane estimate obtained (e.g., from another subsystem), the output of an inertial measurement unit (IMU) sensor 966 related to the orientation and distance of the vehicle 900, a 3D position estimate of an object obtained from a neural network and / or other sensors (such as a LIDAR sensor 964 or a RADAR sensor 960), etc.

[0187] The SoC 904 can include one or more data stores 916 (e.g., memories). The data store 916 can be an on-chip memory of the SoC 904, which can store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and safety, the data store 916 can be large enough in capacity to store multiple instances of the neural network. The data store 916 can include an L2 or L3 cache 912. References to the data store 916 can include references to memories associated with PVAs, DLAs, and / or other accelerators 914 as described herein.

[0188] The SoC 904 may include one or more processors 910 (e.g., embedded processors). The processor 910 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management functions as well as security implementation related. The boot and power management processor may be part of the SoC 904 boot sequence and may provide runtime power management services. The boot power and management processor may provide clock and voltage programming, assist in system low power state transitions, manage the SoC 904 heat and temperature sensors, and / or manage the SoC 904 power state. Each temperature sensor may be implemented as a ring oscillator, whose output frequency is proportional to temperature, and the SoC 904 may use the ring oscillator to detect the temperature of the CPU 906, GPU 908, and / or accelerator 914. If it is determined that the temperature exceeds a threshold, then the boot and power management processor may enter a temperature fault routine and place the SoC 904 in a lower power state and / or place the vehicle 900 in a driver safety stop mode (e.g., safely stop the vehicle 900).

[0189] The processor 910 may further include a set of embedded processors that can be used as an audio processing engine. The audio processing engine may be an audio subsystem that allows for full hardware support for multi-channel audio over multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor with dedicated RAM.

[0190] The processor 910 may further include an always-on processor engine, which may provide the necessary hardware features to support low power sensor management and wake-up use cases. The always-on processor engine may include a processor core, tightly coupled RAM, support peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0191] The processor 910 may further include a security cluster engine, which includes a dedicated processor subsystem for handling the security management of automotive applications. The security cluster engine may include two or more processor cores, tightly coupled RAM, support peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In the secure mode, the two or more cores may operate in a lockstep mode and act as a single core with comparison logic for detecting any differences between their operations.

[0192] The processor 910 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0193] The processor 910 may further include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0194] The processor 910 may include a video image compositor that may be a processing block (e.g., implemented on a microprocessor) that implements the video post-processing functions required for a video playback application to generate the final image for a player window. The video image compositor may perform lens distortion correction on the wide-angle camera 970, the surround camera 974, and / or the in-cab monitoring camera sensor. The in-cab monitoring camera sensor is preferably monitored by a neural network running on another instance of the advanced SoC, configured to identify in-cab events and respond accordingly. The in-cab system may perform lip reading to activate mobile phone services and make calls, dictate emails, change the vehicle destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode and are disabled otherwise.

[0195] The video image compositor may include enhanced temporal noise reduction for spatial and temporal noise reduction. For example, in the case of motion in the video, the noise reduction appropriately weights the spatial information, reducing the weight of the information provided by neighboring frames. In the case where an image or a portion of the image does not include motion, the temporal noise reduction performed by the video image compositor may use information from a previous image to reduce the noise in the current image.

[0196] The video image compositor may also be configured to perform stereo correction on input stereo lens frames. When the operating system desktop is in use and the GPU 908 does not need to continuously render new surfaces, the video image compositor may be further used for user interface composition. Even when the GPU 908 is powered on and active for 3D rendering, the video image compositor may be used to relieve the burden on the GPU 908 to improve performance and responsiveness.

[0197] The SoC 904 may further include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block for receiving video and inputs from cameras and may be used for camera and related pixel input functions. The SoC 904 may further include an input / output controller that may be software-controlled and may be used to receive I / O signals not committed to a specific role.

[0198] The SoC 904 may further include a wide range of peripheral device interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. The SoC 904 can be used to process data from cameras (via Gigabit Multimedia Serial Link and Ethernet connections), sensors (such as LIDAR sensor 964, RADAR sensor 960, etc. that can be connected via Ethernet), data from bus 902 (such as the speed of vehicle 900, steering wheel position, etc.), and data from GNSS sensor 958 (connected via Ethernet or CAN bus). The SoC 904 may further include dedicated high-performance large-capacity storage controllers, which may include their own DMA engines and can be used to free the CPU 906 from routine data management tasks.

[0199] The SoC 904 can be an end-to-end platform with a flexible architecture that spans levels 3 - 5 of automation, thus providing an integrated functional safety architecture for a platform that utilizes and efficiently uses computer vision and ADAS technologies to achieve diversity and redundancy, along with deep learning tools to provide a flexible and reliable driving software stack. The SoC 904 can be faster, more reliable, and even more energy-efficient and space-efficient than conventional systems. For example, when combined with the CPU 906, GPU 908, and data storage 916, the accelerator 914 can provide a fast and efficient platform for level 3 - 5 autonomous vehicles.

[0200] Thus, this technology provides capabilities and functions that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on CPUs that can be configured using high-level programming languages such as the C programming language to perform various processing algorithms across a variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for in-vehicle ADAS applications and for practical level 3 - 5 autonomous vehicles.

[0201] In contrast to conventional systems, the technology described herein allows multiple neural networks to be executed simultaneously and / or sequentially by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, and combining the results to achieve level 3 - 5 autonomous driving functions. For example, a CNN executed on the DLA or a dGPU (such as GPU 920) can include text and word recognition, allowing a supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA may further include a neural network capable of recognizing, interpreting, and providing semantic understanding of the signs and passing that semantic understanding to a path planning module running on the CPU complex.

[0202] As another example, multiple neural networks can operate simultaneously, such as required for level 3, 4, or 5 driving. For example, a warning sign consisting of "Caution: Flashing lights indicate icy conditions" together with the electric lights can be interpreted independently or jointly by several neural networks. The sign itself can be recognized as a traffic sign by a first neural network deployed (e.g., a trained neural network), and the text "Flashing lights indicate icy conditions" can be interpreted by a second neural network deployed, which informs the vehicle's path planning software (preferably executed on the CPU complex) that when the flashing lights are detected, there are icy conditions. The flashing lights can be recognized by operating a third neural network deployed over multiple frames, which informs the vehicle's path planning software of the presence (or absence) of the flashing lights. All three neural networks can operate simultaneously, for example, within the DLA and / or on the GPU 908.

[0203] In some examples, the CNNs for face recognition and vehicle owner recognition can use data from the camera sensors to identify the presence of an authorized driver and / or vehicle owner of the vehicle 900. A processing engine always on the sensor can be used to unlock the vehicle and turn on the lights when the vehicle owner approaches the driver's door, and in a security mode, to disable the vehicle when the vehicle owner leaves the vehicle. In this way, the SoC 904 provides security against theft and / or carjacking.

[0204] In another example, the CNN for emergency vehicle detection and recognition can use data from the microphone 996 to detect and recognize an emergency vehicle siren. In contrast to conventional systems that use a general classifier to detect the siren and manually extract features, the SoC 904 uses a CNN to classify environmental and urban sounds as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to recognize the relative closing rate of an emergency vehicle (e.g., by using the Doppler effect). The CNN can also be trained to recognize emergency vehicles specific to the local area in which the vehicle is operating as recognized by the GNSS sensor 958. Thus, for example, when operating in Europe, the CNN will seek to detect European sirens, and when in the United States, the CNN will seek to recognize only North American sirens. Once an emergency vehicle is detected, with the assistance of the ultrasonic sensor 962, a control program can be used to execute an emergency vehicle safety routine to slow down the vehicle, pull over to the side of the road, stop the vehicle, and / or idle the vehicle until the emergency vehicle passes.

[0205] The vehicle may include a CPU 918 (e.g., a discrete CPU or dCPU) that may be coupled to the SoC 904 via a high-speed interconnect (e.g., PCIe). The CPU 918 may include, for example, an X86 processor. The CPU 918 may be used to perform any of a variety of functions, including, for example, arbitrating potentially inconsistent results between ADAS sensors and the SoC 904, and / or monitoring the status and health of the controller 936 and / or the infotainment SoC 930.

[0206] The vehicle 900 may include a GPU 920 (e.g., a discrete GPU or dGPU) that may be coupled to the SoC 904 via a high-speed interconnect (e.g., NVIDIA's NVLINK). The GPU 920 may provide additional artificial intelligence capabilities, for example, by executing redundant and / or different neural networks, and may be used to train and / or update neural networks based on inputs (e.g., sensor data) from sensors of the vehicle 900.

[0207] The vehicle 900 may further include a network interface 924, which may include one or more wireless antennas 926 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 924 may be used to enable wireless connections to the cloud (e.g., to the server 978 and / or other network devices), to other vehicles, and / or to computing devices (e.g., the client devices of passengers) via the Internet. To communicate with other vehicles, a direct link may be established between the two vehicles, and / or an indirect link (e.g., across a network and via the Internet) may be established. The direct link may be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link may provide the vehicle 900 with information about vehicles approaching the vehicle 900 (e.g., vehicles in front of, to the side of, and / or behind the vehicle 900). This function may be part of the cooperative adaptive cruise control function of the vehicle 900.

[0208] The network interface 924 may include an SoC that provides modulation and demodulation functions and enables the controller 936 to communicate via a wireless network. The network interface 924 may include a radio frequency front end for upconversion from baseband to radio frequency and downconversion from radio frequency to baseband. The frequency conversion may be performed by a known process, and / or may be performed using a super-heterodyne process. In some examples, the radio frequency front end functions may be provided by a separate chip. The network interface may include wireless capabilities for communicating via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0209] Vehicle 900 may further include a data store 928 that may include off-chip (e.g., outside of SoC 904) storage devices. The data store 928 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disks, and / or other components and / or devices that can store at least one bit of data.

[0210] Vehicle 900 may further include a GNSS sensor 958. The GNSS sensor 958 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used to assist mapping, perception, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 958 may be used, including, for example and without limitation, a GPS using a USB connector with an Ethernet to serial (RS-232) bridge.

[0211] Vehicle 900 may further include a RADAR sensor 960. The RADAR sensor 960 may be used by vehicle 900 for remote vehicle detection even in dark and / or adverse weather conditions. The RADAR functional safety level may be ASIL B. The RADAR sensor 960 may use CAN and / or bus 902 (e.g., to transmit data generated by the RADAR sensor 960) for control and access to object tracking data and, in some examples, access Ethernet to access raw data. A variety of RADAR sensor types may be used. For example and without limitation, the RADAR sensor 960 may be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor is used.

[0212] The RADAR sensor 960 may include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, and so on. In some examples, long-range RADAR may be used for adaptive cruise control functions. The long-range RADAR system may provide a wide field of view (e.g., within 250 m) achieved through two or more independent scans. The RADAR sensor 960 may help distinguish between static and moving objects and may be used by the ADAS system for emergency braking assistance and forward collision warning. The long-range RADAR sensor may include a single-station multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the central four antennas may create a focused beam pattern that is designed to record the surroundings of vehicle 900 at a higher rate with minimal traffic interference from adjacent lanes. The other two antennas may extend the field of view, making it possible to quickly detect vehicles entering or leaving the lane of vehicle 900.

[0213] As an example, a mid-range RADAR system can include a range of up to 1260 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 1250 degrees (rear). A short-range RADAR system can include, but is not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor the rear and the blind spots alongside the vehicle.

[0214] The short-range RADAR system can be used in an ADAS system for blind spot detection and / or lane change assistance.

[0215] Vehicle 900 can further include ultrasonic sensors 962. Ultrasonic sensors 962 that can be placed at the front, rear, and / or sides of vehicle 900 can be used for parking assistance and / or creating and updating an occupancy grid. A variety of ultrasonic sensors 962 can be used, and different ultrasonic sensors 962 can be used for different detection ranges (e.g., 2.5 m, 4 m). Ultrasonic sensors 962 can operate at ASIL B for functional safety levels.

[0216] Vehicle 900 can include a LIDAR sensor 964. The LIDAR sensor 964 can be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensor 964 can be at ASIL B for functional safety levels. In some examples, vehicle 900 can include multiple LIDAR sensors 964 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a gigabit Ethernet switch).

[0217] In some examples, the LIDAR sensor 964 may be able to provide a list of objects and their distances for a 360-degree field of view. Commercially available LIDAR sensors 964 can have an advertised range of, for example, approximately 1200 m, an accuracy of 2 cm - 3 cm, and support a 1200 Mbps Ethernet connection. In some examples, one or more non-protruding LIDAR sensors 964 can be used. In such examples, the LIDAR sensor 964 can be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of vehicle 900. In such examples, the LIDAR sensor 964 can provide a field of view of up to 120 degrees horizontally and 35 degrees vertically, with a range of 200 m, even for low-reflectivity objects. The front-mounted LIDAR sensor 964 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0218] In some examples, LIDAR technologies such as 3D flash LIDAR can also be used. 3D flash LIDAR uses the flash of a laser as the emission source to illuminate the surrounding environment of the vehicle up to about 200m. The flash LIDAR unit includes a receiver that records the laser pulse transmission time and the reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LIDAR can allow for the generation of highly accurate and distortion-free images of the surrounding environment using each laser flash. In some examples, four flash LIDAR sensors can be deployed, one on each side of the vehicle 900. Available 3D flash LIDAR systems include solid-state 3D staring array LIDAR cameras (e.g., non-scanning LIDAR devices) that have no moving parts other than a fan. The flash LIDAR device can use class I (eye-safe) laser pulses of 5 nanoseconds per frame and can capture the reflected laser in the form of 3D range point clouds and co-registered intensity data. By using flash LIDAR and because flash LIDAR is a solid-state device with no moving parts, the LIDAR sensor 964 can be less susceptible to motion blur, vibration, and / or shock.

[0219] The vehicle can further include an IMU sensor 966. In some examples, the IMU sensor 966 can be located at the center of the rear axle of the vehicle 900. The IMU sensor 966 can include, for example and without limitation, accelerometers, magnetometers, gyroscopes, magnetic compasses, and / or other sensor types. In some examples, such as in six-axis applications, the IMU sensor 966 can include an accelerometer and a gyroscope, while in nine-axis applications, the IMU sensor 966 can include an accelerometer, a gyroscope, and a magnetometer.

[0220] In some embodiments, the IMU sensor 966 can be implemented as a miniature high-performance GPS-aided inertial navigation system (GPS / INS) that combines microelectromechanical system (MEMS) inertial sensors, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 966 can enable the vehicle 900 to estimate the heading by directly observing the speed changes from the GPS to the IMU sensor 966 and correlating them without the need for input from a magnetic sensor. In some examples, the IMU sensor 966 and the GNSS sensor 958 can be integrated into a single unit.

[0221] The vehicle can include a microphone 996 placed in and / or around the vehicle 900. Among other things, the microphone 996 can be used for emergency vehicle detection and identification.

[0222] The vehicle may further include any number of camera types, including a stereo camera 968, a wide-angle camera 970, an infrared camera 972, a surround camera 974, a long-range and / or mid-range camera 998, and / or other camera types. These cameras can be used to capture image data around the entire periphery of the vehicle 900. The camera types used depend on the embodiment and the requirements of the vehicle 900, and any combination of camera types can be used to provide the necessary coverage around the vehicle 900. Additionally, the number of cameras can vary according to the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As an example and without limitation, these cameras can support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described in more detail herein with respect to Figure 9A and Figure 9B is described in more detail.

[0223] The vehicle 900 may further include a vibration sensor 942. The vibration sensor 942 can measure the vibration of components of the vehicle such as an axle. For example, a change in vibration can indicate a change in the road surface. In another example, when two or more vibration sensors 942 are used, the difference between the vibrations can be used to determine the friction or slip of the road surface (e.g., when there is a vibration difference between a powered drive axle and a free-spinning axle).

[0224] The vehicle 900 may include an ADAS system 938. In some examples, the ADAS system 938 may include a SoC. The ADAS system 938 may include autonomous / adaptive / auto cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0225] The ACC system can use RADAR sensors 960, LIDAR sensors 964, and / or cameras. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately in front of the vehicle 900 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Lateral ACC performs distance keeping and, when necessary, advises the vehicle 900 to change lanes. Lateral ACC is related to other ADAS applications such as LCA and CWS.

[0226] The CACC uses information from other vehicles, which can be received indirectly from other vehicles via the network interface 924 and / or the wireless antenna 926 via a wireless link or through a network connection (e.g., via the Internet). The direct link can be provided by a vehicle-to-vehicle (V2V) communication link, while the indirect link can be an infrastructure-to-vehicle (I2V) communication link. Generally, the V2V communication concept provides information about the immediately preceding vehicle (e.g., the vehicle immediately in front of vehicle 900 and in the same lane as it), while the I2V communication concept provides information about traffic further ahead. The CACC system can include either or both of the I2V and V2V information sources. Given the information of the vehicle in front of vehicle 900, the CACC can be more reliable, and it has the potential to improve the smoothness of traffic flow and reduce road congestion.

[0227] The FCW system is designed to alert the driver of a hazard so that the driver can take corrective action. The FCW system uses a front camera and / or RADAR sensor 960 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibration component. The FCW system can provide warnings in the form of, for example, sound, visual warnings, vibration, and / or rapid braking pulses.

[0228] The AEB system detects an impending front collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system can use a front camera and / or RADAR sensor 960 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid the collision, and if the driver does not take corrective action, then the AEB system can automatically apply the brakes in an effort to prevent or at least mitigate the impact of the predicted collision. The AEB system can include technologies such as dynamic brake support and / or collision imminent braking.

[0229] The LDW system provides visual, auditory, and / or tactile warnings such as steering wheel or seat vibration to alert the driver when vehicle 900 crosses a lane marking. When the driver indicates an intentional lane departure by activating the turn signal, the LDW system is not activated. The LDW system can use a front-side-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibration component.

[0230] The LKA system is a variant of the LDW system. If the vehicle 900 starts to leave the lane, then the LKA system provides a steering input or braking to correct the vehicle 900.

[0231] The BSW system detects and warns the driver of vehicles in the blind spots of the vehicle. The BSW system can provide visual, audible, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses the turn signal. The BSW system can use a rear-facing camera and / or RADAR sensor 960 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0232] The RCTW system can provide visual, audible, and / or tactile notifications when an object is detected outside the rear camera range while the vehicle 900 is in reverse. Some RCTW systems include AEB to ensure that the vehicle brakes are applied to avoid a crash. The RCTW system can use one or more rear RADAR sensors 960 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0233] Conventional ADAS systems may be prone to false positive results, which can be annoying and distracting to the driver, but are typically not catastrophic because the ADAS system alerts the driver and allows the driver to decide whether a safe condition actually exists and act accordingly. However, in an autonomous vehicle 900, in the case of conflicting results, the vehicle 900 itself must decide whether to heed the results from the main computer or an auxiliary computer (e.g., the first controller 936 or the second controller 936). For example, in some embodiments, the ADAS system 938 can be a backup and / or auxiliary computer for providing perception information to a backup computer rationality module. The backup computer rationality monitor can run redundant diverse software on hardware components to detect faults in perception and dynamic driving tasks. The output from the ADAS system 938 can be provided to the supervisory MCU. If the outputs from the main computer and the auxiliary computer conflict, then the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0234] In some examples, the host computer may be configured to provide a confidence score to the supervisory MCU indicating the host computer's confidence in the selected result. If the confidence score exceeds a threshold, then the supervisory MCU may follow the host computer's direction regardless of whether the secondary computer provides conflicting or inconsistent results. In cases where the confidence score does not meet the threshold and where the host computer and the secondary computer indicate different results (e.g., conflict), the supervisory MCU may arbitrate between these computers to determine an appropriate result.

[0235] The supervisory MCU may be configured to run a neural network that is trained and configured to determine conditions under which the secondary computer provides false alarms based on outputs from the host computer and the secondary computer. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer can be trusted and when it cannot. For example, when the secondary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying metal objects that are not in fact dangerous, such as a drain grate or manhole cover that triggers an alarm. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to disregard the LDW when a cyclist or pedestrian is present and lane departure is actually the safest strategy. In embodiments including a neural network running on the supervisory MCU, the supervisory MCU may include at least one of a DLA or a GPU suitable for running the neural network with an associated memory. In a preferred embodiment, the supervisory MCU may include components of the SoC 904 and / or be included as a component of the SoC 904.

[0236] In other examples, the ADAS system 938 may include a secondary computer that performs ADAS functions using traditional computer vision rules. In this way, the secondary computer can use classical computer vision rules (if - then), and the presence of a neural network in the supervisory MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identity make the overall system more fault-tolerant, especially for faults caused by software (or software-hardware interface) functions. For example, if there is a software vulnerability or error in the software running on the host computer and the non-identical software code running on the secondary computer provides the same overall result, then the supervisory MCU can be more confident that the overall result is correct and that the vulnerability in the software or hardware on the host computer does not cause a substantial error.

[0237] In some examples, the output of the ADAS system 938 can be fed to the perception block of the main computer and / or the dynamic driving task block of the main computer. For example, if the ADAS system 938 indicates a forward collision warning due to an object being immediately in front, the perception block can use this information when identifying the object. In other examples, the auxiliary computer can have its own neural network that is trained and thus reduces the risk of false positives as described herein.

[0238] The vehicle 900 can further include an infotainment SoC 930 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system can not be an SoC and can include two or more discrete components. The infotainment SoC 930 can include a combination of hardware and software that can be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation system, rear parking assistance, radio data system, vehicle-related information such as fuel level, total distance covered, brake fuel level, oil level, door open / close, air filter information, etc.) to the vehicle 900. For example, the infotainment SoC 930 can include a radio, a disc player, a navigation system, a video player, USB and Bluetooth connectivity, an in-vehicle computer, in-vehicle entertainment, Wi-Fi, steering wheel audio controls, hands-free voice controls, a head-up display (HUD), an HMI display 934, a telematics device, a control panel (e.g., for controlling various components, features, and / or systems, and / or interacting therewith), and / or other components. The infotainment SoC 930 can further be used to provide information (e.g., visual and / or auditory) to the vehicle's user, such as information from the ADAS system 938, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0239] The infotainment SoC 930 can include GPU functionality. The infotainment SoC 930 can communicate with other devices, systems, and / or components of the vehicle 900 via a bus 902 (e.g., a CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 930 can be coupled to a supervisory MCU such that in the event of a failure of the main controller 936 (e.g., the main and / or backup computer of the vehicle 900), the GPU of the infotainment system can perform some autonomous driving functions. In such examples, the infotainment SoC 930 can place the vehicle 900 in a driver safe parking mode as described herein.

[0240] Vehicle 900 may further include an instrument cluster 932 (such as a digital instrument panel, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 932 may include a controller and / or a supercomputer (such as a discrete controller or supercomputer). The instrument cluster 932 may include a set of instruments, such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seat belt warning light, parking brake warning light, engine malfunction light, airbag (SRS) system information, lighting controls, safety system controls, navigation information, and so on. In some examples, information may be displayed and / or shared between the infotainment SoC 930 and the instrument cluster 932. In other words, the instrument cluster 932 may be included as part of the infotainment SoC 930, or vice versa.

[0241] Figure 9D A system schematic diagram for communication between a cloud-based server and Figure 9A an example autonomous vehicle 900 according to some embodiments of the present disclosure. The system 976 may include a server 978, a network 990, and vehicles including the vehicle 900. The server 978 may include a plurality of GPUs 984(A)-984(H) (collectively referred to herein as GPUs 984), PCIe switches 982(A)-982(H) (collectively referred to herein as PCIe switches 982), and / or CPUs 980(A)-980(B) (collectively referred to herein as CPUs 980). The GPUs 984, CPUs 980, and PCIe switches may be interconnected by high-speed interconnects and / or PCIe connections 986 such as, for example, and without limitation, the NVLink interface 988 developed by NVIDIA. In some examples, the GPUs 984 are connected via NVLink and / or an NVSwitch SoC, and the GPUs 984 and the PCIe switches 982 are connected via a PCIe interconnect. Although eight GPUs 984, two CPUs 980, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each of the servers 978 may include any number of GPUs 984, CPUs 980, and / or PCIe switches. For example, each of the servers 978 may include eight, sixteen, thirty-two, and / or more GPUs 984.

[0242] Server 978 can receive image data from a vehicle via network 990, where the image data represents an image showing an unexpected or changed road condition such as a recently started roadwork. Server 978 can transmit neural network 992, updated neural network 992, and / or map information 994, including information about traffic and road conditions, to the vehicle via network 990. Updates to the map information 994 can include updates to the HD map 922, such as information about construction sites, potholes, curves, floods, or other obstacles. In some examples, neural network 992, updated neural network 992, and / or map information 994 can be represented and / or generated based on data from new training and / or data received from any number of vehicles in the environment and / or experience from training performed at a data center (e.g., using server 978 and / or other servers).

[0243] Server 978 can be used to train a machine learning model (e.g., a neural network) based on training data. The training data can be generated by vehicles and / or can be generated in a simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., in cases where the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not labeled and / or preprocessed (e.g., in cases where the neural network does not require supervised learning). Training can be performed according to any one or more categories of machine learning techniques, including but not limited to: categories such as supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variants or combinations thereof. Once the machine learning model is trained, the machine learning model can be used by the vehicle (e.g., transmitted to the vehicle via network 990), and / or the machine learning model can be used by server 978 to remotely monitor the vehicle.

[0244] In some examples, server 978 can receive data from a vehicle and apply the data to the latest real-time neural network for real-time intelligent inference. Server 978 can include a deep learning supercomputer powered by GPU 984 and / or a dedicated AI computer, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, server 978 can include a deep learning infrastructure of a data center powered only by a CPU.

[0245] The deep learning infrastructure of server 978 may be capable of fast real-time inference and can use this ability to evaluate and verify the health of the processors, software, and / or associated hardware in vehicle 900. For example, the deep learning infrastructure can receive periodic updates from vehicle 900, such as an image sequence and / or objects located in that image sequence that vehicle 900 has identified (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify the objects and compare them with the objects identified by vehicle 900. If the results do not match and the infrastructure concludes that the AI in vehicle 900 has malfunctioned, then server 978 can transmit a signal to vehicle 900, instructing the fail-safe computer in vehicle 900 to take control, notify the passengers, and complete a safe parking operation.

[0246] For inference, server 978 can include GPU 984 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT). The combination of a GPU-powered server and inference acceleration can enable real-time response. In other examples, such as when performance is less critical, a CPU, FPGA, and other processor-powered servers can be used for inference.

[0247] Example computing device

[0248] Figure 10 FIG. is a block diagram of an example computing device 1000 suitable for implementing some embodiments of the present disclosure. Computing device 1000 can include an interconnect system 1002 that directly or indirectly couples the following devices: a memory 1004, one or more central processing units (CPUs) 1006, one or more graphics processing units (GPUs) 1008, a communication interface 1010, input / output (I / O) ports 1012, input / output components 1014, a power supply 1016, one or more presentation components 1018 (e.g., a display), and one or more logic units 1020. In at least one embodiment, one or more computing devices 1000 can include one or more virtual machines (VMs), and / or any of its components can include virtual components (e.g., virtual hardware components). For non-limiting examples, one or more of GPUs 1008 can include one or more vGPUs, one or more of CPUs 1006 can include one or more vCPUs, and / or one or more of logic units 1020 can include one or more virtual logic units. Thus, one or more computing devices 1000 can include discrete components (e.g., full GPUs dedicated to computing device 1000), virtual components (e.g., a portion of a GPU dedicated to computing device 1000), or a combination thereof.

[0249] Although Figure 10 each box of Figure 10 is shown as being connected via an interconnection system 1002 having circuitry, this is not intended to be restrictive and is for clarity only. For example, in some embodiments, a rendering component 1018 such as a display device may be considered an I / O component 1014 (e.g., if the display is a touchscreen). As another example, the CPU 1006 and / or the GPU 1008 may include memory (e.g., the memory 1004 may represent a storage device in addition to the memory of the GPU 1008, CPU 1006, and / or other components). In other words, Figure 10 the computing device of Figure 10 is merely illustrative. No distinction is made between categories such as "workstation", "server", "laptop", "desktop", "tablet", "client device", "mobile device", "handheld device", "game console", "electronic control unit (ECU)", "virtual reality system", and / or other device or system types because all of these are considered within the scope of the computing device of Figure 10 .

[0250] The interconnection system 1002 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 1002 may include one or more types of buses or links, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there are direct connections between components. For example, the CPU 1006 may be directly connected to the memory 1004. Additionally, the CPU 1006 may be directly connected to the GPU 1008. In cases where there are direct or point-to-point connections between components, the interconnection system 1002 may include a PCIe link to effect the connection. In these examples, a PCI bus need not be included in the computing device 1000.

[0251] The memory 1004 may include any of a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computing device 1000. Computer-readable media can include volatile and non-volatile media as well as removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0252] Computer storage media can include volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1004 can store computer-readable instructions (e.g., which represent programs and / or program elements such as an operating system). Computer storage media can include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disks (DVDs) or other optical disk storage devices, magnetic tape cartridges, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computing device 1000. As used herein, computer storage media does not include signals per se.

[0253] Computer storage media can include computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal such as a carrier wave or other transmission mechanism, and include any information conveyance medium. The term "modulated data signal" can refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example and not limitation, computer storage media can include wired media such as a wired network or direct wired connection, and wireless media such as sound, RF, infrared, and other wireless media. Any of the foregoing combinations should also be included within the scope of computer-readable media.

[0254] CPU 1006 can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 1000 to execute one or more of the methods and / or processes described herein. Each of the CPUs 1006 can include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of simultaneously processing a large number of software threads. CPU 1006 can include any type of processor and can include different types of processors depending on the type of computing device 1000 being implemented (e.g., a processor with fewer cores for a mobile device and a processor with more cores for a server). For example, depending on the type of computing device 1000, the processor can be an advanced RISC machine (ARM) processor implemented using reduced instruction set computing (RISC) or an x86 processor implemented using complex instruction set computing (CISC). In addition to one or more microprocessors or complementary coprocessors such as a math coprocessor, computing device 1000 can also include one or more CPUs 1006.

[0255] In addition to or instead of the CPU 1006, one or more GPUs 1008 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1000 to perform one or more of the methods and / or processes described herein. One or more GPUs 1008 may be integrated GPUs (e.g., having one or more CPUs 1006) and / or one or more GPUs 1008 may be discrete GPUs. In an embodiment, one or more GPUs 1008 may be coprocessors of one or more CPUs 1006. The computing device 1000 may use the GPUs 1008 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, one or more GPUs 1008 may be used for general-purpose computing on GPUs (GPGPU). One or more GPUs 1008 may include hundreds or thousands of cores capable of simultaneously processing hundreds or thousands of software threads. The GPU 1008 may generate pixel data for an output image in response to a rendering command (e.g., a rendering command received from the CPU 1006 via a host interface). The GPU 1008 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. The display memory may be included as part of the memory 1004. One or more GPUs 1008 may include two or more GPUs operating in parallel (e.g., via a link). The link may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs through a switch (e.g., using NVSwitch). When combined, each GPU 1008 may generate pixel data or GPGPU data for different portions of the output or different outputs (e.g., the first GPU for a first image and the second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0256] In addition to or as an alternative to CPU 1006 and / or GPU 1008, logic unit 1020 may be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 1000 to perform one or more of the methods and / or processes described herein. In an embodiment, CPU 1006, GPU 1008, and / or logic unit 1020 may perform any combination of methods, processes, and / or portions thereof, separately or jointly. One or more logic units 1020 may be part of and / or integrated in one or more of CPU 1006 and / or GPU 1008, and / or one or more logic units 1020 may be discrete components or otherwise outside of CPU 1006 and / or GPU 1008. In an embodiment, one or more logic units 1020 may be a coprocessor of one or more CPUs 1006 and / or one or more GPUs 1008.

[0257] Examples of logic unit 1020 include one or more processing cores and / or their components, such as data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree traversal unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU), application specific integrated circuit (ASIC), floating point unit (FPU), input / output (I / O) element, peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) element, etc.

[0258] Communication interface 1010 may include one or more receivers, transmitters, and / or transceivers that enable computing device 1000 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communication. Communication interface 1010 may include components and functionality enabling communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication via Ethernet or InfiniBand), low power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, one or more logic units 1020 and / or communication interface 1010 may include one or more data processing units (DPUs) to directly transfer data received over the network and / or via interconnect system 1002 to (e.g., memory) one or more GPUs 1008.

[0259] The I / O port 1012 can enable the computing device 1000 to be logically coupled to other devices including I / O components 1014, presentation components 1018, and / or other components, some of which may be built into (e.g., integrated into) the computing device 1000. Exemplary I / O components 1014 include microphones, mice, keyboards, joysticks, game pads, game controllers, dish satellite antennas, scanners, printers, wireless devices, and so on. The I / O components 1014 can provide a natural user interface (NUI) that processes user-generated air gestures, voice, or other physiological inputs. In some instances, the input can be transmitted to appropriate network elements for further processing. The NUI can implement any combination of speech recognition, stylus recognition, face recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition associated with the display of the computing device 1000 (described in more detail below). The computing device 1000 can include depth cameras such as stereo camera systems, infrared camera systems, RGB camera systems, touch screen technologies, and combinations thereof for gesture detection and recognition. Additionally, the computing device 1000 can include an accelerometer or gyroscope that enables motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope can be used by the computing device 1000 to render immersive augmented reality or virtual reality.

[0260] The power supply 1016 can include hard-wired power, battery power, or a combination thereof. The power supply 1016 can power the computing device 1000 so that the components of the computing device 1000 can operate.

[0261] The presentation component 1018 can include a display (e.g., a monitor, touch screen, television screen, head-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The presentation component 1018 can receive data from other components (e.g., GPU 1008, CPU 1006, DPU, etc.) and output the data (e.g., as images, videos, sounds, etc.).

[0262] Example data center

[0263] Figure 11 An example data center 1100 that can be used in at least one embodiment of the present disclosure is shown. The data center 1100 can include a data center infrastructure layer 1110, a framework layer 1120, a software layer 1130, and / or an application layer 1140.

[0264] As Figure 11As shown, the data center infrastructure layer 1110 may include a resource coordinator 1112, grouped computing resources 1114, and node computing resources (“node C.R.”) 1116(1)-1116(N), where “N” represents any whole positive integer. In at least one embodiment, the node C.R. 1116(1)-1116(N) may include, but is not limited to, any number of central processing units (“CPU”) or other processors (including accelerators, field programmable gate arrays (FPGA), graphics processors or graphics processing units (GPU), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VM”), power modules, and / or cooling modules, and so on. In some embodiments, one or more of the node C.R. 1116(1)-1116(N) may correspond to a server having one or more of the above computing resources. Additionally, in some embodiments, the node C.R. 1116(1)-11161(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more of the node C.R. 1116(1)-1116(N) may correspond to a virtual machine (VM).

[0265] In at least one embodiment, the grouped computing resources 1114 may include separate groupings of the node C.R. 1116 housed within one or more racks (not shown), or many racks within a data center located at different geographical locations (also not shown). Separate groupings of the node C.R. 1116 within the grouped computing resources 1114 may include grouped computing, network, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, several node C.R. 1116 including CPUs, GPUs, DPUs, and / or other processors may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any combination of any number of power modules, cooling modules, and / or network switches.

[0266] The resource coordinator 1122 may configure or otherwise control one or more of the node C.R. 1116(1)-1116(N) and / or the grouped computing resources 1114. In at least one embodiment, the resource coordinator 1122 may include a software design infrastructure (“SDI”) management entity for the data center 1100. The resource coordinator 1122 may include hardware, software, or some combination thereof.

[0267] In at least one embodiment, as Figure 11As shown, the framework layer 1120 may include a job scheduler 1133, a configuration manager 1134, a resource manager 1136, and / or a distributed file system 1138. The framework layer 1120 may include a framework for software 1132 that supports the software layer 1130 and / or one or more applications 1142 of the application layer 1140. The software 1132 or the application 1142 may respectively include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1120 may be, but is not limited to, a type of free and open-source software web application framework (such as Apache SparkTM (hereinafter referred to as "Spark")) that can utilize the distributed file system 1138 for large-scale data processing (e.g., "big data"). In at least one embodiment, the job scheduler 1133 may include a Spark driver to facilitate scheduling workloads supported by different layers of the data center 1100. The configuration manager 1134 may be able to configure different layers, such as the software layer 1130 and the framework layer 1120 (which includes Spark and the distributed file system 1138 for supporting large-scale data processing). The resource manager 1136 may be able to manage the clustered or grouped computing resources that are mapped to the distributed file system 1138 and the job scheduler 1133 or are allocated to support the distributed file system 1138 and the job scheduler 1133. In at least one embodiment, the clustered or grouped computing resources may include the grouped computing resources 1114 in the data center infrastructure layer 1110. The resource manager 1136 may coordinate with the resource coordinator 1112 to manage these mapped or allocated computing resources.

[0268] In at least one embodiment, the software 1132 included in the software layer 1130 may include software used by at least a portion of the node C.R.s 1116(1)-1116(N), the grouped computing resources 1114, and / or the distributed file system 1138 of the framework layer 1120. One or more types of software may include, but are not limited to, Internet web search software, email virus scanning software, database software, and streaming video content software.

[0269] In at least one embodiment, the applications 1142 included in the application layer 1140 may include one or more types of applications used by at least a portion of the nodes C.R. 1116(1)-1116(N), the grouped computing resources 1114, and / or the distributed file system 1138 of the framework layer 1120. One or more types of applications may include, but are not limited to, any number of genomic applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0270] In at least one embodiment, any one of the configuration manager 1134, the resource manager 1136, and the resource coordinator 1112 may implement any number and type of self-modifying actions based on any amount and type of data obtained in any technically feasible manner. The self-modifying actions may save the data center operator of the data center 1100 from making potentially poor configuration decisions and may avoid underutilization and / or poorly performing portions of the data center.

[0271] According to one or more embodiments described herein, the data center 1100 may include tools, services, software, or other resources to train one or more machine learning models or to use one or more machine learning models to predict or infer information. For example, (one or more) machine learning models may be trained by calculating weight parameters according to a neural network architecture by using the software and / or computing resources described above with respect to the data center 1100. In at least one embodiment, the trained or deployed machine learning models corresponding to one or more neural networks may be used to infer or predict information by using the weight parameters calculated by one or more training techniques (such as, but not limited to, those described herein) and using the resources described above with respect to the data center 1100.

[0272] In at least one embodiment, the data center 1100 may use a CPU, an application specific integrated circuit (ASIC), a GPU, an FPGA, and / or other hardware (or their corresponding virtual computing resources) to perform training and / or inference using the above resources. In addition, one or more of the software and / or hardware resources described above may be configured to allow a user to train or perform a service for inferring information, such as image recognition, speech recognition, or other artificial intelligence services.

[0273] Example Network Environment

[0274] The network environment suitable for implementing the embodiments of the present disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be implemented on one or more instances of the Figure 10 computing device 1000 - for example, each device may include similar components, features, and / or functions of the computing device 1000. Additionally, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of a data center 1100, an example of which is described herein with respect to Figure 11 more detail.

[0275] The components of the network environment may communicate with each other via a network, which may be wired, wireless, or both. The network may include multiple networks, or one network among multiple networks. For example, the network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the Internet and / or the public switched telephone network (PSTN), and / or one or more private networks. In the case where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) may provide wireless connectivity.

[0276] A compatible network environment may include one or more peer - to - peer network environments (in which case, servers may not be included in the network environment) and one or more client - server network environments (in which case, one or more servers may be included in the network environment). In a peer - to - peer network environment, the functions described herein with respect to servers may be implemented on any number of client devices.

[0277] In at least one embodiment, the network environment may include one or more cloud - based network environments, distributed computing environments, combinations thereof, etc. A cloud - based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework for supporting one or more applications of a software layer and / or an application layer. The software or application may respectively include network - based service software or applications. In an embodiment, one or more client devices may use network - based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open - source software web application framework, for example, which may use a distributed file system for large - scale data processing (e.g., “big data”).

[0278] Cloud-based network environments can provide cloud computing and / or cloud storage that perform any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these various functions can be distributed across multiple locations of a central or core server (e.g., across one or more data centers in a state, region, country, globe, etc.). If the connection to a user (e.g., a client device) is relatively close to an edge server, the core server can assign at least a portion of the function to the edge server. Cloud-based network environments can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0279] Client devices can include at least some of the components, features, and functions of the example computing device 1000 described herein with respect to Figure 10 As examples, and not limitations, client devices can be embodied as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance device or system, vehicle, boat, aircraft, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, in-vehicle computer system, embedded system controller, remote control, appliance, consumer electronic device, workstation, edge device, any combination of these devices described, or any other suitable device.

[0280] This disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, executed by a computer or other machine such as a personal digital assistant or other handheld device. Generally, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs particular tasks or implements particular abstract data types. This disclosure can be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. This disclosure can also be practiced in a distributed computing environment where tasks are performed by remote processing devices linked through a communications network.

[0281] As used herein, the recitation of "and / or" with respect to two or more elements shall be construed to refer to only one element or a combination of elements. For example, "element A, element B, and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. Further, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Still further, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0282] The subject matter of the present disclosure is described in detail herein to meet statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. On the contrary, the inventors have contemplated that the claimed subject matter may also be embodied in other ways, including different steps or combinations of steps similar to those described herein in connection with other current or future technologies. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of a method employed, these terms should not be construed as implying any particular order among or between the various steps disclosed herein unless the order of the steps is expressly recited.

Claims

1. A method, comprising: Receiving road data representing a road segment and lane data representing a lane, associated with a position of a self - machine; Determining that the continuous road segment matches the lane, at least based on a geometric similarity between the lane and the continuous road segment; And Generating a representation of the lane being mapped to the continuous road segment, at least based on determining that the continuous road segment matches the lane; And Performing one or more operations corresponding to the self - machine, at least based on the representation of the lane being mapped to the continuous road segment.

2. The method according to claim 1, wherein Determining that the continuous road segment matches the lane, at least based on the geometric similarity, includes: determining that the lane is within a distance threshold of one or more points along the continuous road segment.

3. The method according to claim 1, wherein Determining the continuous road segment that matches the lane, at least based on the geometric similarity, includes: determining that the lane is within an orientation threshold of one or more points along the continuous road segment.

4. The method according to claim 1, wherein Determining the continuous road segment that matches the lane, at least based on the geometric similarity, includes: determining that the lane is within a distance threshold and an orientation threshold of one or more points along the continuous road segment.

5. The method according to claim 1, wherein Determining the continuous road segment that matches the lane, at least based on the geometric similarity, includes: iteratively determining that one or more points along the continuous road segment are within the distance threshold of the lane, at least based on extending one or more vertical line segments from one or more points along the continuous road segment to intersect the lane.

6. The method according to claim 5, wherein The one or more points along the continuous road segment are within a predetermined distance between consecutive points among the one or more points.

7. The method according to claim 1, wherein Determining the continuous road segment that matches the lane, at least based on the geometric similarity, includes: determining that the lane is within a distance threshold and an orientation threshold of at least a predetermined number of points along the continuous road segment.

8. The method according to claim 1, wherein Determining the continuous road segment that matches the lane, at least based on the geometric similarity, includes: determining that the lane is within a distance threshold and an orientation threshold of one or more points located at least a predetermined distance apart along the continuous road segment.

9. The method according to claim 1, further comprising: Generating a representation of the degree to which the continuous road segment matches the lane, at least based on the geometric similarity between the lane and the continuous road segment.

10. The method according to claim 1, wherein, Generating the representation of the degree to which the continuous road segment matches the lane is based on at least the sum of reciprocals of one or more distances between the lane and one or more sampled positions along the continuous road segment.

11. The method according to claim 1 further comprises: Determining to generate the representation of the lane being mapped to the continuous road segment, at least based on the representation of the degree to which the continuous road segment matches the lane being greater than another representation of the degree to which another continuous road segment matches the lane.

12. The method according to claim 1, wherein, The representation of the lane being mapped to the continuous road segment includes: a representation of the start and end positions of a section of the lane, a representation of the identity of at least a portion of the continuous road segment, and a representation of the start and end positions of at least one road segment.

13. The method according to claim 1 further comprises: Select the lane based at least on the lane having positions along its geometry within a maximum distance from the position of the ego machine.

14. The method according to claim 1, wherein the method is performed using at least one of the following: A control system for an autonomous or semi-autonomous machine; A perception system for an autonomous or semi-autonomous machine; A system for performing simulation operations; A system for performing real-time streaming; A system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; A system for performing digital twin operations; A system for performing deep learning operations; A system implemented using edge devices; A system implemented using robots; A system comprising one or more virtual machines (VMs); A system implemented at least partially in a data center; A system for performing optical transmission simulation; A system for performing collaborative content creation of 3D assets; A system for generating synthetic data; or A system implemented at least partially using cloud computing resources.

15. One or more processors, including processing circuitry, the processing circuitry for: Determine that the one or more consecutive road segments match the lane based at least on a geometric similarity between the lane detected by the ego machine and the one or more consecutive road segments detected by the ego machine; and Generate a representation of the lane being mapped to the one or more consecutive road segments based at least on determining that the one or more consecutive road segments match the lane.

16. The one or more processors according to claim 15, wherein, Determining the one or more consecutive road segments that match the lane based at least on the geometric similarity includes an iterative process that determines that the lane is within a distance threshold and a direction threshold of a sequence of one or more points along the one or more consecutive road segments.

17. The one or more processors according to claim 15, wherein the one or more processors are included in at least one of the following: A control system for an autonomous or semi-autonomous machine; A perception system for an autonomous or semi-autonomous machine; A system for performing simulation operations; A system for performing digital twin operations; A system for performing optical transmission simulation; A system for performing collaborative content creation of 3D assets; A system for performing deep learning operations; A system for performing remote operations; A system for performing real-time streaming; A system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; A system implemented using edge devices; A system implemented using robots; A system for performing conversational AI operations; A system implementing one or more language models; A system implementing one or more large language models (LLMs); A system for generating synthetic data; A system for generating synthetic data using AI; A system comprising one or more virtual machines (VMs); A system implemented at least partially in a data center; or A system implemented at least partially using cloud computing resources.

18. A system, comprising one or more processors, the one or more processors for: Determine that the road segment matches the lane based at least on geometric similarity between the lane of a lane map associated with a ego machine and a road segment of a road map associated with the ego machine; Generate a representation of a degree of match between the road segment and the lane based at least on the geometric similarity between the lane and the road segment; Generate a representation of the lane mapped to the road segment based at least on the representation of the degree of match between the road segment and the lane; And Perform one or more operations associated with control of the ego machine based at least on the representation of the lane mapped to the road segment.

19. The system according to claim 18, wherein, The representation of the degree of match between the road segment and the lane is generated based at least on a sum of reciprocals of one or more distances between the lane and one or more sampled positions along the road segment.

20. The system according to claim 18, wherein the system is included in at least one of the following: A control system for an autonomous or semi-autonomous machine; A perception system for an autonomous or semi-autonomous machine; A system for performing simulation operations; A system for performing digital twin operations; A system for performing optical transmission simulation; A system for performing collaborative content creation of 3D assets; A system for performing deep learning operations; A system for performing real-time streaming; A system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; A system implemented using an edge device; A system implemented using a robot; A system for performing conversational AI operations; A system for generating synthetic data; A system including one or more virtual machines (VMs); A system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

Images