Lane positioning determination for autonomous systems and applications

Abstract

The invention discloses lane positioning determination for autonomous systems and applications. In various examples, use of bi-directional output to determine lane localization for autonomous and / or semi-autonomous systems and applications is described herein. Systems and methods described herein may determine a plurality of outputs (e.g., vectors) associated with a lane of a driving surface (e.g., a roadway), where the outputs are indexed from different locations with respect to the driving surface, and then use the plurality of outputs to determine a lane in which the machine is traveling. In some examples, the output may include a vector including a plurality of elements, where a respective element is associated with at least a lane of the driving surface and indicates a probability that the machine is located within the lane. Further, in some examples, the outputs may be indexed from different sides of the driving surface, such as the right and left sides of the driving surface.

Description

BACKGROUND

[0001] To enable vehicles (e.g., autonomous vehicles, semi-autonomous vehicles, and robots, etc.) to safely operate in an environment, the vehicles must be able to effectively perform various vehicle maneuvers, such as lane keeping, lane changing, lane splitting, turning, stopping and starting at intersections, crosswalks, etc., and / or other vehicle or machine maneuvers. For example, for vehicles that navigate on ground streets (e.g., city streets, alleyways, neighborhood streets, etc.) and highways (e.g., multi-lane roads), the vehicles need to navigate between one or more divisions or demarcations of the road (e.g., lanes, intersections, crosswalks, boundaries, etc.), which are often marked using road markings, such as lane lines. Thus, it is important for the vehicles to be able to determine the lane in which the vehicle is navigating using these road markings so that the vehicle can determine how to navigate according to the rules and / or lane position.

[0002] Conventional techniques to determine the lane in which a vehicle is navigating can use an environment map, where the map indicates the number of lanes associated with various roads. For example, using the map, a vehicle can determine the number of lanes associated with the road in which the vehicle is navigating, and then use other data, such as sensor data, to determine the lane in which the vehicle is currently navigating. However, in some cases, these maps can not include information associated with a road, such as if the road has not been traveled by a data collection vehicle (e.g., the road is new), and / or can include inaccurate information, such as if the road has been updated to add and / or remove lanes. In such cases, it can be difficult for a vehicle to determine the lane in which the vehicle is navigating. For example, if a map indicates that a road includes two lanes, but the road actually includes four lanes, a vehicle using the map can not be able to determine whether the vehicle is navigating on the second lane to the right of the road or the third lane to the right of the road. SUMMARY

[0003] Embodiments of the present disclosure relate to using bidirectional outputs to determine lane positioning for autonomous and / or semi-autonomous systems and applications. The systems and methods described herein can determine a plurality of outputs (e.g., vectors) associated with lanes of a driving surface (e.g., a road), where the outputs are indexed from different positions relative to the driving surface, and then use the plurality of outputs to determine the lane in which a machine is navigating. As described more herein, in some examples, the outputs can include vectors that include a plurality of elements, where respective elements are associated with at least a lane of the driving surface and indicate a probability that the machine is positioned within the lane. Further, in some examples, the outputs can include at least a first output indexed from a first side of the driving surface (such as a right side of the driving surface) and a second output indexed from a second side of the driving surface (such as a left side of the driving surface).

[0004] In comparison to conventional systems, the system of the present disclosure can determine a plurality of outputs indexed from different locations relative to a driving surface, and then use the plurality of outputs to determine the lane in which the machine is traveling. This provides an improvement because the system of the present disclosure can not need to rely on information from a map, such as the number of lanes associated with a road, to determine the lane in which the machine is traveling. Moreover, as will be described in greater detail herein, by using at least two different outputs indexed from a plurality of locations associated with a driving surface to determine a lane, the system of the present disclosure can be more accurate or precise because at least one of the outputs can include a high degree of confidence that the machine is within the selected lane. BRIEF DESCRIPTION OF DRAWINGS

[0005] The present systems and methods for determining lane positioning using bidirectional outputs for autonomous and / or semi-autonomous systems and applications are described in detail below with reference to the following drawings, wherein:

[0006] FIG. 1 An example dataflow diagram of a process for determining a lane in which a machine is traveling using bidirectional outputs is shown in accordance with some embodiments of the present disclosure;

[0007] FIG. 2 An example of a machine traveling along a driving surface in an environment is shown in accordance with some embodiments of the present disclosure, wherein the driving surface includes a plurality of lanes;

[0008] FIGS. 3A-3B An example of outputs from one or more perception systems of a machine is shown in accordance with some embodiments of the present disclosure;

[0009] FIG. 4 An example of directional outputs indicating whether a machine is within one or more lanes is shown in accordance with some embodiments of the present disclosure;

[0010] FIGS. 5A-5B An example of updating directional outputs associated with lane selection based at least on a machine switching lanes is shown in accordance with some embodiments of the present disclosure;

[0011] FIG. 6 A flowchart of a method for determining a lane in which a machine is traveling using directional vectors is shown in accordance with some embodiments of the present disclosure;

[0012] FIG. 7 A flowchart of a method for determining a lane in which a machine is traveling using directional outputs is shown in accordance with some embodiments of the present disclosure;

[0013] FIG. 8A A diagram of an example autonomous vehicle in accordance with some embodiments of the present disclosure;

[0014] FIG. 8Bis a block diagram of an example system architecture of an example autonomous vehicle in accordance with some embodiments of the present disclosure FIG. 8A

[0015] FIG. 8C is a block diagram of an example system architecture of an example autonomous vehicle in accordance with some embodiments of the present disclosure FIG. 8A

[0016] FIG. 8D is a block diagram of an example system architecture of an example autonomous vehicle in accordance with some embodiments of the present disclosure FIG. 8A

[0017] FIG. 9 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0018] FIG. 10 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Systems and methods related to determining lane positioning using bidirectional output for autonomous and / or semi-autonomous systems and applications are disclosed. Although the present disclosure can be described with respect to an example autonomous or semi-autonomous vehicle or machine 800 (alternatively referred to herein as“vehicle 800,”“ego vehicle 800,”“ego machine 800,” or“machine 800,” examples of which are described with respect to FIGS. 8A-8D This is not intended to be limiting. For example, the systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles and machines, manned and unmanned robotic or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, dirigibles, ships, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles, drones, and / or other vehicle types. Further, although the present disclosure can be described with respect to lane positioning for autonomous or semi-autonomous systems and applications, this is not intended to be limiting and the systems and methods described herein can be used in augmented reality, virtual reality, mixed reality, robotics, safety and supervision, autonomous or semi-autonomous machine applications, and / or any other technical space that can use positioning or object-to-path / location / lane assignment.

[0020] ​​​For example, the system can receive sensor data generated using one or more sensors of a machine traveling in an environment. As described herein, the sensor data can include, but is not limited to, image data generated using one or more image sensors, LiDAR data generated using one or more LiDAR sensors, RADAR data generated using one or more RADAR sensors, ultrasonic data generated using one or more ultrasonic sensors, and / or any other type of sensor data generated using any other type of sensor. Further, the sensor data can represent at least a driving surface (e.g., a road) located within the environment, where the road includes a plurality of lanes (e.g., one lane, two lanes, five lanes, ten lanes, etc.). In some examples, the system can then use the sensor data, along with map data representing one or more maps, to localize the machine in the environment. As described herein, the maps can include, but are not limited to, navigation maps, standard definition maps, high definition maps, and / or any other type of map.

[0021] The system can then use one or more machine learning models, such as one or more machine learning models associated with one or more perception systems, to process at least a portion of the sensor data to determine information associated with the driving surface represented by the sensor data. For example, in some examples, the system can use one or more first machine learning models to process at least a portion of the sensor data, which are trained to detect boundaries associated with the driving surface and / or the lanes. For example, the first machine learning models can generate and / or output data representing at least locations of the driving surface boundaries, locations of the lane boundaries, locations of driving surface markings, locations of lane markings, types of driving surface markings, types of lane markings, and / or any other information associated with the boundaries. Additionally or alternatively, in some examples, the system can use one or more second machine learning models to process at least a portion of the sensor data, which are trained to detect paths (e.g., lanes) in the environment. For example, the second machine learning models can generate and / or output data representing one or more locations (e.g., one or more ranges) of one or more lanes located within the environment.

[0022] The system can then process at least a portion of the data output from the machine learning model, at least a portion of the map data (e.g., data indicative of a type of road associated with the driving surface), at least a portion of the sensor data, and / or any other type of data using one or more processing components configured to determine information (localization information) associated with one or more lanes in which the machine can be located. As described herein, the processing components can include, but are not limited to, models, machine learning models, neural networks, algorithms, filters, modules, and / or any other type of processing component configured to perform one or more processes described herein. Based at least on processing the data, the processing components can generate and / or output data representative of information associated with the lanes.

[0023] For example, in some examples, the processing components can output a first output associated with a first side of the driving surface (such as a right side of the driving surface) and a second output associated with a second side of the driving surface (such as a left side of the driving surface). In some examples, the first output can include a first vector indexed starting from the first side of the driving surface (e.g., a rightmost lane), where the first vector includes a first number of elements associated with one or more first lanes, and the second output can include a second vector indexed starting from the second side of the driving surface (e.g., a leftmost lane), where the second vector includes a second number of elements associated with one or more second lanes. As described herein, the number of elements associated with a vector can include, but is not limited to, one element, two elements, five elements, eight elements, ten elements, fifteen elements, and / or any other number of elements. Further, the elements can be associated with respective lanes that can or can not be located within the environment. For example, if a vector includes eight elements, but the driving surface includes only four lanes, then four of the elements can be associated with actual lanes within the environment, while four of the elements can be associated with no actual lanes within the environment. In a warehouse example, the two localization results can correspond to a hallway, an aisle, a path, and / or other demarcated, bounded, or determined partition in the environment.

[0024] As an example of a processing component that processes data, a machine can be traveling along a four-lane road within an environment and on the right lane. Further, the processing component can be configured to generate a vector that includes eight elements. Thus, the processing component can generate a first vector that is indexed from the right side of the road, where a first element of the first vector indicates a first probability that the machine is in a first lane from a right boundary, a second element of the first vector indicates a second probability that the machine is in a second lane from the right boundary, a third element of the first vector indicates a third probability that the machine is in a third lane from the right boundary, a fourth element of the first vector indicates a fourth probability that the machine is in a fourth lane from the right boundary, and other elements can indicate other probabilities associated with other lanes that are not present. Further, the processing component can generate a second vector that is indexed from the left side of the road, where a first element of the second vector indicates a first probability that the machine is in a first lane from a left boundary, a second element of the second vector indicates a second probability that the machine is in a second lane from the left boundary, a third element of the second vector indicates a third probability that the machine is in a third lane from the left boundary, a fourth element of the second vector indicates a fourth probability that the machine is in a fourth lane from the left boundary, and other elements of the second vector can indicate other probabilities associated with other lanes that are not present.

[0025] In this example, the first probability indicated by the first element of the first vector can include the highest probability associated with the first vector, followed by the second probability indicated by the second element, the third probability indicated by the third element, the fourth probability indicated by the fourth element, and then the remaining probabilities. Further, the fourth probability indicated by the fourth element of the second vector can include the highest probability associated with the second vector. In some examples, the first element of the first vector and the fourth element of the second vector can include the highest probabilities associated with the vectors because they represent the actual lane in which the machine is traveling. Further, in some examples, the first probability indicated by the first element of the first vector can be greater than the fourth probability indicated by the fourth element of the second vector because, based on the sensor data, it can be easier to detect the machine’s position from the right side of the road based on the machine being in the right lane (e.g., based on road boundaries, which are described in more detail herein).

[0026] The system can then use the output to determine the lane in which the machine is traveling. In some examples, the system can determine that the lane includes the lane associated with the element containing the highest probability of the probabilities. In some examples, the system can determine that the lane includes the lane associated with the element containing the highest probability if the highest probability satisfies (e.g., is equal to or greater than) a threshold probability (e.g., 85%, 90%, 95%, 99%, etc.). While these are merely a few example techniques by which the system can use the output to select the lane, in other examples, the system can use one or more additional and / or alternative techniques to use the output to select the lane.

[0027] In some examples, the system can continue to perform these processes to continue to determine the lane in which the machine is traveling. For example, the system can obtain second sensor data generated using the sensors, generate additional data based at least on processing the second sensor data using the machine learning model, generate additional output based at least on processing the additional data (and / or other data) using the processing component, and then determine an updated lane in which the machine is traveling using the additional output. In some examples, the processing component can generate the additional output by updating the previous output using the additional data. For example, as will be described in detail herein, the processing component can continuously update the probabilities associated with the output as the machine continues to generate new sensor data for processing.

[0028] In some examples, the system can use one or more additional and / or alternative inputs for the processing component. For example, the system can determine when the machine switches from traveling in a current lane to traveling in a new lane. Based at least on this determination, the system can input data indicating that the machine switched lanes, data indicating a probability that the machine switched lanes, data indicating a direction associated with the switch of lanes (e.g., switched to a left lane, switched to a right lane, etc.), and / or any other data associated with the switch of lanes. The processing component can then use this additional data in determining and / or updating the probabilities. For the first example, as described in greater detail herein, if the data indicates that the machine switched to the new lane in a particular direction, the processing component can shift the probabilities associated with the output (or can use the data as a hint to be considered in weighting toward the switch). For the second example, and also as described in greater detail herein, if the data indicates a probability that the machine switched to the new lane and a particular direction, the processing component can use this probability in updating the probabilities associated with the output. While these are merely a few example techniques by which the processing component can update the probabilities based on data associated with switching lanes, in other examples, the processing component can use additional and / or alternative techniques to update the probabilities based on this data.

[0029] In some examples, the system can then perform one or more operations based at least on the determination of the lane in which the machine is traveling. For example, the system can determine one or more trajectories for the machine to travel based at least on the lane in which the machine is traveling. For example, if the machine is to make a right turn and the machine is currently in the right lane, the system can determine a trajectory that includes only the right turn being made by the machine. However, if the machine needs to make a right turn but is in another lane, the system can determine a trajectory that includes the machine initially switching lanes to enter the right lane.

[0030] The systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles and machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, spacecraft, ships, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles, drones, and / or other vehicle types. Further, the systems and methods described herein can be used for various purposes, such as, by way of example and not limitation, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, safety and supervision, simulation and digital twin, autonomous or semi-autonomous machine applications, deep learning, environment simulation, object or actor simulation and / or digital twin, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, and / or any other suitable application.

[0031] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aerial systems, medical systems, boating systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing large language models (LLMs), systems implementing one or more visual language models (VLMs), systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in data centers, systems for performing conversational AI operations, systems for performing light transport simulation, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0032] Reference is made to FIG. 1 , FIG. 1Example data flow diagrams are shown for a process 100 for determining the lane a machine is traveling in using bidirectional output, according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are illustrative only. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groups, etc.) may be used in addition to or in place of the arrangements or elements shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components and implemented in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, the various functions can be implemented by a processor that executes instructions stored in memory. In some embodiments, a combination of... FIGS. 8A-8D Example of autonomous vehicles 800 FIG. 9 Example computing device 900, and / or FIG. 10 The components, features, and / or functions of the example data center 1000 are similar to those of other components, features, and / or functions that perform the systems, methods, and processes described herein.

[0033] Process 100 may include one or more sensing systems 102 that receive sensor data 104 generated using a machine (e.g., autonomous vehicle 800). As described herein, sensor data 104 may include, but is not limited to, image data generated using one or more image sensors, LiDAR data generated using one or more LiDAR sensors, RADAR data generated using one or more RADAR sensors, ultrasonic data generated using one or more ultrasonic sensors, and / or any other type of sensor data generated using any other type of sensor. Furthermore, sensor data 104 may represent one or more sensor representations (e.g., one or more images, one or more point clouds, etc.) associated with the environment in which the machine is located. For example, sensor data 104 may at least represent the driving surface (e.g., road) on which the machine is traveling, where the road includes one or more lanes (e.g., one lane, two lanes, five lanes, ten lanes, etc.). In some cases, a driving or drivable surface may correspond to other areas besides roads, such as parks, parking lots, warehouses, buildings, facilities, factories, etc., and the demarcated or defined areas may include corridors, lobbies, marked areas, unmarked drivable areas, paths, etc.

[0034] For example, FIG. 2An example is shown of a machine 202 traveling within an environment along a drive surface 204 in accordance with some embodiments of the present disclosure, where the drive surface 204 includes a plurality of lanes 206(1)-206(5) (also referred to singularly as “lane 206” or in the plural as “lanes 206”). In FIG. 2 In examples, while traveling, the machine 202 can generate sensor data (e.g., sensor data 204) representative of at least an environment (such as the drive surface 204) surrounding the machine 202. The machine 202 can then use the sensor data 104, along with map data representative of the environment (e.g., map data 106), to localize the machine 202 with respect to the environment and on the drive surface 204. While FIG. 2 In examples of

[0035] Referring back to examples of FIG. 1 In examples, the process 100 can include the perception system 102 processing at least a portion of the sensor data 104 and generating output data 108 based at least on the processing. As described herein, the perception system 102 can include one or more perception systems associated with the machine, such as a first perception system 102 trained to detect boundaries of a drive surface (e.g., a road) and / or a lane and a second perception system 102 trained to detect a location (e.g., extent) of a lane. Accordingly, the perception system 102 can include and / or use one or more machine learning models, one or more neural networks, one or more algorithms, one or more models, and / or any other type of processing component configured to perform the processes described herein with respect to the processing system 102.

[0036] As shown, the output data 108 can include at least boundary data 110 and lane data 112. For example, the first perception system 102 can process the data and generate, based at least on the processing, boundary data 110 representative of information associated with boundaries of a drive surface and / or a lane. As described herein, the information can include, but is not limited to, a location of a drive surface boundary, a location of a lane boundary, a location of a drive surface marking, a location of a lane marking, a type of drive surface marking, a type of lane marking, and / or any other information associated with a boundary. Further, the second perception system 102 can process the data and generate, based at least on the processing, lane data 112 representative of information associated with one or more lanes of a drive surface. As described herein, the information can include, but is not limited to, one or more locations (e.g., one or more extents) of one or more lanes located within an environment.

[0037] In some examples, the output data 108 can represent two-dimensional (2D) information, three-dimensional (3D) information, and / or any other information associated with the environment. For a first example, the boundary data 110 can represent information indicating 2D locations of boundaries depicted by one or more sensor representations represented by the sensor data 104 and / or the lane data 112 can represent information indicating 2D locations of lanes depicted by the sensor representations. For a second example, the boundary data 110 can represent information indicating 3D locations of boundaries within the environment and / or the lane data 112 can represent information indicating 3D locations of lanes within the environment.

[0038] For example, FIGS. 3A-3B Examples of output from one or more perception systems of a machine in accordance with some embodiments of the disclosure are shown. As FIG. 3A As shown in the example of FIG. 3, the perception system 102 can process at least a portion of the sensor data obtained from the machine 202, where the sensor data includes image data representing at least the image 302. Based at least on this processing, the perception system 102 can generate and / or output data (e.g., the boundary data 110) representing at least locations of the driving surface boundaries 304(1)-304(2) depicted by the image 302 and locations of the lane boundaries 306(1)-306(4) depicted by the image 302. In addition, in some examples, the perception system 102 can output additional information, such as types associated with the driving surface boundaries 304(1)-304(2) and / or types associated with the lane boundaries 306(1)-306(4). The perception system 102 can then continue to perform these processes while processing additional sensor data.

[0039] As shown in the example of FIG. 3, the perception system 102 can process at least a portion of the sensor data obtained from the machine 202, where the sensor data includes image data representing at least the image 302. Based at least on this processing, the perception system 102 can generate and / or output data (e.g., the boundary data 110) representing at least locations of the driving surface boundaries 304(1)-304(2) depicted by the image 302 and locations of the lane boundaries 306(1)-306(4) depicted by the image 302. In addition, in some examples, the perception system 102 can output additional information, such as types associated with the driving surface boundaries 304(1)-304(2) and / or types associated with the lane boundaries 306(1)-306(4). The perception system 102 can then continue to perform these processes while processing additional sensor data. FIG. 3B As shown in the example of FIG. 3, the perception system 102 can process at least a portion of the sensor data obtained from the machine 202, where the sensor data includes image data representing at least the image 302. Based at least on this processing, the perception system 102 can generate and / or output data (e.g., the boundary data 110) representing at least locations of the driving surface boundaries 304(1)-304(2) depicted by the image 302 and locations of the lane boundaries 306(1)-306(4) depicted by the image 302. In addition, in some examples, the perception system 102 can output additional information, such as types associated with the driving surface boundaries 304(1)-304(2) and / or types associated with the lane boundaries 306(1)-306(4). The perception system 102 can then continue to perform these processes while processing additional sensor data. FIGS. 3A-3B As shown in the example of FIG. 3, the perception system 102 can process at least a portion of the sensor data obtained from the machine 202, where the sensor data includes image data representing at least the image 302. Based at least on this processing, the perception system 102 can generate and / or output data (e.g., the boundary data 110) representing at least locations of the driving surface boundaries 304(1)-304(2) depicted by the image 302 and locations of the lane boundaries 306(1)-306(4) depicted by the image 302. In addition, in some examples, the perception system 102 can output additional information, such as types associated with the driving surface boundaries 304(1)-304(2) and / or types associated with the lane boundaries 306(1)-306(4). The perception system 102 can then continue to perform these processes while processing additional sensor data.

[0040] Referring back to the example of FIG. 3, the perception system 102 can generate and / or output data (e.g., the lane data 112) representing at least locations of the lanes 308(1)-308(5) depicted by the image 302 based at least on processing at least a portion of the sensor data. In some examples, the perception system 102 can output additional information, such as a location of the lane 308(2) in which the machine 202 is currently traveling. While the example of FIG. 3 shows the perception system 102 generating and / or outputting data representing 2D information associated with the driving surface 204 and / or the lanes 206, in other examples, the perception system 102 can generate and / or output 3D information associated with the driving surface 204 and / or the lanes 206. FIG. 1For example, process 100 may include one or more lane components 114 that process at least a portion of output data 108. As described herein, lane components 114 may include and / or utilize one or more models, one or more machine learning models, one or more neural networks, one or more algorithms, one or more filters, one or more modules, and / or any other type of component configured to perform one or more processes described herein. In some examples, and as... FIG. 1 As further illustrated in the example, lane assembly 114 can process additional data, such as at least a portion of sensor data 104 and / or at least a portion of map data 106 representing the environment. As described herein, the map can include, but is not limited to, navigation maps, standard definition maps, high definition maps, and / or any other type of map. For example, lane assembly 114 can at least process map data 106 of the type indicating the driving surface on which the machine is traveling. As described herein, the type of driving surface can include, but is not limited to, rural roads, highways, expressways, expressway entrances, expressway exits, and / or any other type of driving surface.

[0041] Then, process 100 may include: processing data at least based on lane component 114, which generates and / or outputs data 116 representing information associated with lanes on the driving surface. As described herein, in some examples, lane component 114 may output a first directional output 118(1) associated with a first side of the driving surface (e.g., the right side of the driving surface) and a second directional output 118(2) associated with a second side of the driving surface (e.g., the left side of the driving surface). In some examples, the first directional output 118(1) may include a first vector (and / or any other type of output) comprising a first number of elements associated with one or more first lanes, and the second directional output 118(2) may include a second vector (and / or any other type of output) comprising a second number of elements associated with one or more second lanes. As described herein, the plurality of elements associated with a vector may include, but is not limited to, one element, two elements, five elements, eight elements, ten elements, fifteen elements, and / or any other number of elements. Furthermore, elements may be associated with corresponding lanes located within and / or not located within the environment. For example, if a vector contains eight elements, but the driving surface only includes four lanes, then four of the elements can be associated with the actual lanes in the environment, while four of the elements can be unassociated with the actual lanes in the environment.

[0042] As described herein, the directional outputs 118(1)-118(2) can indicate one or more probabilities that the machine is located within a lane. For example, the first directional output 118(1) can include a first element indicating a first probability that the machine is located in a first lane from a right boundary, a second element indicating a second probability that the machine is located in a second lane from the right boundary, a third element indicating a third probability that the machine is located in a third lane from the right boundary, a fourth element indicating a fourth probability that the machine is located in a fourth lane from the right boundary, and so on. Further, the second directional output 118(2) can include a first element indicating a first probability that the machine is located in a first lane from a left boundary, a second element indicating a second probability that the machine is located in a second lane from the left boundary, a third element indicating a third probability that the machine is located in a third lane from the left boundary, a fourth element indicating a fourth probability that the machine is located in a fourth lane from the left boundary, and so on.

[0043] In some examples, the probabilities can be expressed in percentages, such as 10%, 50%, 75%, 99%, and / or any other percentage. In these examples, the total percentage of all probabilities can add up to a maximum percentage, such as 100% (and / or any other percentage). In some examples, the probabilities can be expressed in numbers, e.g., 1 / 8, 1 / 4, 1 / 2, 3 / 4, and so on. In these examples, the probabilities can again add up to a maximum value, such as 1 (and / or any other number). While these are just a few examples of probabilities expressed using the directional outputs 118(1)-118(2), in other examples, the directional outputs 118(1)-118(2) can include any other information indicating whether the machine is located within one or more lanes of a driving surface.

[0044] For example, FIG. 4 Examples of directional outputs 402(1)-402(2) indicating whether the machine is located within one or more lanes (which can be similar to and / or include the directional outputs 118(1)-118(2)) are shown in accordance with some embodiments of the present disclosure. As shown, the first directional output 402(1) can include a plurality of elements associated with respective lanes 404(1)-404(6). For example, a first element can be associated with a first lane 404(1) from a right boundary, a second element can be associated with a second lane 404(2) from the right boundary, a third element can be associated with a third lane 404(3) from the right boundary, a fourth element can be associated with a fourth lane 404(4) from the right boundary, a fifth element can be associated with a fifth lane 404(5) from the right boundary, and a sixth element can be associated with a sixth lane 404(6) from the right boundary.

[0045] The first direction output 402(1) can further indicate a first probability 406(1) that the machine 202 is in the first lane 404(1), a second probability 406(2) that the machine 202 is in the second lane 404(2), a third probability 406(3) that the machine 202 is in the third lane 404(3), a fourth probability 406(4) that the machine 202 is in the fourth lane 404(4), a fifth probability 406(5) that the machine 202 is in the fifth lane 404(5), and a sixth probability 406(6) that the machine 202 is in the sixth lane 404(6). Although FIG. 4 The example of FIG. 4A illustrates the first direction output 402(1) as including six elements associated with the six lanes 404(1)-404(6), in other examples, the first direction output 402(1) can include any number of elements associated with any number of lanes.

[0046] As further illustrated, the second direction output 402(2) can include a plurality of elements associated with respective lanes 408(1)-408(6). For example, a first element can be associated with a first lane 408(1) from a left boundary, a second element can be associated with a second lane 408(2) from the left boundary, a third element can be associated with a third lane 408(3) from the left boundary, a fourth element can be associated with a fourth lane 408(4) from the left boundary, a fifth element can be associated with a fifth lane 408(5) from the left boundary, and a sixth element can be associated with a sixth lane 408(6) from the left boundary.

[0047] The second direction output 402(2) can further indicate a first probability 410(1) that the machine 202 is in the first lane 408(1), a second probability 410(2) that the machine 202 is in the second lane 408(2), a third probability 410(3) that the machine 202 is in the third lane 408(3), a fourth probability 410(4) that the machine 202 is in the fourth lane 408(4), a fifth probability 410(5) that the machine 202 is in the fifth lane 408(5), and a sixth probability 410(6) that the machine 202 is in the sixth lane 408(6). Although FIG. 4 The example of FIG. 4A illustrates the second direction output 402(2) as including six elements associated with the six lanes 408(1)-408(6), in other examples, the second direction output 402(2) can include any number of elements associated with any number of lanes.

[0048] In FIG. 4In the example of FIG. 4, first lane 404(1) can correspond to lane 206(1), second lane 404(2) can correspond to lane 206(2), third lane 404(3) can correspond to lane 206(3), fourth lane 404(4) can correspond to lane 206(4), fifth lane 404(5) can correspond to lane 206(5), and sixth lane 404(6) can not correspond to any lane. Further, first lane 408(1) can correspond to lane 206(5), second lane 408(2) can correspond to lane 206(4), third lane 408(3) can correspond to lane 206(3), fourth lane 408(4) can correspond to lane 206(2), fifth lane 408(5) can correspond to lane 206(1), and sixth lane 408(6) can not correspond to any lane.

[0049] Accordingly, because machine 202 is located in lane 206(2), second probability 406(2) associated with second lane 206(2) can include the highest probability of probabilities 406(1)-406(6). Further, because machine 202 is again located in lane 206(2), fourth probability 410(4) associated with fourth lane 408(4) can include the highest probability of probabilities 410(1)-410(6). However, second probability 406(2) can include a higher probability than fourth probability 410(4) because, based at least on sensor data, it is easier to detect that machine 202 is located closer to one side 208(1) of driving surface 202 than to the other side 208(2) of driving surface 202 (e.g., because machine 202 is closer to right side 208(1) of driving surface 204 than to left side 208(2), a field of view and / or a sensing field of a sensor of machine 202 can have a better or closer view of right side 208(1)). In some examples, this can be because, based on the location of machine 202, perception system 102 can more accurately detect a location of surface boundary 304(1) associated with one side 208(1) of driving surface 202 than a location of surface boundary 304(2) associated with the other side 208(2) of driving surface 202.

[0050] Referring back to FIG. 4, FIG. 1In some examples, process 100 can include one or more selection components 120 processing at least a portion of output data 116 and, based at least on the processing, generating and / or outputting selection data 122 representing a lane in which the machine is located. In some examples, selection component 120 can determine that the lane includes the lane associated with the element including the highest probability of the probabilities. In some examples, selection component 120 can determine that the lane includes the lane associated with the element including the highest probability if the highest probability satisfies (e.g., is equal to or greater than) a threshold probability (e.g., 85%, 90%, 95%, 99%, etc.). While these are just a few example techniques by which selection component 120 can use output data 116 to select a lane, in other examples, selection component 120 can use additional and / or alternative techniques to use output data 116 to select a lane.

[0051] As described herein, in some examples, as the machine continues to travel within the environment and along the driving surface, process 100 can continue to repeat. For example, perception system 102 can process additional sensor data 104 to generate additional output data 108, lane component 114 can continue to process additional output data 108 to generate additional output data 116, and lane component 120 can continue to process additional output data 116 to generate additional selection data 122 representing one or more lanes in which the machine is traveling. In some examples, as described herein, as additional output data 116 is generated, lane component 114 can update probabilities from previous output data 116 as lane component 114 continues to process additional output data 108. FIG. 1 As indicated by the double arrows in the example of FIG. 1, as additional output data 116 is generated, lane component 114 can update probabilities from previous output data 116 as lane component 114 continues to process additional output data 108.

[0052] For example, if the machine continues to travel in the same lane, the probability associated with the lane represented by first direction output 118(1) can continue to increase and / or the probability associated with the lane represented by second direction output 118(2) can continue to increase. Additionally, one or more probabilities associated with one or more additional lanes represented by direction outputs 118(1)-118(2) can continue to decrease. In some examples, these probabilities can increase and / or decrease as lane component 114 can become more accurate as lane component 114 continues to process additional data. However, if the machine switches lanes and / or the number of lanes associated with the driving surface changes (e.g., increases or decreases), the probability associated with the previous lane in which the machine was traveling can begin to decrease. Additionally, the probability associated with the new lane represented by first direction output 118(1) can begin to increase and / or the probability associated with the new lane represented by second direction output 118(2) can begin to increase.

[0053] For example, returning reference toFIG. 4 In the example of FIG. 4, as the lane component 114 continues to process the additional data and if the machine 202 continues to travel within the lane 206(2), the lane component 114 can continue to generate and / or output the directional outputs 402(l)-402(2). Further, as the machine 202 continues to travel within the lane 206(2), the second probability 406(2) associated with the second lane 404(2) corresponding to the lane 206(2) of the travel surface 202 can continue to increase, while the probability 406(l) and / or the probabilities 406(3)-406(6) can continue to decrease. Further, the fourth probability 410(4) associated with the fourth lane 408(4) corresponding to the lane 206(2) of the travel surface 202 can also continue to increase, while the probabilities 410(l)-410(3) and the probabilities 410(5)-410(6) can continue to decrease.

[0054] Referring back to FIG. 1 In the example of FIG. 4, as the lane component 114 continues to process the additional data and if the machine 202 continues to travel within the lane 206(2), the lane component 114 can continue to generate and / or output the directional outputs 402(l)-402(2). Further, as the machine 202 continues to travel within the lane 206(2), the second probability 406(2) associated with the second lane 404(2) corresponding to the lane 206(2) of the travel surface 202 can continue to increase, while the probability 406(l) and / or the probabilities 406(3)-406(6) can continue to decrease. Further, the fourth probability 410(4) associated with the fourth lane 408(4) corresponding to the lane 206(2) of the travel surface 202 can also continue to increase, while the probabilities 410(l)-410(3) and the probabilities 410(5)-410(6) can continue to decrease.

[0055] The lane component 114 can then use this additional switch data 128 (or change data 128) in generating and / or updating the output data 116. For the first example, if the switch data 128 indicates that the machine switched lanes in a particular direction to a new lane, the lane component 114 can change the probabilities associated with the output data 116. For example, the lane component 114 can change the probabilities such that the probabilities associated with the previous lane in which the machine was traveling are now associated with the new lane in which the machine is traveling. For the second example, if the switch data 128 indicates a probability that the machine switched lanes in a particular direction to a new lane, the lane component 114 can use this probability in updating the probabilities associated with the output data 116. While these are just a few example techniques by which the lane component 114 can use switch data 128 associated with switching lanes to update the output data 118, in other examples, the lane component 114 can use additional and / or alternative techniques to update the probabilities based on the switch data 128.

[0056] For example, FIGS. 5A-5B An example is shown of updating the directional outputs 402(1)-402(2) associated with lane selection based at least on the machine 202 switching lanes, in accordance with some embodiments of the present disclosure. As shown in the example of FIG. 4, the machine 202 can switch from traveling in lane 206(2) to traveling in lane 206(1), which is represented by switch 502. Accordingly, the switch component 124 can perform one or more processes described herein to detect the switch 502 of lanes 206 and / or determine a probability that the switch 502 of lanes 206 occurred. Further, the switch component 124 can generate and / or output data (e.g., switch data 128) representing that the switch 502 occurred and / or a probability that the switch 502 occurred. FIG. 5A For example, FIG. 2 As shown in the example of FIG. 4, the machine 202 can switch from traveling in lane 206(2) to traveling in lane 206(1), which is represented by switch 502. Accordingly, the switch component 124 can perform one or more processes described herein to detect the switch 502 of lanes 206 and / or determine a probability that the switch 502 of lanes 206 occurred. Further, the switch component 124 can generate and / or output data (e.g., switch data 128) representing that the switch 502 occurred and / or a probability that the switch 502 occurred.

[0057] Accordingly, as FIG. 5BAs shown in the example, lane component 114 can use data output by switching component 124 and / or additional data (e.g., additional output data 108) to update the percentages 406(1)-406(6) and 410(1)-410(6) associated with direction outputs 402(1)-402(2). For example, as shown, the first direction output 504(1) (which may represent the updated first direction output 402(1)) may include the updated probabilities 506(1)-506(6) associated with lanes 404(1)-404(6). For example, the first direction output 504(1) can indicate the first probability 506(1) that the machine 202 is in the first lane 404(1), the second probability 506(2) that the machine 202 is in the second lane 404(2), the third probability 506(3) that the machine 202 is in the third lane 404(3), the fourth probability 506(4) that the machine 202 is in the fourth lane 404(4), the fifth probability 506(5) that the machine 202 is in the fifth lane 404(5), and the sixth probability 506(6) that the machine 202 is in the sixth lane 404(6).

[0058] Furthermore, the second directional output 504(2) (which may represent the updated second directional output 402(2)) may include the updated probabilities 508(1)-508(6) associated with lanes 408(1)-408(6). For example, the second directional output 504(2) may indicate a first probability 508(1) of machine 202 being in the first lane 408(1), a second probability 508(2) of machine 202 being in the second lane 408(2), a third probability 508(3) of machine 202 being in the third lane 408(3), a fourth probability 508(4) of machine 202 being in the fourth lane 408(4), a fifth probability 508(5) of machine 202 being in the fifth lane 408(5), and a sixth probability 508(6) of machine 202 being in the sixth lane 408(6).

[0059] As described herein, in some examples, at least based on the lane switching by machine 202, lane component 114 may “change” one or more of probabilities 406(1)-406(6) to determine one or more of probabilities 506(1)-506(6). For example, a first probability 506(1) may be determined at least based on a second probability 406(2), a second probability 506(2) may be determined at least based on a third probability 406(3), a third probability 506(3) may be determined at least based on a fourth probability 406(4), a fourth probability 506(4) may be determined at least based on a fifth probability 406(5), and a fifth probability 506(5) may be determined at least based on a sixth probability 406(6). As described herein, a probability may be determined at least based on another probability, which is at least based on probabilities that include other probabilities and / or include other probabilities, but may be updated based on other data.

[0060] Furthermore, based at least on the lane switching by machine 202, lane component 114 can “change” one or more of probabilities 410(1)-410(6) to determine one or more of probabilities 508(1)-508(6). For example, a second probability 508(2) can be determined at least based on a first probability 410(1), a third probability 508(3) can be determined at least based on a second probability 410(2), a fourth probability 508(4) can be determined at least based on a third probability 410(3), a fifth probability 508(5) can be determined at least based on a fourth probability 410(4), and a sixth probability 508(6) can be determined at least based on a fifth probability 410(5). As described herein, a probability can be determined at least based on another probability, which is at least based on probabilities that include other probabilities and / or include other probabilities, but will be updated based on other data.

[0061] Therefore, the first probability 506(1) associated with the first lane 404(1) corresponding to lane 206(1) may include the highest probability among probabilities 506(1)-506(6) because the machine 202 is located in lane 206(1). Furthermore, the fifth probability 508(5) associated with the fifth lane 408(5) corresponding to lane 206(1) may include the highest probability among probabilities 508(1)-508(6) because the machine 202 is again located in lane 206(1). However, the first probability 506(1) may include a higher probability than the fifth probability 508(5) because, based on sensor data, it is easier to detect that the machine 202 is located on the side 208(1) closer to the driving surface 202 than on the side 208(2) of the driving surface 202. In some examples, this may be because, based on the position of the machine 202, the perception system 102 can detect the position of the surface boundary 304(1) associated with one side 208(1) of the driving surface 202 more accurately than detecting the position of the surface boundary 304(2) associated with one side 208(2) of the driving surface 202.

[0062] Return to reference FIG. 1 For example, process 100 may include: one or more control components 124 of the machine using at least selection data 122 to determine one or more actions the machine will perform. For example, control component 124 may determine one or more controls the machine will perform (e.g., change speed, turn, continue straight, etc.), one or more trajectories the machine will travel, one or more plans for the machine's future travel, one or more safety measures to be taken, and / or any other type of action associated with the machine. As described herein, control component 124 may use the selected lane to determine one or more actions of the machine. For example, if the machine is to turn right and the machine is currently in the right lane, control component 124 may determine a trajectory that includes only the machine making the right turn. However, if the machine needs to turn right but is in another lane, control component 124 may determine a trajectory that includes the machine initially changing lanes to enter the right lane.

[0063] Now for reference FIG. 6 and FIG. 7 Each block of methods 600 and 700 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, individual functions can be implemented by a processor executing instructions stored in memory. Methods 600 and 700 can also be embodied as computer-usable instructions stored on a computer storage medium. Methods 600 and 700 can be provided by a standalone application, service, or managed service (alone or in combination with another managed service) or plug-in to another product, to name a few. Furthermore, by way of example, regarding... FIG. 1Methods 600 and 700 are described. However, methods 600 and 700 may be performed additionally or alternatively by any system or any combination of systems, including but not limited to those systems described herein.

[0064] FIG. 6 A flowchart of a method 600 for determining the lane a machine is traveling in using direction vectors, according to some embodiments of the present disclosure, is shown. At block B602, method 600 may include: obtaining sensor data generated using one or more sensors of the machine, the sensor data representing one or more lanes of a driving surface in the environment. For example, perception system 102 may obtain sensor data 104 generated using the machine. As described herein, sensor data 104 may include, but is not limited to, image data, LiDAR data, RADAR data, ultrasonic data, and / or any other type of sensor data. In some examples, perception system 102 may then process at least a portion of sensor data 104 and generate output data 108 associated with the driving surface and / or lane, at least based on this processing.

[0065] At box B604, method 600 may include: determining, at least based on sensor data, a first probability vector indexed from a first side of the driving surface and a second probability vector indexed from a second side of the driving surface. For example, lane assembly 114 may process output data 108 (and / or, in some examples, at least a portion of sensor data 104 and / or map data 106), and generate output data 116 at least based on this processing. Output data 116 may include: a first direction output 118(1) including the first probability vector indexed from the first side of the driving surface and a second direction output 118(2) including the second probability vector indexed from the second side of the driving surface.

[0066] At box B606, method 600 may include determining that the machine is in a lane among one or more lanes, based at least on a first probability vector and a second probability vector. For example, selection component 120 may process output data 116, such as a first direction output 118(1) and a second direction output 118(2). Based at least on this processing, selection component 120 may generate and / or output selection data 122 representing the lane in which the machine is traveling. As described herein, in some examples, selection component 120 may use one or more techniques to determine the lane, such as by selecting the lane associated with the highest probability and / or selecting the lane associated with a probability that satisfies a threshold probability.

[0067] At box B608, method 600 may include causing the machine to perform one or more operations, at least based on the machine being within the lane. For example, control component 124 may determine one or more operations, at least based on the lane the machine is traveling in. As described herein, determining one or more operations may include: determining one or more controls, determining one or more trajectories, determining one or more safety measures, and / or determining any other type of operation associated with the machine. Control component 124 may then cause the machine to perform operations, such as by causing the machine to travel along the trajectory.

[0068] FIG. 7 A flowchart of a method 700 for determining the lane a machine is traveling in using a direction output, according to some embodiments of the present disclosure, is shown. At block B702, method 700 may include: obtaining sensor data generated using one or more sensors of the machine, the sensor data representing a driving surface in the environment. For example, perception system 102 may obtain sensor data 104 generated using the machine. As described herein, sensor data 104 may include, but is not limited to, image data, LiDAR data, RADAR data, ultrasonic data, and / or any other type of data. In some examples, perception system 102 may then process at least a portion of sensor data 104 and generate output data 108 associated with the driving surface and / or lane, at least based on the processing.

[0069] At box B704, method 700 may include: determining, at least based on sensor data, a first output indicating first information associated with one or more first lanes of the driving surface and a second output indicating second information associated with one or more second lanes of the driving surface. For example, lane component 114 may process output data 108 (and / or, in some examples, at least a portion of sensor data 104 and / or map data 106), and generate output data 116 at least based on this processing. The output data 116 may include: a first direction output 118(1) including the first information associated with the first lane and a second direction output 118(2) including the second information associated with the second lane. In some examples, the first lanes include the second lanes, while in other examples, at least one of the first lanes is different from at least one of the second lanes.

[0070] At block B706, method 700 may include causing the machine to perform one or more operations based at least on a first output and a second output. For example, lane assembly 114 may use at least the first and second outputs to determine the lane the machine is traveling in. Control assembly 124 may then determine one or more operations based at least on the lane the machine is traveling in. As described herein, determining one or more operations may include determining one or more controls, determining one or more trajectories, determining one or more safety measures, and / or determining any other type of operation associated with the machine. Control assembly 124 may then cause the machine to perform operations, such as by causing the machine to travel along a trajectory.

[0071] Example autonomous vehicles

[0072] FIG. 8A This is an illustration of an example autonomous vehicle 800 according to some embodiments of the present disclosure. The autonomous vehicle 800 (or, alternatively, referred to herein as “vehicle 800”) may include, but is not limited to, passenger vehicles such as cars, trucks, buses, first response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police vehicles, ambulances, boats, construction vehicles, underwater vessels, robotic vehicles, drones, aircraft, vehicles coupled to trailers (e.g., semi-trailer trucks for hauling goods) and / or other types of vehicles (e.g., driverless and / or vehicles accommodating one or more passengers). Autonomous vehicles are typically described according to the levels of automation defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) in its "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 800 may be able to perform one or more functions corresponding to Levels 3 through 5 of the autonomous driving level. Vehicle 800 may be able to perform one or more functions according to Levels 1 through 5 of the autonomous driving level. For example, depending on the embodiment, vehicle 800 may be able to perform driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). As used herein, the term “autonomy” can include any and / or all types of autonomy for the vehicle 800 or other machines, such as full autonomy, high autonomy, conditional autonomy, partial autonomy, providing auxiliary autonomy, semi-autonomy, primary autonomy, or other names.

[0073] Vehicle 800 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. Vehicle 800 may include a propulsion system 850, such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. Propulsion system 850 may be connected to the drivetrain of vehicle 800, which may include a transmission, to enable propulsion of vehicle 800. Propulsion system 850 may be controlled in response to receiving a signal from throttle / accelerator 852.

[0074] A steering system 854, which may include a steering wheel, can be used to steer the vehicle 800 (e.g., along a desired path or route) when the propulsion system 850 is operating (e.g., when the vehicle is in motion). The steering system 854 may receive signals from the steering actuator 856. For fully automatic (level 5) functionality, the steering wheel may be optional.

[0075] The brake sensor system 846 can be used to operate the vehicle brakes in response to receiving signals from the brake actuator 848 and / or the brake sensor.

[0076] It can include one or more System-on-a-Chip (SoC) 804 ( FIG. 8C One or more controllers 836, including and / or one or more GPUs, can provide signals (e.g., signals representing commands) to one or more components and / or systems of vehicle 800. For example, one or more controllers can send signals to operate vehicle brakes via one or more brake actuators 848, to operate steering system 854 via one or more steering actuators 856, and to operate propulsion system 850 via one or more throttles / accelerators 852. One or more controllers 836 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving vehicle 800. One or more controllers 836 may include a first controller 836 for autonomous driving functions, a second controller 836 for functional safety functions, a third controller 836 for artificial intelligence functions (e.g., computer vision), a fourth controller 836 for infotainment functions, a fifth controller 836 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 836 can handle two or more of the functions described above, and two or more controllers 836 can handle a single function, and / or any combination thereof.

[0077] One or more controllers 836 may provide signals for controlling one or more components and / or systems of vehicle 800 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example, but not limited to, Global Navigation Satellite System (“GNSS”) sensors 858 (e.g., Global Positioning System sensors), RADAR sensors 860, ultrasonic sensors 862, LIDAR sensors 864, inertial measurement unit (IMU) sensors 866 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 896, stereo cameras 868, wide-angle cameras 870 (e.g., fisheye cameras), infrared cameras 872, surround cameras 874 (e.g., 360-degree cameras), long-range and / or medium-range cameras 898, speed sensors 844 (e.g., for measuring the rate of vehicle 800), vibration sensors 842, steering sensors 840, braking sensors (e.g., as part of braking sensor system 846), and / or other sensor types.

[0078] One or more controllers 836 may receive inputs (e.g., represented by input data) from the instrument cluster 832 of the vehicle 800 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 834, an auditory signaling device, a speaker, and / or via other components of the vehicle 800. These outputs may include information such as vehicle speed, rate, time, map data (e.g., [missing information]). FIG. 8C Information such as high-definition (“HD”) maps 822, location data (e.g., the location of vehicle 800 on the map), orientation, and the location of other vehicles (e.g., occupying grids), as well as information about objects and their states perceived by controller 836. For example, HMI display 834 may display information about the existence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, leaving 34B in two miles, etc.).

[0079] The vehicle 800 also includes a network interface 824, which can communicate via one or more networks using one or more wireless antennas 826 and / or a modem. For example, the network interface 824 may be able to communicate via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multicarrier (“CDMA2000”), etc. One or more wireless antennas 826 may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using one or more local area networks such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc., and / or one or more low-power wide area networks (“LPWAN”) such as LoRaWAN, SigFox, etc.

[0080] FIG. 8B For use in accordance with some embodiments of this disclosure FIG. 8A This is an example of the camera position and field of view of an autonomous vehicle 800. The camera and its respective field of view are an example embodiment and are not intended to be limiting. For example, additional and / or replaceable cameras may be included, and / or these cameras may be located at different positions on the vehicle 800.

[0081] The camera type used for the camera may include, but is not limited to, a digital camera suitable for use with components and / or systems of vehicle 800. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The camera may be able to use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red-white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a sharp-pixel camera, such as a camera with RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to improve light sensitivity.

[0082] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. One or more of the cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0083] One or more of the cameras can be mounted in mounting components such as custom-designed (3D-printed) components to cut off stray light and reflections from inside the vehicle (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capabilities. Regarding the wing mirror mounting components, the wing mirror components can be custom-3D printed so that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side-view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0084] A camera with a field of view that includes the environment in front of the vehicle 800 (e.g., a front-facing camera) can be used for surround view to help identify forward paths and obstacles, and, with the assistance of one or more controllers 836 and / or control SoCs, to provide information crucial for generating an occupancy grid and / or determining a preferred vehicle path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used in ADAS functions and systems, including Lane Departure Warning (“LDW”), Autonomous Cruise Control (“ACC”), and / or other functions such as traffic sign recognition.

[0085] A variety of cameras can be used in front-facing configurations, including, for example, monocular camera platforms including complementary metal-oxide-semiconductor (“CMOS”) color imagers. Another example could be a wide-angle camera 870, which can be used to perceive objects entering the field of view from the periphery (such as pedestrians, traffic at intersections, or bicycles). Although FIG. 8B The middle image shows only one wide-angle camera, but any number (including zero) of wide-angle cameras 870 can be present on vehicle 800. Furthermore, any number of remote cameras 898 (e.g., long-view stereo camera pairs) can be used for depth-based object detection, especially for objects for which neural networks have not yet been trained. Remote cameras 898 can also be used for object detection and classification, as well as basic object tracking.

[0086] Any number of stereo cameras 868 may also be included in the front-mounted configuration. In at least one embodiment, one or more stereo cameras 868 may include an integrated control unit comprising a scalable processing unit that can provide a multi-core microprocessor and programmable logic (“FPGA”) with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. Alternative stereo cameras 868 may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip capable of measuring the distance from the vehicle to a target object and using the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 868 may be used in addition to those described herein or alternatively.

[0087] Cameras with a field of view including the side portion of the vehicle 800 (e.g., side-view cameras) can be used for surround view, providing information for creating and updating occupancy grids and generating side-impact collision warnings. For example, surround camera 874 (e.g., ... FIG. 8B The four surround cameras 874 shown can be mounted on the vehicle 800. The surround cameras 874 can include wide-angle cameras 870, fisheye cameras, 360-degree cameras, and / or the like. Four examples are provided; the four fisheye cameras can be positioned at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 874 (e.g., left, right, and rear) and can utilize one or more other cameras (e.g., forward-facing cameras) as a fourth surround-view camera.

[0088] A camera with a field of view that includes the environment behind the vehicle 800 (e.g., a rear-view camera) can be used for assisted parking, surround view, rear collision warning, and creating and updating occupancy grids. A wide variety of cameras can be used, including but not limited to those also suitable as front-facing cameras as described herein (e.g., long-range and / or mid-range camera 898, stereo camera 868, infrared camera 872, etc.).

[0089] FIG. 8C For use in accordance with some embodiments of this disclosure FIG. 8AThe example autonomous vehicle 800 is illustrated in the block diagram of an example system architecture. It should be understood that this arrangement, and other arrangements described herein, are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities, which may be implemented as discrete or distributed components or in combination with other components, and in any suitable combination and location. The various functions described herein as being performed by these entities can be implemented via hardware, firmware, and / or software. For example, the various functions can be implemented by a processor executing instructions stored in memory.

[0090] FIG. 8C Each component, feature, and system in vehicle 800 is illustrated as being connected via bus 802. Bus 802 may include a Controller Area Network (CAN) data interface (or, alternatively, referred to herein as the "CAN bus"). CAN may be a network within vehicle 800 used to assist in the control of various features and functions of vehicle 800, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, etc. The CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus can be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0091] Although bus 802 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet may be used in addition to or alternatively to a CAN bus. Furthermore, although bus 802 is represented by a single line, this is not intended to be limiting. For example, any number of buses 802 may exist, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 802 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 802 may be used for collision avoidance functions, and a second bus 802 may be used for drive control. In any example, each bus 802 may communicate with any component of vehicle 800, and two or more buses 802 may communicate with the same component. In some examples, each SoC 804, each controller 836, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors of vehicle 800) and may be connected to a common bus such as a CAN bus.

[0092] Vehicle 800 may include one or more controllers 836, such as those described herein. FIG. 8A The controllers described herein. Controller 836 can be used for a wide variety of functions. Controller 836 can be coupled to any other different components and systems of vehicle 800 and can be used for the control of vehicle 800, artificial intelligence of vehicle 800, infotainment and / or the like for vehicle 800.

[0093] Vehicle 800 may include one or more System-on-Chip (SoC) 804s. SoC 804 may include a CPU 806, GPU 808, processor 810, cache 812, accelerator 814, data storage 816, and / or other components and features not shown. SoC 804 can be used to control vehicle 800 across a wide variety of platforms and systems. For example, one or more SoCs 804s may be combined with an HD map 822 in a system (e.g., the system of vehicle 800), the HD map being transmitted via a network interface 824 from one or more servers (e.g., [server name missing]). FIG. 8D One or more servers (878) receive map refresh and / or updates.

[0094] The CPU 806 may include CPU clusters or CPU complexes (or, alternatively, referred to herein as "CCPLEX"). The CPU 806 may include multiple cores and / or L2 cache. For example, in some embodiments, the CPU 806 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 806 may include four dual-core clusters, each with a dedicated L2 cache (e.g., 2MB L2 cache). The CPU 806 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, such that any combination of clusters of the CPU 806 can be active at any given time.

[0095] The CPU 806 can implement power management capabilities including one or more of the following features: automatic clock gating of hardware blocks when idle to conserve dynamic power; clock gating of each core when the core is not actively executing instructions due to the execution of WFI / WFE instructions; independent power gating of each core; independent clock gating of each core cluster when all cores are clock-gated or power-gated; and / or independent power gating of each core cluster when all cores are power-gated. The CPU 806 can further implement enhanced algorithms for managing power states, where allowed power states and desired wake-up times are specified, and the hardware / microcode determines the optimal power state to enter for the core, cluster, and CCPLEX. The processing core can support simplified power state entry sequences in software, with this work offloaded to the microcode.

[0096] The GPU 808 may include an integrated GPU (or, alternatively, referred to herein as an "iGPU"). The GPU 808 may be programmable and efficient for parallel workloads. In some examples, the GPU 808 may use an enhanced tensor instruction set. The GPU 808 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with at least 96KB of storage capacity), and two or more of these streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage capacity). In some embodiments, the GPU 808 may include at least eight streaming microprocessors. The GPU 808 may use a computation application programming interface (API). Furthermore, the GPU 808 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0097] In automotive and embedded applications, the GPU 808 can be power-optimized for optimal performance. For example, the GPU 808 can be fabricated on FinFETs. However, this is not intended to be limiting, and the GPU 808 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can combine several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor cores for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, dispatch units, and / or a 64KB register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to leverage the mixture of computation and addressing computations for efficient execution of workloads. The streaming microprocessor can include independent thread scheduling capabilities to allow for finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors can include a combination of L1 data cache and shared memory units to improve performance while simplifying programming.

[0098] The GPU 808 may include, in some examples, a High Bandwidth Memory (HBM) and / or a 16GB HBM2 memory subsystem providing a peak memory bandwidth of approximately 900GB / s. In some examples, in addition to HBM memory or alternatively, Synchronous Graphics Random Access Memory (SGRAM), such as Generation 5 Graphics Double Data Rate Synchronous Random Access Memory (GDDR5), may be used.

[0099] The GPU 808 may include unified memory technology, which includes access counters to allow memory pages to be migrated more precisely to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, Address Translation Service (ATS) support can be used to allow the GPU 808 to directly access the CPU 806 page tables. In such examples, when the GPU 808 Memory Management Unit (MMU) experiences a miss, the address translation request can be transferred to the CPU 806. In response, the CPU 806 can look up the virtual-physical mapping for the address in its page tables and transfer the translation back to the GPU 808. Thus, unified memory technology can allow a single unified virtual address space for the memory of both the CPU 806 and GPU 808, simplifying GPU 808 programming and porting applications to the GPU 808.

[0100] In addition, the GPU 808 may include access counters that track how frequently the GPU 808 accesses the memory of other processors. Access counters can help ensure that memory pages are moved to the physical memory of the processor that accesses those pages most frequently.

[0101] SoC 804 may include any number of caches 812, including those described herein. For example, cache 812 may include an L3 cache available to both CPU 806 and GPU 808 (e.g., it is connected to both CPU 806 and GPU 808). Cache 812 may include a write-back cache, which can track the state of rows, for example, using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache may include 4 MB or more, but a smaller cache size may also be used.

[0102] SoC 804 may include an arithmetic logic unit (ALU) that can be utilized in processing of any of the various tasks or operations performed with respect to vehicle 800, such as processing a DNN. Furthermore, SoC 804 may include a floating-point unit (FPU) (or other mathematical coprocessor or digital coprocessor type) for performing mathematical operations within the system. For example, SoC 804 may include one or more FPUs integrated as execution units within CPU 806 and / or GPU 808.

[0103] SoC 804 may include one or more accelerators 814 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, SoC 804 may include a hardware accelerator cluster, which may include optimized hardware accelerators and / or large on-chip memory. This large on-chip memory (e.g., 4MB SRAM) can enable the hardware accelerator cluster to accelerate neural networks and other computations. The hardware accelerator cluster can be used to complement GPU 808 and offload some tasks from GPU 808 (e.g., freeing up more cycles of GPU 808 to perform other tasks). As an example, accelerator 814 can be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to be easily controlled for acceleration. When used herein, the term "CNN" can include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0104] Accelerator 814 (e.g., a hardware accelerator cluster) may include a Deep Learning Accelerator (DLA). A DLA may include one or more Tensor Processing Units (TPUs) that can be configured to provide an additional trillion operations per second for deep learning applications and inference. TPUs may be accelerators configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and optimized for performing image processing functions. DLAs may be further optimized for a specific set of neural network types and floating-point operations as well as inference. DLAs are designed to provide higher performance per millimeter than general-purpose GPUs and significantly outperform CPUs. TPUs can perform several functions, including single-instance convolution functions, support for INT8, INT16, and FP16 data types for both features and weights, and post-processor functions.

[0105] DLA can execute neural networks, especially CNNs, quickly and efficiently on processed or unprocessed data for any function across a wide variety of applications, such as, but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and recognition using data from microphones; CNNs for face recognition and vehicle owner recognition using data from camera sensors; and / or CNNs for safety and / or safety-related events.

[0106] The DLA can perform any function of the GPU 808, and by using inference accelerators, for example, a designer can make either the DLA or the GPU 808 target any function. For example, a designer can focus the CNN processing and floating-point operations on the DLA and leave other functions to the GPU 808 and / or other accelerators 814.

[0107] Accelerator 814 (e.g., a hardware accelerator cluster) may include a programmable vision accelerator (PVA), which may alternatively be referred to herein as a computer vision accelerator. A PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. A PVA can provide a balance between performance and flexibility. For example, each PVA may include, for example, but not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0108] RISC cores can interact with image sensors (such as the image sensor of any camera described herein), image signal processors, and / or the like. Each of these RISC cores may include any amount of memory. Depending on the embodiment, the RISC core may use any of several protocols. In some examples, the RISC core may execute a real-time operating system (RTOS). RISC cores may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0109] DMA enables PVA components to access system memory independently of the CPU 806. DMA can support any number of features to provide optimizations to the PVA, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which can include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0110] A vector processor can be a programmable processor designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as, for example, a Single Instruction Multiple Data (SIMD) or Very Long Instruction Word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and speed.

[0111] Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. As a result, in some examples, each of the vector processors may be configured to execute independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even different algorithms on a sequence of images or portions of an image. Among other things, any number of PVAs may be included in a cluster of hardware accelerators, and any number of vector processors may be included in each of these PVAs. Furthermore, the PVA may include additional error correction code (ECC) memory to enhance overall system security.

[0112] Accelerator 814 (e.g., a hardware accelerator cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for accelerator 814. In some examples, on-chip memory may include at least 4MB of SRAM consisting of, for example, but not limited to, eight field-configurable memory blocks, accessible by both the PVA and DLA. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA may access memory via a backbone that provides high-speed memory access to the PVA and DLA. The backbone may include (e.g., using an APB) an on-chip computer vision network that interconnects the PVA and DLA to memory.

[0113] On-chip computer vision networks can include interfaces that ensure both the PVA and DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such interfaces can provide separate phases and channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0114] In some examples, SoC 804 may include, for example, a real-time ray tracing hardware accelerator as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. This real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the location and extent of objects (e.g., within a world model) to generate real-time visualization simulations for RADAR signal interpretation, sound propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison with LiDAR data for localization and / or other functional purposes, and / or other uses. In some embodiments, one or more Tree Traversal Units (TTUs) may be used to perform one or more ray tracing-related operations.

[0115] Accelerators 814 (e.g., hardware accelerator clusters) have broad applications in autonomous driving. PVAs can be programmable vision accelerators used in critical processing stages of ADAS and autonomous vehicles. The capabilities of PVAs are a good match for algorithmic domains requiring predictable processing, low power, and low latency. In other words, PVAs perform well in semi-dense or dense rule computation, even on small datasets requiring predictable runtimes with low latency and low power. Therefore, in the context of platforms for autonomous vehicles, PVAs are designed to run classical computer vision algorithms because they are efficient in object detection and integer mathematical operations.

[0116] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. In some examples, semi-global matching-based algorithms may be used, but this is not intended to be limiting. Many applications for Level 3-5 autonomous driving require instantaneous motion estimation / stereo matching (e.g., from moving structures, pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions on input from two monocular cameras.

[0117] In some examples, PVA can be used to perform intensive optical flow, providing processed RADAR data from the raw RADAR data (e.g., using 4D Fast Fourier Transform). In other examples, PVA is used for time-of-flight depth processing, which, for example, involves processing raw time-of-flight data to provide processed time-of-flight data.

[0118] DLA can be used to run any type of network to enhance control and driving safety, including, for example, neural networks that output a confidence metric for each object detection. Such a confidence value can be interpreted as a probability or as providing a relative “weight” for each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a threshold for the confidence and only consider detections exceeding the threshold as true positives. In an Automatic Emergency Braking (AEB) system, false positives can cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. DLA can run a neural network to regress the confidence value. This neural network can take at least a subset of parameters as input, such as bounding box dimensions, ground plane estimates (e.g., from another subsystem), inertial measurement unit (IMU) sensor 866 outputs related to vehicle orientation and distance, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LiDAR sensor 864 or RADAR sensor 860), etc.

[0119] The SoC 804 may include one or more data storage units 816 (e.g., memory). The data storage unit 816 may be on-chip memory of the SoC 804, which may store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and security, the data storage unit 816 may be large enough to store multiple instances of the neural network. The data storage unit 812 may include an L2 or L3 cache 812. References to the data storage unit 816 may include references to memory associated with the PVA, DLA, and / or other accelerators 814 as described herein.

[0120] SoC 804 may include one or more processors 810 (e.g., embedded processors). Processor 810 may include a startup and power management processor, which may be a dedicated processor and subsystem for handling startup power and management functions, as well as safety implementation. The startup and power management processor may be part of the SoC 804 startup sequence and may provide runtime power management services. The startup power and management processor may provide clock and voltage programming, auxiliary system low-power state transitions, SoC 804 thermal and temperature sensor management, and / or SoC 804 power state management. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and SoC 804 may use the ring oscillator to detect the temperature of CPU 806, GPU 808, and / or accelerator 814. If it is determined that the temperature exceeds a threshold, the startup and power management processor may enter a temperature fault routine and place SoC 804 into a lower power state and / or place vehicle 800 into a driver-safe parking mode (e.g., safely stop vehicle 800).

[0121] The processor 810 may also include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio via multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor and dedicated RAM.

[0122] The processor 810 may also include an always-on-processor engine that can provide the necessary hardware features to support low-power sensor management and wake-up use cases. This always-on-processor engine may include a processor core, tightly coupled RAM, support for peripherals (such as timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0123] The processor 810 may also include a security cluster engine, which comprises a dedicated processor subsystem for handling security management for automotive applications. The security cluster engine may include two or more processor cores, tightly coupled RAM, support for peripheral devices (e.g., timers, interrupt controllers, etc.), and / or routing logic. In secure mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic that detects any differences between their operations.

[0124] The processor 810 may also include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0125] The processor 810 may also include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0126] Processor 810 may include a video image compositer, which may be (e.g., implemented on a microprocessor) a processing block, implementing video post-processing functions required by the video playback application to generate the final image for the player window. The video image compositer may perform lens distortion correction on the wide-angle camera 870, the surround camera 874, and / or the in-cabin monitoring camera sensor. The in-cabin monitoring camera sensor is preferably monitored by a neural network running on another instance of an advanced SoC, configured to recognize in-cabin events and respond accordingly. The in-cabin system may perform lip reading to activate mobile phone services and make calls, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. Some functions are only available to the driver when the vehicle is operating in autonomous mode and are disabled in other situations.

[0127] Video image compositers can include enhanced temporal denoising for both spatial and temporal noise reduction. For example, in the case of motion in the video, denoising appropriately weights spatial information, reducing the weight of information provided by neighboring frames. In cases where the image or part of the image does not contain motion, the temporal denoising performed by the video image compositer can use information from previous images to reduce noise in the current image.

[0128] The video image compositer can also be configured to perform stereo correction on input stereo camera frames. When the operating system desktop is in use and the GPU 808 does not need to continuously render new surfaces, the video image compositer can be further used for user interface components. Even when the GPU 808 is powered on and active, performing 3D rendering, the video image compositer can be used to offload the GPU 808 to improve performance and responsiveness.

[0129] The SoC 804 may also include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and pixel-related input functions for receiving video and input from a camera. The SoC 804 may also include an input / output controller that can be software-controlled and can be used to receive I / O signals not assigned to a specific role.

[0130] SoC 804 may also include a wide range of peripheral interfaces to enable communication with peripherals, audio codecs, power management and / or other devices. SoC 804 can be used to process data from cameras and sensors (e.g., LIDAR sensor 864, RADAR sensor 860, etc., which can be connected via Gigabit Multimedia Serial Link and Ethernet), data from bus 802 (e.g., vehicle 800 speed, steering wheel position, etc.), and data from GNSS sensor 858 (connected via Ethernet or CAN bus). SoC 804 may also include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine and can be used to free up CPU 806 from routine data management tasks.

[0131] The SoC 804 can be an end-to-end platform with a flexible architecture spanning Automation Levels 3-5, providing a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS technologies for diversity and redundancy, along with deep learning tools to deliver a flexible and reliable driving software stack. The SoC 804 can be faster, more reliable, and even more energy- and space-efficient than conventional systems. For example, when combined with the CPU 806, GPU 808, and data storage 816, the accelerator 814 can provide a fast and efficient platform for Level 3-5 autonomous vehicles.

[0132] Therefore, this technology offers capabilities and functionalities that cannot be achieved through conventional systems. For example, computer vision algorithms can be executed on CPUs, which can be configured using high-level programming languages ​​such as C to execute a wide variety of processing algorithms across a diverse range of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for automotive ADAS applications and practical Level 3-5 autonomous vehicles.

[0133] In contrast to conventional systems, the techniques described in this paper, by providing CPU complexes, GPU complexes, and hardware accelerator clusters, allow multiple neural networks to be executed simultaneously and / or sequentially, and the results combined to achieve Level 3–5 autonomous driving capabilities. For example, a CNN executed on a DLA or dGPU (e.g., GPU 820) could include text and word recognition, allowing a supercomputer to read and understand traffic signs, including those for which neural networks have not yet been specifically trained. The DLA could also include a neural network capable of recognizing, interpreting, and providing semantic understanding of the signs, and passing that semantic understanding to a path planning module running on the CPU complex.

[0134] As another example, multiple neural networks can operate simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of "Caution: Flashing lights indicate icy conditions," along with a light, can be interpreted independently or jointly by several neural networks. The sign itself can be recognized as a traffic sign by a deployed first neural network (e.g., a trained neural network), and the text "Flashing lights indicate icy conditions" can be interpreted by a deployed second neural network that informs the vehicle's path planning software (preferably executing on a CPU complex) that icy conditions exist when the flashing lights are detected. The flashing lights can be identified by a deployed third neural network operating across multiple frames, informing the vehicle's path planning software of the presence (or absence) of the flashing lights. All three neural networks can operate simultaneously, for example, within a DLA and / or on a GPU 808.

[0135] In some examples, the CNN used for facial recognition and owner identification can use data from camera sensors to identify the presence of an authorized driver and / or owner of vehicle 800. A processing engine always on the sensors can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in safe mode, to disable the vehicle when the owner leaves. In this way, SoC 804 provides security against theft and / or carjacking.

[0136] In another example, the CNN used for emergency vehicle detection and identification can use data from microphone 896 to detect and identify emergency vehicle siren. In contrast to conventional systems that use a general classifier to detect siren and manually extract features, SoC 804 uses a CNN to classify environmental and urban sounds as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative shut-off rate of emergency vehicles (e.g., by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the localized area in which the vehicle operates, as identified by GNSS sensor 858. Thus, for example, when operating in Europe, the CNN will seek to detect European siren, and when operating in the United States, the CNN will seek to identify siren only in North America. Once an emergency vehicle is detected, with the assistance of ultrasonic sensor 862, a control program can be used to execute emergency vehicle safety routines, causing the vehicle to slow down, pull over to the side of the road, stop, and / or idle until the emergency vehicle passes.

[0137] The vehicle may include a CPU 818 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 804 via a high-speed interconnect (e.g., PCIe). The CPU 818 may include, for example, an x86 processor. The CPU 818 can be used to perform any of a wide variety of functions, including, for example, arbitrating the results of potential inconsistencies between ADAS sensors and the SoC 804, and / or monitoring the status and health of the controller 836 and / or the infotainment SoC 830.

[0138] Vehicle 800 may include a GPU 820 (e.g., a discrete GPU or dGPU) that can be coupled to SoC 804 via a high-speed interconnect (e.g., NVIDIA's NVLINK). GPU 820 may provide additional artificial intelligence capabilities, for example by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based at least in part on inputs (e.g., sensor data) from sensors of vehicle 800.

[0139] Vehicle 800 may also include a network interface 824, which may include one or more wireless antennas 826 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). Network interface 824 can be used to enable wireless connectivity via the Internet to the cloud (e.g., with server 878 and / or other network devices), with other vehicles, and / or with computing devices (e.g., passenger client devices). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across networks and via the Internet). A direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide vehicle 800 with information about vehicles approaching vehicle 800 (e.g., vehicles in front, to the side, and / or behind vehicle 800). This functionality can be part of vehicle 800's cooperative adaptive cruise control function.

[0140] Network interface 824 may include a SoC that provides modulation and demodulation functions and enables controller 836 to communicate over a wireless network. Network interface 824 may include an RF front-end for up-conversion from baseband to RF and down-conversion from RF to baseband. Frequency conversion can be performed using known processes and / or using a superheterodyne process. In some examples, the RF front-end functionality may be provided by a separate chip. The network interface may include wireless functions for communication over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0141] Vehicle 800 may also include data storage 828, which may include off-chip (e.g., off-chip SoC 804) storage devices. Data storage 828 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0142] The vehicle 800 may also include a GNSS sensor 858. The GNSS sensor 858 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used for auxiliary mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 858 can be used, including, for example, but not limited to, GPS using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0143] Vehicle 800 may also include a RADAR sensor 860. The RADAR sensor 860 can be used by vehicle 800 for remote vehicle detection even in dark and / or inclement weather conditions. The RADAR functional safety level may be ASIL B. The RADAR sensor 860 can use CAN and / or bus 802 (e.g., to transmit data generated by the RADAR sensor 860) for control and access to object tracking data, and in some examples, Ethernet access for accessing raw data. A wide variety of RADAR sensor types can be used. For example, and without limitation, the RADAR sensor 860 can be adapted for front, rear, and side RADAR use. In some examples, a pulse Doppler RADAR sensor is used.

[0144] The RADAR sensor 860 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In some examples, the long-range RADAR can be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view (e.g., within 250m) achieved through two or more independent scans. The RADAR sensor 860 can help distinguish between stationary and moving objects and can be used by ADAS systems for emergency braking assist and forward collision warning. The long-range RADAR sensor can include a single-site multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record the vehicle 800's surroundings at higher rates with minimal traffic interference from adjacent lanes. The other two antennas can extend the field of view, enabling rapid detection of vehicles entering or leaving the vehicle 800's lane.

[0145] As an example, a mid-range RADAR system can include a range of up to 860m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 850 degrees (rear). Short-range RADAR systems can include, but are not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor blind spots behind and beside the vehicle.

[0146] Short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0147] Vehicle 800 may also include ultrasonic sensors 862. Ultrasonic sensors 862, which may be positioned at the front, rear, and / or sides of vehicle 800, can be used for parking assistance and / or creating and updating occupancy grids. A wide variety of ultrasonic sensors 862 can be used, and different ultrasonic sensors 862 can be used for different detection ranges (e.g., 2.5m, 4m). Ultrasonic sensors 862 can operate at functional safety level ASIL B.

[0148] Vehicle 800 may include a LIDAR sensor 864. The LIDAR sensor 864 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensor 864 may be of functional safety level ASIL B. In some examples, vehicle 800 may include multiple LIDAR sensors 864 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0149] In some examples, the LiDAR sensor 864 may be able to provide a list of objects and their distances within a 360-degree field of view. Commercially available LiDAR sensors 864 may have an advertising range of, for example, approximately 800m, with an accuracy of 2cm-3cm, and support for 800Mbps Ethernet connectivity. In some examples, one or more non-protruding LiDAR sensors 864 may be used. In such examples, the LiDAR sensor 864 may be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of a vehicle 800. In such examples, the LiDAR sensor 864 may provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, even for low-reflectivity objects, with a range of 200m. Front-mounted LiDAR sensors 864 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0150] In some examples, LiDAR technologies such as 3D flash LiDAR can also be used. 3D flash LiDAR uses a flash of laser light as the emission source to illuminate the vehicle's surroundings up to approximately 200 meters. A flash LiDAR unit includes a receiver that records the laser pulse propagation time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LiDAR allows for the generation of highly accurate and distortion-free images of the surrounding environment using each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of the vehicle. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras (e.g., non-scanning LiDAR devices) without moving parts other than a fan. Flash LiDAR devices can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-registered intensity data. By using a flash LiDAR, and because a flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 864 is less susceptible to motion blur, vibration, and / or shock.

[0151] The vehicle may also include an IMU sensor 866. In some examples, the IMU sensor 866 may be located at the center of the rear axle of the vehicle 800. The IMU sensor 866 may include, for example, but not limited to, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 866 may include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 866 may include an accelerometer, a gyroscope, and a magnetometer.

[0152] In some embodiments, the IMU sensor 866 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filter algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 866 can enable the vehicle 800 to estimate heading by directly observing and correlating velocity changes from GPS to the IMU sensor 866 without input from a magnetic sensor. In some examples, the IMU sensor 866 and the GNSS sensor 858 can be combined into a single integrated unit.

[0153] The vehicle may include a microphone 896 placed in and / or around the vehicle 800. Among other things, the microphone 896 may be used for emergency vehicle detection and identification.

[0154] The vehicle may also include any number of camera types, including stereo camera 868, wide-angle camera 870, infrared camera 872, surround camera 874, long-range and / or mid-range camera 898, and / or other camera types. These cameras can be used to capture image data around the entire perimeter of the vehicle 800. The types of cameras used depend on the embodiment and the requirements of the vehicle 800, and any combination of camera types can be used to provide the necessary coverage around the vehicle 800. Furthermore, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As an example and without limitation, these cameras may support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described herein with respect to... FIG. 8A and FIG. 8B It was described in more detail.

[0155] Vehicle 800 may also include vibration sensor 842. Vibration sensor 842 can measure vibrations of vehicle components such as axles. For example, changes in vibration can indicate changes in the road surface. In another example, when two or more vibration sensors 842 are used, differences between vibrations can be used to determine friction or slippage on the road surface (e.g., when there is a vibration difference between a power drive shaft and a free-rotating shaft).

[0156] Vehicle 800 may include ADAS system 838. In some examples, ADAS system 838 may include SoC. ADAS system 838 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic warning (RCTW), collision warning system (CWS), lane centering (LC) and / or other features and functions.

[0157] The ACC system can use a RADAR sensor 860, a LIDAR sensor 864, and / or a camera. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to vehicles immediately in front of vehicle 800 and automatically adjusts the vehicle speed to maintain a safe distance. Lateral ACC performs distance holding and, if necessary, advises vehicle 800 to change lanes. Lateral ACC is associated with other ADAS applications such as LCA and CWS.

[0158] CACC uses information from other vehicles, which can be received indirectly from other vehicles via a wireless link or network connection (e.g., via the Internet) through network interface 824 and / or wireless antenna 826. Direct links can be provided by vehicle-to-vehicle (V2V) communication links, while indirect links can be infrastructure-to-vehicle (I2V) communication links. Typically, the V2V communication concept provides information about vehicles immediately ahead (e.g., vehicles immediately in front of vehicle 800 and in the same lane), while the I2V communication concept provides information about traffic further ahead. A CACC system can include either or both of these I2V and V2V information sources. Given information about vehicles ahead of vehicle 800, CACC can be more reliable, and it has the potential to improve traffic flow and reduce road congestion.

[0159] The Forward-Looking Warning (FCW) system is designed to alert the driver to hazards, enabling the driver to take corrective action. The FCW system uses a front-facing camera and / or RADAR sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components. The FCW system can provide warnings in the form of, for example, audible, visual, haptic, and / or rapid braking pulses.

[0160] An AEB (Autonomous Emergency Braking) system detects an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system can use a front-facing camera and / or RADAR sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes to attempt to prevent or at least mitigate the effects of the predicted collision. The AEB system may include technologies such as dynamic brake support and / or collision proximity braking.

[0161] The Lane Departure Warning (LDW) system provides visual, auditory, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle crosses lane markings. When the driver indicates intentional lane departure, the LDW system is deactivated by activating a turn signal. The LDW system can utilize a front-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0162] The LKA system is a variation of the LDW system. If vehicle 800 begins to leave the lane, the LKA system provides corrective steering input or braking to vehicle 800.

[0163] The BSW system detects and warns the driver of vehicles in the vehicle's blind spot. The BSW system can provide visual, auditory, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses turn signals. The BSW system can utilize a rear-facing camera and / or RADAR sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0164] RCTW systems can provide visual, auditory, and / or tactile notifications when an object is detected outside the range of a rear-view camera while the vehicle is reversing. Some RCTW systems include AEB (Autonomous Emergency Braking) to ensure the application of the vehicle's brakes to avoid a collision. RCTW systems may use one or more rear-view RADAR sensors 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0165] Conventional ADAS systems can be prone to false positives, which can be annoying and distracting for the driver, but typically not catastrophic, as they alert the driver and allow them to determine whether a safe condition truly exists and take appropriate action. However, in an autonomous vehicle 800, in the event of conflicting results, the vehicle 800 itself must decide whether to heed the results from the main computer or auxiliary computer (e.g., the first controller 836 or the second controller 836). For example, in some embodiments, the ADAS system 838 may be a backup and / or auxiliary computer used to provide perception information to a backup computer rationality module. The backup computer rationality monitor may run redundant and varied software on hardware components to detect faults in perception and dynamic driving tasks. Outputs from the ADAS system 838 may be provided to a supervisory MCU. If the outputs from the main computer and the auxiliary computer conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0166] In some examples, the master computer can be configured to provide a confidence score to the supervisory MCU, indicating the master computer's confidence level in the selected result. If the confidence score exceeds a threshold, the supervisory MCU can follow the master computer's direction regardless of whether the auxiliary computer provides conflicting or inconsistent results. If the confidence score does not meet the threshold and the master and auxiliary computers indicate different results (e.g., conflict), the supervisory MCU can arbitrate between these computers to determine the appropriate result.

[0167] The supervisory MCU can be configured to run a neural network trained and configured to determine, at least in part, the conditions under which the auxiliary computer provides a false alarm, based on outputs from both the host and auxiliary computers. Thus, the neural network in the supervisory MCU can learn when the output of the auxiliary computer can be trusted and when it cannot. For example, when the auxiliary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying a metallic object that is not actually dangerous, such as a drain grid or manhole cover that triggers an alarm. Similarly, when the auxiliary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU may include at least one of a DLA or GPU suitable for running the neural network using associated memory. In a preferred embodiment, the supervisory MCU may include components of and / or be included as components of the SoC 804.

[0168] In other examples, the ADAS system 838 may include an auxiliary computer that performs ADAS functions using conventional computer vision rules. This allows the auxiliary computer to use classic computer vision rules (if-then), and the presence of neural networks in the supervising MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identity make the entire system more fault-tolerant, especially for failures caused by software (or software-hardware interface) functionality. For instance, if a software vulnerability or bug exists in the software running on the host computer and non-identical software code running on the auxiliary computer provides the same overall result, the supervising MCU can be more confident that the overall result is correct and that the vulnerability in the software or hardware on the host computer does not cause a substantial error.

[0169] In some examples, the output of ADAS system 838 can be fed to the perception block and / or the dynamic driving task block of the main computer. For example, if ADAS system 838 issues a forward collision warning because an object is immediately in front, the perception block can use this information when identifying the object. In other examples, the assistance computer can have its own neural network, which is trained and thus reduces the risk of false positives as described herein.

[0170] Vehicle 800 may also include an infotainment SoC 830 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 830 may include a combination of hardware and software that can be used to provide vehicle 800 with audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.) and / or information services (e.g., navigation system, rear parking assistance, radio data system, vehicle-related information such as fuel level, total coverage distance, brake fuel level, fuel level, door opening / closing, air filter information, etc.). For example, the infotainment SoC 830 may include a radio, disc player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, WiFi, steering wheel audio controls, hands-free voice controls, head-up display (HUD), HMI display 834, telematics device, control panel (e.g., for controlling and / or interacting with various components, features, and / or systems) and / or other components. The infotainment SoC 830 may further be used to provide information (e.g., visual and / or auditory) to the vehicle's users, such as information from the ADAS system 838, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0171] The infotainment SoC 830 may include GPU functionality. The infotainment SoC 830 can communicate with other devices, systems, and / or components of the vehicle 800 via bus 802 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 830 may be coupled to a supervisory MCU, allowing the GPU of the infotainment system to perform autonomous driving functions in the event of a failure of the main controller 836 (e.g., the primary and / or backup computer of the vehicle 800). In such an example, the infotainment SoC 830 may place the vehicle 800 into a driver-safe parking mode as described herein.

[0172] Vehicle 800 may also include an instrument cluster 832 (e.g., a digital instrument panel, electronic instrument cluster, digital instrument panel, etc.). The instrument cluster 832 may include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 832 may include a set of instruments such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seatbelt warning light, parking brake warning light, engine malfunction indicator, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 830 and the instrument cluster 832. In other words, the instrument cluster 832 may be included as part of the infotainment SoC 830, or vice versa.

[0173] FIG. 8D For cloud-based servers and according to some embodiments of this disclosure FIG. 8A This is a system diagram illustrating communication between example autonomous vehicles 800. System 876 may include server 878, network 890, and vehicles including vehicle 800. Server 878 may include multiple GPUs 884(A)-884(H) (collectively referred to herein as GPU 884), PCIe switches 882(A)-882(H) (collectively referred to herein as PCIe switch 882), and / or CPUs 880(A)-880(B) (collectively referred to herein as CPU 880). GPUs 884, CPUs 880, and PCIe switches may be interconnected with high-speed interconnects and / or PCIe connections 886, such as, but not limited to, NVLink interface 888 developed by NVIDIA. In some examples, GPUs 884 are connected via NVLink and / or NVSwitch SoCs, and GPUs 884 and PCIe switches 882 are connected via PCIe interconnects. Although eight GPUs 884, two CPUs 880, and two PCIe switches are shown in the diagram, this is not intended to be limiting. Depending on the embodiment, each of the servers 878 may include any number of GPUs 884, CPUs 880, and / or PCIe switches. For example, each of the servers 878 may include eight, sixteen, thirty-two, and / or more GPUs 884.

[0174] Server 878 can receive image data from vehicles via network 890, representing images of unexpected or changed road conditions such as recently commenced roadworks. Server 878 can also transmit neural network 892, updated neural network 892, and / or map information 894, including information about traffic and road conditions, to vehicles via network 890. Updates to map information 894 may include updates to HD map 822, such as information about construction sites, potholes, bends, floods, or other obstacles. In some examples, neural network 892, updated neural network 892, and / or map information 894 may have been generated from new training and / or data received from any number of vehicles in the environment, and / or based on experience gained from training performed at a data center (e.g., using server 878 and / or other servers).

[0175] Server 878 can be used to train machine learning models (e.g., neural networks) based on training data. Training data can be generated by the vehicle and / or generated in a simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., where the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not labeled and / or preprocessed (e.g., where the neural network does not require supervised learning). Training can be performed according to any one or more classes of machine learning techniques, including but not limited to: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, it can be used by the vehicle (e.g., transmitted to the vehicle via network 890), and / or the machine learning model can be used by server 878 to remotely monitor the vehicle.

[0176] In some examples, server 878 can receive data from vehicles and apply that data to state-of-the-art real-time neural networks for real-time intelligent inference. Server 878 may include a deep learning supercomputer powered by GPU 884 and / or a dedicated AI computer, such as the DGX and DGX station machines developed by NVIDIA. However, in some examples, server 878 may include a deep learning infrastructure in a data center that uses only CPU power.

[0177] The deep learning infrastructure of server 878 is capable of rapid, real-time inference and can be used to assess and verify the health of the processor, software, and / or associated hardware in vehicle 800. For example, the deep learning infrastructure can receive periodic updates from vehicle 800, such as image sequences and / or objects located within those image sequences by vehicle 800 (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them to those identified by vehicle 800. If the results do not match and the infrastructure concludes that the AI ​​in vehicle 800 has malfunctioned, then server 878 can transmit a signal to vehicle 800 instructing its fail-safe computer to take control, notify passengers, and complete a safe stopping operation.

[0178] For inference, server 878 may include GPU 884 and one or more programmable inference accelerators (such as NVIDIA's TensorRT). The combination of a GPU-powered server and inference acceleration enables real-time response. In other examples, such as where performance is less critical, CPU, FPGA, and other processor-powered servers can be used for inference.

[0179] Example computing device

[0180] FIG. 9 This is a block diagram of an example computing device 900 suitable for implementing some embodiments of the present disclosure. The computing device 900 may include an interconnect system 902 directly or indirectly coupled to: a memory 904, one or more central processing units (CPUs) 906, one or more graphics processing units (GPUs) 908, a communication interface 910, input / output (I / O) ports 912, input / output components 914, a power supply 916, one or more presentation components 918 (e.g., one or more displays), and one or more logic units 920. In at least one embodiment, one or more computing devices 900 may include one or more virtual machines (VMs), and / or any of their components may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 908 may include one or more vGPUs, one or more CPUs 906 may include one or more vCPUs, and / or one or more logic units 920 may include one or more virtual logic units. Thus, one or more computing devices 900 may include discrete components (e.g., a full GPU dedicated to computing device 900), virtual components (e.g., a portion of the GPU dedicated to computing device 900), or a combination thereof.

[0181] althoughFIG. 9 The various blocks are shown as connected via interconnect system 902 using lines, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 918 (such as a display device) may be considered I / O component 914 (e.g., if the display is a touchscreen). As another example, CPU 906 and / or GPU 908 may include memory (e.g., memory 904 may represent a storage device other than the memory of GPU 908, CPU 906, and / or other components). In other words, FIG. 9 The computing devices described are for illustrative purposes only. No distinction is made between such categories as “workstation,” “server,” “laptop computer,” “desktop computer,” “tablet computer,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, as all are considered within the scope of… FIG. 9 Within the scope of computing devices.

[0182] Interconnect system 902 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 902 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Fast Peripheral Component Interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 906 may be directly connected to memory 904. Further, CPU 906 may be directly connected to GPU 908. In cases where there is a direct or point-to-point connection between components, interconnect system 902 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required to be included in computing device 900.

[0183] The memory 904 may include any computer-readable medium from a variety of computer-readable media. A computer-readable medium may be any available medium accessible by the computing device 900. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0184] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 904 may store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, Digital Universal Disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 900. As used herein, computer storage media does not include the signal itself.

[0185] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and includes any information transmission medium. The term "modulated data signal" can refer to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the above should also be included within the scope of computer-readable media.

[0186] CPU 906 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. Each CPU 906 may contain one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling numerous software threads simultaneously. CPU 906 may contain any type of processor and may contain different types of processors depending on the type of computing device 900 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 900, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors (such as math coprocessors), computing device 900 may also include one or more CPUs 906.

[0187] In addition to or in lieu of one or more CPUs 906, one or more GPUs 908 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. One or more GPUs 908 may be integrated GPUs (e.g., having one or more CPUs 906) and / or one or more GPUs 908 may be discrete GPUs. In embodiments, one or more GPUs 908 may be coprocessors of one or more CPUs 906. GPUs 908 may be used by computing device 900 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 908 may be used for general-purpose computing on a GPU (GPGPU). GPUs 908 may include hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. GPUs 908 may generate pixel data for an output image in response to rendering commands (e.g., rendering commands received from CPUs 906 via a host interface). GPU 908 may include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 904. GPU 908 may include two or more GPUs operating in parallel (e.g., via links). The links may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 908 may produce pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0188] In addition to or in lieu of CPU 906 and / or GPU 908, logic unit 920 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 906, one or more GPUs 908, and / or one or more logic units 920 may execute any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 920 may be a portion of one or more CPUs 906 and / or GPUs 908 and / or integrated into one or more CPUs 906 and / or GPUs 908, and / or one or more logic units 920 may be discrete components or otherwise external to CPUs 906 and / or GPUs 908. In embodiments, one or more logic units 920 may be coprocessors of one or more CPUs 906 and / or GPUs 908.

[0189] Examples of logic unit 920 include one or more processing cores and / or components thereof, such as data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree lateral unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU), application-specific integrated circuit (ASIC), floating-point unit (FPU), input / output (I / O) element, peripheral component interconnect (PCI) or fast peripheral component interconnect (PCIe) element, etc.

[0190] The communication interface 910 may include one or more receivers, transmitters, and / or transceivers enabling the computing device 900 to communicate with other computing devices via electronic communication networks (including wired and / or wireless communications). The communication interface 910 may include components and functions for enabling communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or wirelessband), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, one or more logic units 920 and / or the communication interface 910 may include one or more data processing units (DPUs) to directly transmit data received via a network and / or via interconnect system 902 to one or more GPUs 908 (e.g., memory).

[0191] I / O port 912 enables computing device 900 to be logically coupled to other devices including I / O component 914, one or more presentation components 918, and / or other components, some of which may be built into (e.g., integrated into) computing device 900. Illustrative I / O component 914 includes microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dish antennas, scanners, printers, wireless devices, etc. I / O component 914 can provide a natural user interface (NUI) that processes aerial gestures, voice, or other physiological input generated by the user. In some cases, input may be transmitted to appropriate network elements for further processing. The NUI can implement any combination of voice recognition, pen recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, aerial gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of computing device 900. Computing device 900 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. Additionally, the computing device 900 may include an accelerometer or gyroscope that enables motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the computing device 900 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0192] Power supply 916 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 916 may provide power to computing device 900 so that the components of computing device 900 can operate.

[0193] The presentation component 918 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 918 may receive data from other components (e.g., GPU 908, CPU 906, DPU, etc.) and output the data (e.g., as images, videos, sounds, etc.).

[0194] Example Data Center

[0195] FIG. 10 An example data center 1000 that may be used in at least one embodiment of this disclosure is shown. The data center 1000 may include a data center infrastructure layer 1010, a framework layer 1020, a software layer 1030, and / or an application layer 1040.

[0196] like FIG. 10As shown, the data center infrastructure layer 1010 may include a resource coordinator 1012, grouped computing resources 1014, and node computing resources (“nodes CR”) 1016(1)-1016(N), where “N” represents any complete positive integer. In at least one embodiment, nodes CR 1016(1)-1016(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules and / or cooling modules, etc. In some embodiments, one or more node CRs from nodes CR 1016(1)-1016(N) may correspond to servers having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CR1016(1)-10161(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more of nodes CR1016(1)-1016(N) may correspond to virtual machines (VMs).

[0197] In at least one embodiment, the grouped computing resources 1014 may include individual groups of nodes CR1016 housed within one or more racks (not shown), or a plurality of racks housed within a data center in different geographical locations (also not shown). Individual groups of nodes CR1016 within the grouped computing resources 1014 may include grouped computing, networking, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, a plurality of nodes CR1016, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0198] Resource coordinator 1012 may be configured or otherwise control one or more nodes CRs 1016(1)-1016(N) and / or grouped computing resources 1014. In at least one embodiment, resource coordinator 1012 may include a Software Design Infrastructure (SDI) management entity for data center 1000. Resource coordinator 1012 may include hardware, software, or some combination thereof.

[0199] In at least one embodiment, such as FIG. 10As shown, framework layer 1020 may include job scheduler 1033, configuration manager 1034, resource manager 1036, and / or distributed file system 1038. Framework layer 1020 may include a framework of software 1032 supporting software layer 1030 and / or one or more applications 1042 supporting application layer 1040. Software 1032 or application 1042 may respectively contain web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 1020 may be, but is not limited to, a free and open-source software web application framework (such as Apache Spark™ (hereinafter “Spark”)) that can leverage distributed file system 1038 for large-scale data processing (e.g., “big data”). In at least one embodiment, job scheduler 1033 may include Spark drivers to facilitate the scheduling of workloads supported by different layers of data center 1000. Configuration manager 1034 may be able to configure different layers, such as software layer 1030 and framework layer 1020 (which includes Spark and distributed file system 1038 for supporting large-scale data processing). Resource manager 1036 may be able to manage clusters or groups of computing resources mapped to distributed file system 1038 and job scheduler 1032 or allocated to support clusters or groups of distributed file system 1038 and job scheduler 1033. In at least one embodiment, clusters or groups of computing resources may include grouped computing resources 1014 in data center infrastructure layer 1010. Resource manager 1036 may coordinate with resource coordinator 1012 to manage these mapped or allocated computing resources.

[0200] In at least one embodiment, the software 1032 included in the software layer 1030 may include software used in at least a portion of the nodes CRs 1016(1)-1016(N), the grouped computing resources 1014, and / or the distributed file system 1038 of the framework layer 1020. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0201] In at least one embodiment, the application 1042 included in the application layer 1040 may include one or more types of applications used at least in part by nodes CRs 1016(1)-1016(N), grouped computing resources 1014, and / or the distributed file system 1038 of the framework layer 1020. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in combination with one or more embodiments.

[0202] In at least one embodiment, any of the configuration manager 1034, resource manager 1036, and resource coordinator 1012 can implement any number and type of self-modification actions based on any amount and type of data obtained in any technically feasible manner. Self-modification actions can free the data center operator of data center 1000 from making potentially poor configuration decisions and may prevent underutilization and / or poor performance of the data center.

[0203] According to one or more embodiments described herein, data center 1000 may include tools, services, software, or other resources to train one or more machine learning models or to use one or more machine learning models to predict or infer information. For example, one or more machine learning models can be trained by using the software and / or computing resources described above with respect to data center 1000 to compute weight parameters according to a neural network architecture. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks can be used to infer or predict information using the resources described above with respect to data center 1000 by using weight parameters computed through one or more training techniques (such as, but not limited to, those described herein).

[0204] In at least one embodiment, the data center 1000 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the software and / or hardware resources described above may be configured to allow a user to train or perform services that infer information, such as image recognition, speech recognition, or other artificial intelligence services.

[0205] Example network environment

[0206] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... FIG. 9 It is implemented on one or more instances of one or more computing devices 900—for example, each device may include similar components, features, and / or functions of one or more computing devices 900. Furthermore, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of a data center 1000, examples of which are described in this document. FIG. 10 To describe in more detail.

[0207] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0208] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein for the server can be implemented on any number of client devices.

[0209] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework supporting software at the software layer and / or application at the application layer. The software or application may respectively include network-based service software or applications. In embodiments, one or more client devices may use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open-source software network application framework that can use a distributed file system for large-scale data processing (e.g., "big data").

[0210] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., distributed across one or more data centers at the state, region, country, global, etc.). The core server may assign at least a portion of the functionality to the edge server if the connection to the user (e.g., a client device) is relatively close to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0211] One or more client devices may include the information described in this article. FIG. 9 At least some of the components, features, and functions of one or more example computing devices 900 described. By way of example and not limitation, the client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0212] This disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, which are executed by a computer or other machine such as a personal digital assistant or other handheld device. Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a wide variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. This disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices linked via a communication network.

[0213] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0214] This document describes in detail the subject matter of this disclosure to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the discloser has envisioned that the claimed subject matter may be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of the method employed, these terms should not be construed as suggesting any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

[0215] Example paragraph

[0216] A: A method comprising: obtaining sensor data generated using one or more sensors of a machine, the sensor data representing one or more lanes of a driving surface in an environment; calculating a first vector based at least on the sensor data, the first vector representing one or more first probabilities from a first side of the driving surface that the machine is located within the one or more lanes; calculating a second vector based at least on the sensor data, the second vector representing one or more second probabilities from a second side of the driving surface other than the first side of the driving surface that the machine is located within the one or more lanes; positioning the machine in a lane of the one or more lanes based at least on the first vector and the second vector; and causing the machine to perform one or more operations based at least on the machine being in the lane.

[0217] B. The method as described in paragraph A, wherein: the one or more lanes include at least the lane located on the first side near the driving surface and the second lane located on the second side near the driving surface; the first vector includes: a first element associated with the first lane, followed by a second element associated with the second lane; and the second vector includes at least: a first element associated with the second lane, followed by a second element associated with the first lane.

[0218] C. The method as described in paragraph B, wherein: the one or more first probabilities include at least a first probability associated with the first element of the first vector and a second probability associated with the second element of the first vector; and the one or more second probabilities include at least a third probability associated with the first element of the second vector and a fourth probability associated with the second element of the second vector.

[0219] D. The method as described in any one of paragraphs A, C, and D, further comprising: obtaining second sensor data generated using the one or more sensors of the machine, the second sensor data representing the one or more lanes of the driving surface within the environment; determining a third vector by updating the one or more first probabilities to include one or more third probabilities and determining a fourth vector by updating the one or more second probabilities to include one or more fourth probabilities, at least based on the second sensor data; and determining at least one of the lanes or the second lane in the one or more lanes in which the machine is driving, at least based on the third vector and the fourth vector.

[0220] E. The method as described in any one of paragraphs A, D, and E, further comprising: determining that the machine switches from the lane to a second lane among the one or more lanes; determining a third vector by updating the one or more first probabilities to include one or more third probabilities and determining a fourth vector by updating the one or more second probabilities to include one or more fourth probabilities, based at least on data instructing the machine to switch from the lane to the second lane; and determining the second lane among the one or more lanes in which the machine is traveling, based at least on the third vector and the fourth vector.

[0221] F. The method as described in any one of paragraphs A, E, further comprises: processing the sensor data based on at least one or more machine learning models, determining a first output indicating at least one of one or more lane boundaries or one or more road boundaries, and a second output indicating one or more locations of the one or more lanes, wherein determining the first vector and the second vector is based at least on the first output and the second output.

[0222] G. A system comprising: one or more processors, the one or more processors being configured to: determine, based at least on sensor data obtained using one or more sensors of a machine, a first output indicating, from a first side of a driving surface, one or more first probabilities, that the machine is located within one or more lanes, and a second output indicating, from a second side of the driving surface, one or more second probabilities that the machine is located within the one or more lanes; and to cause the machine to perform one or more operations based at least on the first output and the second output.

[0223] H. The system as described in paragraph G, wherein the one or more processors are further configured to: determine, based at least on the first output and the second output, that the machine is located within a lane of the one or more lanes, wherein, based at least on the machine being located within the lane, the machine is caused to perform the one or more operations.

[0224] I. The system as described in either paragraph G or paragraph H, wherein: the one or more first probabilities indicated by the first output are indexed starting from a first lane among the one or more lanes located on the first side of the driving surface; and the one or more second probabilities indicated by the second output are indexed starting from a second lane among the one or more lanes located on the second side of the driving surface.

[0225] J. The system as described in any one of paragraphs GI, wherein: the one or more first probabilities include at least: a first probability associated with a first lane among the one or more lanes, followed by a second probability associated with a second lane among the one or more lanes; and the one or more second probabilities include at least: a third probability associated with the second lane, followed by a fourth probability associated with the first lane.

[0226] K. The system as described in any one of paragraphs GJ, wherein: the first output comprises a first vector having a first number of elements associated with the one or more lanes, wherein individual elements of the first number of elements are associated with individual probabilities of the one or more first probabilities; and the second output comprises a second vector having a second number of elements associated with the one or more lanes, wherein individual elements of the second number of elements are associated with individual probabilities of the one or more second probabilities.

[0227] L. The system as described in any one of paragraphs GK, wherein the one or more processors are further configured to: determine a third output by updating the one or more first probabilities to include one or more third probabilities, based at least on second sensor data obtained using the one or more sensors of the machine, and determine a fourth output by updating the one or more second probabilities to include one or more fourth probabilities.

[0228] M. The system as described in any one of paragraphs GL, wherein the one or more processors are further configured to: determine that the machine has changed lanes; and, at least based on the machine changing lanes, determine a third output by updating the one or more first probabilities to include one or more third probabilities and determine a fourth output by updating the one or more second probabilities to include one or more fourth probabilities.

[0229] N. The system as described in paragraph M, wherein: determining that the machine has switched lanes includes: determining the probability that the machine has switched lanes; and determining that the third output and the fourth output are based at least on the probability that the machine has switched lanes.

[0230] O. The system as described in any one of paragraphs GN, wherein the one or more processors are further configured to: process the sensor data based on at least one or more machine learning models, determine a third output indicating at least one of one or more lane boundaries or one or more road boundaries, and a fourth output indicating one or more locations of the one or more lanes, wherein the determination of the first output and the second output is based at least on the third output and the fourth output.

[0231] P. The system as described in any one of paragraphs GO, wherein the one or more processors are further configured to: determine the road type associated with the driving surface based at least on a map associated with the environment including the driving surface, wherein the determination of the first output and the second output is further based at least on the road type.

[0232] Q. A system as described in any one of paragraphs GP, wherein the system comprises at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more analog operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more visual language models (VLMs); a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system comprising one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0233] R. One or more processors, comprising: processing circuitry configured to cause the machine to perform one or more operations based at least on positioning the machine in a lane, wherein the lane is determined at least based on a first output indicating one or more first probabilities that the machine is located in one or more first lanes and a second output indicating one or more second probabilities that the machine is located in one or more second lanes, the first output being associated with a first side of a driving surface and the second output being associated with a second side of the driving surface.

[0234] S. One or more processors as described in paragraph R, wherein: the first output includes a first probability vector indexed starting from the first side of the driving surface; and the second output includes a second probability vector indexed starting from the second side of the driving surface.

[0235] T. One or more processors as described in any of paragraphs R or S, wherein the one or more processors are included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more analog operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more visual language models (VLMs); a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

Claims

1. A method comprising: Obtain sensor data generated using one or more sensors of the machine, the sensor data representing one or more lanes of a driving surface in the environment; A first vector is calculated based at least on the sensor data, the first vector representing one or more first probabilities from a first side of the driving surface that the machine is located in one or more lanes; as well as A second vector is calculated based at least on the sensor data, the second vector representing one or more second probabilities that the machine is located in one or more lanes from a second side of the driving surface that is different from the first side of the driving surface; Based at least on the first vector and the second vector, the machine is positioned in one or more lanes; as well as At least based on the machine being within the lane, the machine performs one or more operations.

2. The method of claim 1, wherein: The one or more lanes include at least the lane located on the first side close to the driving surface and the second lane located on the second side close to the driving surface; The first vector includes: a first element associated with the first lane, followed by a second element associated with the second lane; and The second vector includes at least a first element associated with the second lane, followed by a second element associated with the first lane.

3. The method of claim 2, wherein: The one or more first probabilities include at least a first probability associated with the first element of the first vector and a second probability associated with the second element of the first vector; and The one or more second probabilities include at least a third probability associated with the first element of the second vector and a fourth probability associated with the second element of the second vector.

4. The method of claim 1, further comprising: Obtain second sensor data generated using one or more sensors of the machine, the second sensor data representing one or more lanes of the driving surface within the environment; Based at least on the second sensor data, a third vector is determined by updating one or more first probabilities to include one or more third probabilities, and a fourth vector is determined by updating one or more second probabilities to include one or more fourth probabilities; as well as Based at least on the third vector and the fourth vector, determine at least one of the lanes or the second lane in one or more lanes in which the machine is traveling.

5. The method of claim 1, further comprising: Determine whether the machine switches from the lane to a second lane in one or more lanes; A third vector is determined by updating one or more first probabilities to include one or more third probabilities, and a fourth vector is determined by updating one or more second probabilities to include one or more fourth probabilities, based at least on data instructing the machine to switch from the first lane to the second lane; as well as The second lane is determined from the one or more lanes in which the machine is traveling, based at least on the third and fourth vectors.

6. The method of claim 1, further comprising: The sensor data is processed based on at least one or more machine learning models to determine a first output indicating at least one of one or more lane boundaries or one or more road boundaries, and a second output indicating one or more locations of the one or more lanes. The determination of the first vector and the second vector is based at least on the first output and the second output.

7. A system comprising: One or more processors, said one or more processors being used for: Based at least on sensor data obtained using one or more sensors of the machine, determine a first output indicating one or more first probabilities from a first side of the driving surface that the machine is located in one or more lanes, and a second output indicating one or more second probabilities from a second side of the driving surface that the machine is located in the one or more lanes; as well as The machine performs one or more operations based at least on the first output and the second output.

8. The system of claim 7, wherein the one or more processors are further configured to: Based at least on the first output and the second output, it is determined that the machine is located within one or more lanes. The machine is made to perform one or more operations, at least based on the fact that the machine is located within the lane.

9. The system of claim 7, wherein: The one or more first probabilities indicated by the first output are indexed starting from the first lane of the one or more lanes located on the first side of the driving surface; as well as The one or more second probabilities indicated by the second output are indexed starting from the second lane of the one or more lanes located on the second side near the driving surface.

10. The system of claim 7, wherein: The one or more first probabilities include at least: a first probability associated with a first lane in the one or more lanes, followed by a second probability associated with a second lane in the one or more lanes; and The one or more second probabilities include at least a third probability associated with the second lane, followed by a fourth probability associated with the first lane.

11. The system of claim 7, wherein: The first output includes a first vector having a first number of elements associated with the one or more lanes, wherein individual elements of the first number of elements are associated with individual probabilities of the one or more first probabilities; as well as The second output includes a second vector having a second number of elements associated with the one or more lanes, wherein individual elements of the second number of elements are associated with individual probabilities of the one or more second probabilities.

12. The system of claim 7, wherein the one or more processors are further configured to: determine a third output by updating the one or more first probabilities to include one or more third probabilities, based at least on second sensor data obtained using the one or more sensors of the machine, and determine a fourth output by updating the one or more second probabilities to include one or more fourth probabilities.

13. The system of claim 7, wherein the one or more processors are further configured to: It was determined that the machine had switched lanes; and At least based on the machine switching lanes, a third output is determined by updating one or more first probabilities to include one or more third probabilities, and a fourth output is determined by updating one or more second probabilities to include one or more fourth probabilities.

14. The system of claim 13, wherein: Determining that the machine switched lanes includes: determining the probability that the machine switched lanes; and The third and fourth outputs are determined based at least on the probability that the machine switched lanes.

15. The system of claim 7, wherein the one or more processors are further configured to: The sensor data is processed based on at least one or more machine learning models to determine a third output indicating at least one of one or more lane boundaries or one or more road boundaries, and a fourth output indicating one or more locations of the one or more lanes. The determination of the first output and the second output is based at least on the third output and the fourth output.

16. The system of claim 7, wherein the one or more processors are further configured to: The road type associated with the driving surface is determined, at least based on a map associated with the environment including the driving surface. The determination of the first output and the second output is further based at least on the road type.

17. The system of claim 7, wherein the system comprises at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing one or more conversational AI operations; A system for generating synthetic data; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

18. One or more processors, comprising: A processing circuit for causing the machine to perform one or more operations based at least on positioning the machine in a lane, wherein the lane is determined at least based on a first output indicating that the machine is located in one or more first lanes with a first probability and a second output indicating that the machine is located in one or more second lanes with a second probability, the first output being associated with a first side of a driving surface and the second output being associated with a second side of the driving surface.

19. One or more processors as claimed in claim 18, wherein: The first output includes a first probability vector indexed starting from the first side of the driving surface; and The second output includes a second probability vector indexed starting from the second side of the driving surface.

20. The one or more processors as claimed in claim 18, wherein the one or more processors are included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing one or more conversational AI operations; A system for generating synthetic data; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

Images