Speed limit fusion for automotive systems and applications

Abstract

The invention relates to speed limit fusion for automotive systems and applications. In various examples, the systems and methods of the present disclosure correspond to a speed limit fusion of automotive systems and applications. Systems and methods are disclosed that fuse speed limit data from one or more map sources (e.g., navigation maps, HD maps, SD maps, etc.) and a perceptual system, and that employ rules to help resolve speed limit conflicts for the perceptual system and maps. For example, if a collision occurs, the rule may indicate whether to use speed limit data from perceptual data, speed limit data from a map, and / or whether to use another type of speed limit data, such as cached speed limit data.

Description

Background Art

[0001] Vehicles (e.g., semi-autonomous vehicles and autonomous vehicles) typically rely on speed limit information for path planning, vehicle control, and / or other processes. In some systems, the speed limit information at the current location is determined using the vehicle's perception system. For example, a vehicle equipped with a perception system (e.g., cameras, Light Detection and Ranging (LiDAR), Radio Detection and Ranging (RADAR), etc.) can use one or more deep neural networks (DNNs) to process the perception data to determine the speed limit. In some cases, the vehicle can also obtain local speed limit data from a navigation map (“nav-map”), a Standard Definition (SD) map, a High Definition (HD) map, and / or other map types. An example of a navigation map is the Advanced Driver Assistance System Interface Specification (ADASIS), which is a prediction map-based driver assistance system. However, in some cases, the perception system and the navigation map relied on alone are not completely reliable sources of local speed limit data.

[0002] For example, the accuracy of the speed limit data in the perception system is limited by the training data and the underlying DNN architecture, and the perception system may not be able to correctly read certain road signs and / or the road signs may be blocked, damaged, etc., which may result in inaccurate perception information. As another example, the perception system may identify the vehicle as being in the wrong lane, e.g., a lane other than the lane in which the vehicle is traveling. Thus, any association of speed limit signs with the lane may be misinterpreted, and the vehicle may rely on the speed limit information to determine the wrong driving lane. The navigation map may also have certain limitations; for example, the speed limit data in the navigation map may be outdated, and some parts of a given city may not be mapped at all. Given these limitations of the perception system itself and the navigation map itself, there is a need to improve the accuracy of the speed limit data that may be used by an autonomous vehicle or by the driver of a semi-autonomous vehicle. Summary of the Invention

[0003] Embodiments of the present disclosure relate to speed limit fusion for automotive systems and applications. Systems and methods for fusing speed limit data from one or more map sources (e.g., navigation maps, HD maps, SD maps, etc.) and a perception system are disclosed, and the systems and methods employ rules to help resolve speed limit conflicts between the perception system and the map. For example, if a conflict occurs, the rules can indicate whether to use the speed limit data from the perception data, the speed limit data from the map, and / or whether to use another type of speed limit data, such as cached speed limit data.

[0004] Compared with traditional systems (such as the systems described above that use the perception system itself or the map itself to obtain local speed limit data), in some embodiments, the current system obtains speed limit data from two sources and then applies one or more rules to publish enhanced-accuracy speed limit data, which is then acted upon by an autonomous vehicle or provided to a vehicle driver. For example, for a speed limit sign, the current system can use the perception system to determine at least a first speed limit and use the map to determine a second speed limit. Then, the current system can use rules to determine a final speed limit using the first speed limit and the second speed limit. By using speed limit data from two sources to determine the final speed limit associated with a speed limit sign, the current system can more accurately determine the speed limit for a vehicle traveling in the environment in some embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The present system and method for speed limit fusion for automotive systems and applications will be described in detail below with reference to the accompanying drawings, in which:

[0006] Figure 1 FIG. 9 shows an example data flow diagram of a process for fusing data from multiple sources to determine a speed limit associated with a speed limit sign according to some embodiments of the present disclosure;

[0007] Figure 2A FIG. 13 is a diagram of an environmental map according to some embodiments of the present disclosure;

[0008] Figure 2B FIG. 17 is a diagram of an environment in which a vehicle equipped with a fusion component passes a first speed limit sign along a first direction on a first road according to some embodiments of the present disclosure;

[0009] Figure 2C FIG. 21 is according to some embodiments of the present disclosure and Figure 2B is similar to FIG. 21, except that it shows a Figure 2B vehicle traveling along a second direction on a second road, the vehicle being equipped with a fusion component;

[0010] Figure 2D FIG. 29 is a diagram of a second environment according to some embodiments of the present disclosure, in which Figure 2B the vehicle is shown traveling along a third direction on a third road, the vehicle being equipped with a fusion component;

[0011] Figures 3A to 3B FIG. 35 is an example of fusing speed limit data from a perception system with speed limit data associated with a map using one or more rules according to some embodiments of the present disclosure;

[0012] Figure 4 FIG. 39 is a flowchart providing an overview of algorithms performed by a fusion component and associated input and output units according to some embodiments of the present disclosure;

[0013] Figure 5 is a flowchart showing a series of processing operations represented by block 408 in Figure 4 ;

[0014] Figure 6 is a block diagram representing a rule book stored electronically in a memory unit according to some embodiments of the present disclosure;

[0015] Figure 7 is a flowchart showing a method for determining a final speed limit from a first speed limit based at least on sensor data and a second speed limit based on map data according to some embodiments of the present disclosure;

[0016] Figure 8A is an illustration of an example autonomous vehicle according to some embodiments of the present disclosure;

[0017] Figure 8B is according to some embodiments of the present disclosure Figure 8A an example of the camera positions and fields of view of an example autonomous vehicle;

[0018] Figure 8C is according to some embodiments of the present disclosure Figure 8A a block diagram of an example system architecture of an example autonomous vehicle;

[0019] Figure 8D is a system diagram for communicating between a cloud-based server and Figure 8A an example autonomous vehicle according to some embodiments of the present disclosure;

[0020] Figure 9 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0021] Figure 10 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] Systems and methods related to speed limit fusion for automotive systems and applications are disclosed. Although the present disclosure may be directed to example autonomous or semi-autonomous vehicle 800 (alternatively referred to herein as "vehicle 800", "ego vehicle 800", "machine 800", or "ego machine 800"), examples of which are directed to Figures 8A - 8Ddescribed), but this is not restrictive. For example, the systems and methods described herein can be used by non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more Advanced Driver Assistance Systems (ADAS)), autonomous vehicles, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, airships, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, airplanes, engineering vehicles, underwater vehicles, drones, and / or other vehicle types, but are not limited thereto. Additionally, although the present disclosure may be described with respect to obtaining speed limit data for autonomous or semi-autonomous vehicles, this is not intended to be limiting, and the systems and methods described herein can be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technical field where speed limit data (and / or other data types that can rely on two information sources) can be used for analysis and publication.

[0023] The systems and methods of the present disclosure can obtain speed limit data simultaneously from a perception system and a map (e.g., navigation map, HD map, SD map, Most Probable Path (MPP) map, etc.), and then apply one or more rules to publish speed limit data with enhanced accuracy that can be acted upon by a vehicle (e.g., autonomous vehicle, semi-autonomous vehicle, robot, machinery, etc.) and / or by a vehicle driver. Speed Limit Fusion (SLF) can include the process of collecting speed limits from two or more sources (e.g., both the perception system and the map), resolving any conflicts in the data collected from these two sources, and then publishing the most accurate speed limit representing the World Model (WM) lane of the vehicle's lane (also referred to as the "ego lane"). For example, SLF can rely on confidence values associated with the perceived speed limits and their age of use to resolve speed limit conflicts. In some non-limiting embodiments, SLF can publish speed limits only for lanes that are reachable by the ego vehicle, which include the ego lane, the bifurcations of the ego lane, and the immediate neighbor lanes of the ego lane. The speed limits are published in the ego space, with appropriate lane assignments and entry and exit offsets indicating the lane range to which the applicable speed limit applies.

[0024] For example, a system (e.g., a fusion component) can use the confidence value (score) of the speed limit from a perception system and the age of the speed limit to resolve speed limit conflicts. When comparing the speed limit data obtained from each source, the system can ensure that the speed limit data from each source corresponds to the same coordinate space, so that the speed limit data from these sources can be more simply compared. This assurance can be achieved by defining the coordinate space as an "offset", which is the cumulative distance of the vehicle (also referred to as the "ego vehicle" or "ego car") along its lane. The system can associate a lane with the offset controlled by a given speed limit and associate the lane with the speed limit in front of its own lane. The system can publish speed limits in the ego space only for lanes that the vehicle can reach (including its own lane, the forks of its own lane, and the adjacent neighbor lanes of its own lane). Therefore, the published speed limits can be associated with appropriate lane assignments and with the entry and exit offsets indicating the range of lanes to which the published speed limits apply.

[0025] For example, the system can use the perception system to determine a first speed limit associated with a lane and use the map to determine a second speed limit associated with the lane. Then, the system can determine whether the first speed limit corresponds to (e.g., matches) the second speed limit. At least based on determining that the first speed limit corresponds to the second speed limit, the system can determine that the final speed limit includes the first speed limit and / or the second speed limit. However, at least based on determining that the first speed limit does not correspond to (e.g., does not match) the second speed limit, the system can then use one or more rules to determine the final speed limit.

[0026] For a first example, if the system determines that the confidence of the first speed limit determined by perception meets (e.g., equals or is greater than) a threshold confidence, the system can determine to use the first speed limit as the final speed limit for the road. In this first example, the system can use one or more additional factors when making the determination, such as the age of the first speed limit (e.g., the time period since the perception system determined the first speed limit, the distance traveled by the vehicle since the perception system determined the first speed limit, etc.), the road type (e.g., highway, rural area, etc.), and / or any other information. For example, if the system determines that the time period since the perception system determined the first speed limit exceeds a certain threshold time period and / or the distance traveled by the vehicle since the perception system determined the first speed limit exceeds a certain distance threshold (e.g., offset), the system can determine to use the second speed limit as the final speed limit.

[0027] For a second example, if the system determines that the path of the vehicle has changed, such as the vehicle turning onto a new road, the system may determine to use the second speed limit as the final speed limit. In some examples, the system may determine to use the second speed limit until the sensing system determines a new speed limit (e.g., with a confidence that meets a threshold) while navigating along the new path. Although these are just a few example techniques that the system can use to determine the final speed limit when the first speed limit does not correspond to the second speed limit, in other examples, the system may use additional and / or alternative techniques.

[0028] The systems and methods described herein can be used for a variety of purposes, such as but not limited to machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or participant simulation, and / or digital twins, data center processing, conversational artificial intelligence (AI), light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, generative AI, (large) language models, and / or any other suitable applications.

[0029] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, sensing systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, boating systems, intelligent area surveillance systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems that include one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing light transport simulation, systems for performing collaborative content creation for 3D assets, systems implemented at least partially using cloud computing resources, systems for performing generative AI operations, systems that implement or use large language models (LLMs) to perform operations, and / or other types of systems.

[0030] Figure 1FIG. 0 shows an example data flow diagram of a process for fusing data from multiple sources to determine a speed limit associated with a speed limit sign. It should be understood that such arrangements and other arrangements described herein are presented only as examples. As supplements or alternatives to the arrangements and elements shown, other arrangements and elements (e.g., machines, interfaces, functions, sequences, function groupings, etc.) may be used, and some elements may be omitted entirely. In addition, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components, and may be implemented in any suitable combination and location. The various functions performed by the entities described herein may be executed by hardware, firmware, and / or software. For example, the various functions may be executed by a processor executing instructions stored in a memory. In some embodiments, the systems, methods, and processes described herein may use components, features, and / or functions similar to those of Figures 8A to 8D example autonomous vehicle 800, Figure 9 example computing device 900, and / or Figure 10 example data center 1000.

[0031] Process 100 may include a perception component 102 (e.g., a perception system) receiving sensor data 104 from one or more sensors 106. As described herein, sensors 106 may include, but are not limited to, image sensors (e.g., cameras), LiDAR sensors, RADAR sensors, position sensors (e.g., global positioning system (GPS) sensors, etc.), ultrasonic sensors, and / or any other type of sensor. Thus, sensor data 104 may include image data, video data, LiDAR data, RADAR data, position data (e.g., GPS data), ultrasonic data, and / or any other type of sensor data. The perception component 102 may include functions that perform object detection, segmentation (e.g., of objects, instances, etc.), classification, tracking, and / or other processes. For example, the perception component 102 may output data indicating detected lanes and boundaries on a driving road surface, detected drivable free space, detected poles or signs, detected traffic lights, detected traffic signs (e.g., speed limit signs), detected objects in the environment (e.g., vehicles, pedestrians, animals, inanimate objects, etc.), detected waiting conditions, and intersections, etc. In additional or alternative examples, the perception component 102 may generate data indicating one or more characteristics associated with the detected objects and / or the environment in which the objects are located. Characteristics associated with an object may include, but are not limited to, an x position (global and / or local position), a y position (global and / or local position), a z position (global and / or local position), an orientation (e.g., roll, pitch, yaw), an object classification (e.g., object type), a speed, an acceleration, a range (size) of the object, etc.

[0032] In those examples where the sensing component 102 performs detection, the sensing component 102 can generate data indicative of the detected objects in the image. Such detections can include two-dimensional (2D) and / or three-dimensional (3D) bounding shapes (e.g., bounding boxes, bounding cuboids, etc.), volumes, and / or masks of the detected objects. Additionally, in some examples, the data can indicate probabilities (e.g., confidence levels) associated with the objects, such as probabilities associated with the location of the object, the classification of the object, etc. In some examples, the detections of the sensing component 102 can use machine learning methods (e.g., Scale-Invariant Feature Transform (SIFT), Histogram of Oriented Gradients (HOG), etc.), and then use a Support Vector Machine (SVM) to classify the objects depicted in the image represented by the sensor data 106. Additionally, or alternatively, the detections can use deep learning methods based on Convolutional Neural Networks (CNNs) to classify the objects depicted in the image represented by the sensor data 106.

[0033] For example, if the sensor data 104 is image data representing one or more images depicting a speed limit sign, the sensing component 102 can process the image data to determine information associated with the speed limit sign. As described herein, the information can include at least the speed limit associated with the speed limit sign (referred to as the "first speed limit" or "perceived speed limit" in some examples), the probability associated with the first speed limit, and / or any other information. For example, if the speed limit sign is a 50 miles per hour (MPH) speed limit sign, the sensing component 102 can process the image data to detect the sign and / or the text "50 MPH" or "Speed Limit 50 MPH". Then, the sensing component 102 can use the detection to determine that the speed limit sign includes a 50 MPH speed limit sign. Additionally, the sensing component 102 can determine the probability that the speed limit sign includes a 50 MPH sign, such as 50%, 75%, 90%, 95%, 99%, etc.

[0034] The process 100 can include the sensing component 102 then generating and / or outputting perceived speed data 108 representative of the information associated with the speed limit sign. For example, using the example where the sensing component 102 detects a 50 MPH speed limit sign as described above, the perceived speed data 108 can indicate that the speed limit sign includes a 50 MPH speed limit sign, the speed zone associated with the environment in which the vehicle is traveling is 50 MPH, and / or the probability associated with the speed limit sign including a 50 MPH speed limit sign.

[0035] Process 100 may include processing component 110 processing map data 112 to determine another speed limit associated with a speed limit sign (referred to as "second speed limit" or "map-based speed limit" in some examples). For example, processing component 110 may use one or more processes to determine the pose of the vehicle in the environment. In some examples, processing component 110 uses at least a portion of sensor data 104 (e.g., position data generated by a position sensor (e.g., GPS data)) to determine the pose. Additionally, or alternatively, in some examples, processing component 110 uses additional sensor data 104 (e.g., through a positioning process) to determine and / or refine the pose. For example, processing component 110 may use image data, LiDAR data, RADAR data, etc. to determine and / or update the initial pose of the vehicle. For example, to improve the initial pose, processing component 110 may compare one or more features represented by the additional sensor data 104 with one or more features represented by the map data 112, where the map data 112 represents a map of the environment in which the vehicle is located. At least based on the comparison, processing component 110 may update the initial pose of the vehicle to an estimated pose (e.g., a positioning pose) of the vehicle in the environment. As described herein, the pose may represent the position of the vehicle (e.g., x-coordinate position, y-coordinate position, and / or z-coordinate position), the orientation of the vehicle (e.g., yaw, pitch, and / or roll), and / or any other position, pose, or orientation information.

[0036] Then, processing component 110 may use the determined pose to analyze the map data 112 to identify the speed limit sign represented by the map data 112. For example, processing component 110 may determine that the speed limit sign is on the path of the vehicle and / or associated with the lane in which the vehicle is traveling. Additionally, processing component 110 may use the map data 112 to determine a second speed limit sign. For example, as described herein, the map data 112 may represent information associated with the environment, such as a speed limit associated with a speed limit sign located within the environment. As shown, then, process 100 may include processing component 110 generating and / or outputting map speed data 114 representing at least the second speed limit associated with the speed limit sign.

[0037] Process 110 may include a fusion component 116 processing sensed speed data 108 and map speed data 114 and determining a final speed limit (e.g., a fused speed limit) associated with a speed limit sign based at least on that processing. For example, as described herein, the fusion component 116 may initially associate a first speed limit of a speed limit sign with a second speed limit of the speed limit sign. In some examples, the fusion component 116 may associate the speed limits at least based on the sensing component 102 and the processing component 110 indicating that the speed limit sign is associated with the same path (e.g., the same road and / or lane) on which the vehicle is traveling. In some examples, the fusion component 116 may associate the speed limits at least based on the first speed limit being the last speed limit determined by the sensing component 02 and the second speed limit being the last speed limit determined by the processing component 110. While these are just two example techniques of how the fusion component 116 may associate speed limits, in other examples, the fusion component 116 may use additional and / or alternative techniques to associate speed limits.

[0038] As described herein, the fusion component 116 may then use the sensed speed data 108, the map speed data 114, the age associated with the sensed speed data 108, road information (which may be represented by the map speed data 114), and / or one or more rules to determine a final speed limit using at least the first speed limit represented by the sensed speed data 108 and the second speed limit represented by the map speed data 114, where the rules are represented by rule data 118. For example, if the fusion component 116 determines that the first speed limit corresponds to (e.g., matches) the second speed limit, the fusion component 116 may use a first rule that the final speed limit includes the first speed limit and / or the second speed limit. However, if the fusion component 116 determines that the first speed limit does not correspond to (e.g., does not match) the second speed limit, the fusion component 116 may use one or more additional rules to determine the first speed limit.

[0039] For a first example, the fusion component 116 may use a second rule based at least on the first speed limit including a probability (e.g., confidence) that meets (e.g., is equal to or greater than) a threshold probability (represented by threshold data 120), where the second rule indicates that the final speed limit includes the first speed limit, or determine to use a second speed limit when the probability does not meet (e.g., is less than) the threshold probability. As described herein, the threshold probability may include, but is not limited to, 75%, 90%, 95%, 99%, and / or any other probability. For a second example, the fusion component 116 may use a third rule based at least on the first speed limit including an age that meets a threshold age (which may correspond to an offset described herein) (also represented by threshold data 120), where the third rule indicates that the final speed limit includes the first speed limit, or determine to use a second speed limit when the age does not meet the threshold age. As described herein, the age may be associated with the distance traveled since the sensing component 102 determined the first speed limit and / or the time period elapsed since the sensing component 102 determined the first speed limit. For example, the age may meet the threshold age when the distance traveled is less than or equal to a threshold distance (e.g., 5 meters, 10 meters, 50 meters, 100 meters, etc.) (also represented by threshold data 120), or the age may be determined not to meet the threshold age when the distance traveled is greater than the threshold distance. Additionally, the age may meet the threshold age when the time period is less than or equal to a threshold time period (e.g., 5 seconds, 10 seconds, 50 seconds, 100 seconds, etc.) (also represented by threshold data 120), or the age may be determined not to meet the threshold age when the time period is greater than the threshold time period.

[0040] For a third example, the fusion component 116 may determine whether the path of the vehicle has changed since the sensing component 102 determined the first speed limit. In some examples, the fusion component 116 may determine that the path of the vehicle has changed based at least on the vehicle turning, selecting a lane when there is a fork in the road, and / or any other path that may change the speed limit. If the fusion component 116 determines that the vehicle has not changed path (e.g., continues to travel in the same lane) and the probability of the first speed limit meets the threshold probability and / or the age of the first speed limit meets a first age, the fusion component 116 may use a fourth rule that indicates that the final speed limit includes the first speed limit. However, if the fusion component 116 determines that the vehicle has changed path (e.g., the vehicle has turned), and even if the probability of the first speed limit meets the threshold probability and the age of the first speed limit meets the first age, the fusion component 116 may also use the fourth rule, further indicating that the final speed limit includes the second speed limit.

[0041] For a fourth example, the fusion component 116 may determine that the sensed speed data 108 indicates that the sensing component 102 has detected a traffic sign associated with a temporary speed limit, such as a construction sign, an emergency sign, and / or any other traffic sign that may change the speed limit. The fusion component 116 may then use a fifth rule that indicates that the final speed limit includes a first speed limit associated with the traffic sign indicating the temporary speed limit. In some examples, the fusion component 116 may determine to use the first speed limit because the map data 112 may not have been updated to indicate the temporary change in the speed limit.

[0042] Although the first two examples describe that when there is a conflict between the first speed limit and the second speed limit determined using the processing component 110 and the probability meets the threshold probability and / or the age meets the threshold age, the fusion component 116 uses the first speed limit determined using the sensing component 102, in other examples, the fusion component 116 may use the second speed limit. For example, in such an example, the fusion component 116 may use the second speed limit based on map speed data 112 representing a particular type of map (such as a high-definition map) where speed limit signs are continuously updated and / or known to be accurate. Although these are just a few example rules that the fusion component 116 may use to fuse speed limits, in other examples, the fusion component 116 may use additional and / or alternative rules.

[0043] The process 100 may then include the fusion component 116 generating and / or outputting speed limit data 122 representing the final speed limit associated with the speed limit sign, such as outputting to one or more computing systems 124 associated with the vehicle and / or other systems. For a first example, when the computing system 124 is associated with a vehicle, for example, the computing system 124 may use the speed limit data 122 when determining one or more operations for navigating the vehicle in the environment. For example, the computing system 124 may cause the vehicle to navigate at the final speed limit represented by the speed limit data 122 (e.g., when it is safe to do so). For a second example, when the computing system 124 is associated with one or more systems that generate the map speed data 112, for example, the computing system 124 may use the speed limit data 122 to update the map speed data 112. For example, if the speed limit data 122 indicates that the speed limit of the speed limit sign is incorrect (e.g., the speed limit has changed since the map was last updated), the computing device 124 may update the map data 112 to indicate the final speed limit.

[0044] Figure 2A is a diagram of a map 200 of an environment 200(1) according to some embodiments of the present disclosure (at least in Figure 2B(shown in). Environment 200(1) includes a first road 206(1) having a first driving lane 207(1) (“first lane 207(1)”), the first road 206(1) intersects a second road 206(2), and the second road includes a second driving lane 207(2) (“second lane 207(2)”). The first lane 207(1) is associated with a first speed limit sign 208(1) and a yield sign 210, and the second lane 207(2) is associated with a second speed limit sign 208(2). According to the map 200, the first speed limit sign 208(1) may indicate a speed limit of 35 MPH, and it is assumed that the orientation 212(1) perpendicular to the front face of the first speed limit sign 208(1) extends in a direction substantially parallel to the driving direction along the first lane 207(1). Although Figure 2A the front face of the yield sign 210 is visible to the viewer as shown, it should be understood that the orientation of the yield sign 210 relative to the first lane 207(1) is the same as the orientation of the first speed limit sign 208(1). Similarly, according to the map 200, the second speed limit sign 208(2) may indicate a speed limit of 45 MPH, and the orientation 212(2), i.e., the normal to the front face of the second speed limit sign 208(2), extends in a direction substantially parallel to the driving direction along the second lane 207(2).

[0045] In Figure 2A the example of, the vehicle 202 (e.g., at least as shown in Figure 2B ) can use the map 200 to determine the speed limits associated with the speed limit signs 208(1)-(2). For example, the vehicle 202 can use sensor data (e.g., sensor data 104) to determine the position of the vehicle relative to the map 200. Then, the vehicle 202 can use this position to determine the speed limit. For the first example, if the vehicle 202 determines that the position 224(1) of the vehicle 202 is on the first road 206(1) and / or in the first lane 207(1), the vehicle 202 can use the map 200 to determine the speed limit associated with the speed limit sign 208(1) (e.g., a speed limit of 35 MPH). For the second example, if the vehicle 202 determines that the position 224(2) is on the second road 206(2) and / or in the second lane 207(2), the vehicle 202 can use the map 200 to determine the speed limit associated with the second speed limit sign 208(2) (e.g., a speed limit of 45 MPH).

[0046] Figure 2B is a diagram of the first environment 200(1), where the vehicle 202 is traveling in the first lane 207(1) of the first road 206(1) in the first direction 204(1). The vehicle 202 is equipped with a fusion component 116 according to some embodiments of the present disclosure, and if it is autonomous or semi-autonomous, the vehicle 202 may have the features described herein with respect to Figures 8A to 8DSome or all of the features discussed for vehicle 800. In Figure 2B , vehicle 202 is shown using a perception system (e.g., Figure 8C perception component 102 and / or ADAS 838) of Figure 2A to read a yield sign 210 and is receiving additional speed limit data as map information 200 from a Figure 8D map. The map information 200 and its transmission (arrow 203) to vehicle 800 are discussed in more detail with reference to Figure 2B . From this discussion, it can be understood that the term "map information" encompasses HD map data, SD map data, and / or navigation map data. Thus, although the examples discussed herein primarily focus on map information in the form of navigation map data, other embodiments of the present disclosure may obtain map data in the form of HD maps rather than navigation map data. In

[0047] In Figure 2B the example of Figure 2B , the fusion component 116 has associated a first lane 207(1) with a first offset 220(1), which is the cumulative distance along the ego lane (corresponding to the first lane 207(1)) from the ego vehicle representing vehicle 202. The length of the first offset 220(1) is the distance over which the speed limit indicated by the first speed limit sign 208(1) continues to apply in the first lane 207(1). Figure 2BIn the example, after the perception system of vehicle 202 reads the yield sign 210, it can provide data to other systems in vehicle 202, causing vehicle 202 to start decelerating to a speed below 35 MPH, even though this speed limit has been applicable to the first lane 207(1) until the yield sign 210. Map information 200 can also provide such data to vehicle 202 to cause the same or a similar speed reduction.

[0048] Figure 2C is similar to Figure 2B Figure of the first environment 200(1) which is similar, except that it shows vehicle 202 traveling in the second lane 207(2) of the second road 206(2) in the second direction 204(2). As shown, vehicle 202 is using its perception system to read the speed limit from the second speed limit sign 208(2) and continues to receive map information 200 as additional speed limit data. Regarding Figure 8D Map information 200 and its transmission to vehicle 202 (arrow 203) are discussed in more detail. The fusion component 116 has assigned two offsets to the portion of the second lane 207(2) shown in Figure 2C i.e., the second offset 220(2) and the third offset 220(3). From the perspective of the second direction 212(2), the second offset 220(2) extends a distance behind (and behind vehicle 202) the second speed limit sign 208(2), because the portion of the second lane 207(2) in front of the second speed limit sign 208(2) is subject to a speed limit other than 45 MPH. Then, once the perception system of vehicle 202 reads the 45 MPH speed limit due to the second speed limit sign 208(2), and / or map information 200 notifies of the 45 MPH speed limit due to the second speed limit sign 208(2), another offset applies, i.e., the third offset 220(3), which extends along the direction 204(2) to a point outside the environment 200(1).

[0049] Figure 2DIt is a view of the second environment 200(2), in which a vehicle 202 is shown traveling in the third lane 207(3) of a third road 206(3) in a third direction 204(3). The third lane 207(3) is shown associated with a third speed limit sign 208(3) that indicates a speed limit of 55 MPH, and is associated with a zone sign 214 which, for illustrative purposes, is understood to require a speed lower than 55 MPH. The zone sign 214 should be understood as any sign that imposes a speed limit lower than the speed limit governing a portion of the lane immediately preceding the zone sign 214. For example, the zone sign 214 may indicate the start of a school zone or a road construction zone. The zone is understood to extend a predetermined distance along the third lane 207(3) to an end point (not shown), at which point the higher speed limit applied to the third lane immediately preceding the zone sign 214 is restored. In Figure 2D the vehicle 202 has passed the third speed limit sign 208(3). The vehicle 202 is shown using its sensing system to read the zone sign 214, and it continues to receive additional speed limit data as map information 222 from its navigation map system. The map information 222 and its transmission to the vehicle 202 (arrow 203) are discussed in more detail with respect to Figure 8D The fusion component 116 has assigned three offsets to the portion of the third lane 207(3) shown in Figure 2D namely, a fourth offset 220(4), a fifth offset 220(5), and a sixth offset 220(6). From the perspective of the third direction 212(3), the fourth offset 220(4) extends a distance behind the third speed limit sign 208(3) (and behind the vehicle 202) because the portion of the third lane 207(3) before the third speed limit sign 208(3) is governed by a speed limit other than 55 MPH. Then, once the sensing system of the vehicle 202 reads the 55 MPH speed limit due to the third speed limit sign 208(3), and / or the map information 222 notifies of the 55 MPH speed limit due to the third speed limit sign 208(3), another offset applies, namely the fifth offset 220(5), which extends in the direction 204(3) in front of the third speed limit sign 208(3) along that direction to the location of the zone sign 214. Then, the fusion component 116 assigns a sixth offset, the distance of which is equal to the distance of the zone demarcated by the zone sign 214.

[0050] Although Figures 2A - 2D the examples of

[0051] only show a few examples of a vehicle fusing data to determine speed limits, in other examples, the vehicle may use additional and / or alternative rules to fuse data to determine speed limits, which will be described in more detail herein. Figures 3A to 3Bis an example of fusing speed limit data from a perception system with speed limit data associated with a map using one or more rules according to some embodiments of the present disclosure. For example, as Figure 3A shown in the example of, vehicle 302 may be traveling along the first lane 207(1) and approaching the first speed limit sign 208(1). During navigation, vehicle 302 may use one or more sensors (e.g., sensor 106) to generate sensor data (e.g., sensor data 104) representing the first speed limit sign 208(1). Then, vehicle 302 may process the sensor data using a perception system (e.g., perception component 102) and determine at least based on the processing a first speed limit associated with the first speed limit sign 208(1) and / or a probability associated with the first speed limit. In some examples, vehicle 302 may then associate the first speed limit with the first lane 207(1) on which vehicle 202 is traveling.

[0052] Vehicle 302 may also use map 200 of environment 200(1) to determine a second speed limit associated with the first speed limit sign 208(1). For example, vehicle 302 may use an additional portion of the sensor data (e.g., position data) to determine an attitude associated with vehicle 302 within the environment (which will be described in more detail herein). Then, vehicle 302 may determine at least based on the attitude that vehicle 302 is traveling along the first lane 207(1) within environment 200(1). Additionally, then, vehicle 302 may use map 200 to determine that the first speed limit sign 208(1) is associated with the first lane 207(1). At least based on this determination, vehicle 302 may also use map 200 to determine that the second speed limit is associated with the first speed limit sign 208(1) and thus with the first lane 207(1).

[0053] Then, vehicle 302 (e.g., fusion component 116) may determine whether the first speed limit corresponds to (e.g., matches) the second speed limit. If vehicle 302 determines that the first speed limit does indeed correspond to the second speed limit, vehicle 302 may determine that the final speed limit includes the first speed limit and / or the second speed limit. However, if vehicle 302 determines that the first speed limit does not correspond to (e.g., does not match) the second speed limit, vehicle 302 may determine whether the final speed limit includes the first speed limit or the second speed limit. For the first example, vehicle 302 may determine that the final speed limit includes the first speed limit at least based on the probability meeting a threshold probability, or vehicle 302 may determine that the final speed limit includes the speed limit at least based on the probability not meeting the threshold probability.

[0054] For the second example, vehicle 302 may additionally or alternatively determine whether the age of the first speed limit meets a threshold age. In some examples, vehicle 302 may useFigure 2B The offset 220(1) in the example is determined. For example, vehicle 202 can determine that the age meets the threshold age at least based on the fact that vehicle 302 is still traveling within the area associated with the offset 220(1). At least based on this determination, vehicle 302 can determine that the final speed limit includes the first speed limit.

[0055] Next, regarding Figure 3B In the example of, vehicle 302 can steer into the new lane 304. Thus, vehicle 302 can use the map 200 of the environment to determine the second speed limit associated with the new lane 304 (e.g., a speed limit sign associated with the new lane 304, which is not shown since vehicle 302 has not yet approached the speed limit sign). For example, vehicle 302 can use sensor data (e.g., position data) to determine the attitude associated with vehicle 302 within the environment (which will be described in more detail herein). Then, vehicle 302 can determine that vehicle 302 is traveling along the new lane 304 within the environment at least based on this attitude. In addition, vehicle 302 can subsequently use the map 200 to determine that the speed limit sign is associated with the new lane 304. At least based on this determination, vehicle 302 can also use the map 200 to determine that the second speed limit is associated with the speed limit sign and thus with the new lane 304.

[0056] However, since vehicle 302 has not yet approached the speed limit sign in the Figure 3B example, vehicle 302 may still use the first speed limit associated with the first speed limit sign 208(1) as the perceived speed limit. Then, vehicle 302 (e.g., the fusion component 116) can determine whether the first speed limit corresponds to (e.g., matches) the second speed limit. If vehicle 302 determines that the first speed limit does indeed correspond to the second speed limit, then vehicle 302 can determine that the final speed limit includes the first speed limit and / or the second speed limit. However, if vehicle 302 determines that the first speed limit does not correspond to (e.g., does not match) the second speed limit, then vehicle 302 can determine whether the final speed limit includes the first speed limit or the second speed limit.

[0057] For example, vehicle 302 can determine that vehicle 302 has changed its path from traveling along the first lane 207(1) to traveling along the new lane 304 (e.g., vehicle 302 has turned). In addition, vehicle 302 can determine that no new perceived speed limit has been determined since the path was changed. At least based on these determinations, vehicle 302 can determine that the final speed limit includes the second speed limit.

[0058] Figure 4An example of a fusion algorithm according to some embodiments of the present disclosure is shown. Each block of the method 400 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. These methods can also be embodied as computer-usable instructions stored on a computer storage medium. These methods can be provided by a stand-alone application, service, or hosted service (stand-alone or in combination with another hosted service), or a plug-in of another product, to name a few. Additionally, the method 400 is described by way of example with respect to Figure 1 the fusion component 116. However, these methods can be alternatively or additionally performed by any one system or any combination of systems, including but not limited to the systems described herein.

[0059] The method 400 begins at start 402, which can be any event that triggers the generation of the data described in blocks 406 and 408. Block 404 describes the functionality of a "navigation map tracker", which can include the processing component 110 ( Figure 1 ) or be part of the processing component 110. The navigation map tracker can (e.g., continuously) monitor the stream of navigation map data / information 200 ( Figures 2B to 2D ) received by the vehicle 202 ( Figures 2B to 2D ). At a predetermined frequency (e.g., 15 Hertz), the navigation map tracker can take a snapshot of the navigation map data, with each snapshot defining a navigation map data frame. The navigation map tracker caches historical navigation map data obtained at a predetermined distance (e.g., 2000 meters (m)) behind the vehicle 202 while the vehicle 202 is traveling in a lane. Then, the navigation map tracker can "publish" each navigation map data frame together with the cached navigation map historical data corresponding to the data frame to the fusion component 116 ( Figure 1 ). In some embodiments, the navigation map tracker can also preprocess the received navigation map data 200 before sending it to the fusion component 116. Such preprocessing can include converting the received navigation map data into a format more conducive to further processing, duplicate data removal of speed limit values, and removal of outdated data. Attaching the cached navigation map speed limit data to the navigation map data frame allows the fusion component 116 to have a complete picture, thereby enabling it to make informed decisions when performing the methods described herein to select which speed limit value to publish as the speed limit data 122 to the computing system 124 ( Figure 1)。The box 404 represents the only stateful part of the method 400, which means that the operations represented by the box 404 are the only operations in the method 400 that rely on previously stored data. The stateful nature of the box 404 allows the operations represented by the remaining boxes in the method 400 to be stateless, which means that they are independent of previously stored data. The stateless nature of these operations of the method 400 makes it easier to debug the fusion component 116. Once the operations of the box 404 are executed, the method 400 progresses from the box 404 to the box 408. However, since the method 400 also progresses from the box 406 to the box 408, the box 406 will be described before the box 408.

[0060] At the box 406, the method 400 generates speed limit segments for the driving lanes. At the box 406, the method 400 first collects the sensed speed limits in the driving direction, and each sensed speed limit has been associated with an offset. The box 406 represents a preprocessing operation for the sensed speed limits. The sensed speed limits typically adopt the World Model (WM) format. The WM encodes one or more (e.g., each) speed limits as its own independent entry. In the box 406, the method 400 converts the collected sensed speed limits from the WM format to a format that is more conducive to further processing. The sensed speed limits preprocessed in this way are called "speed limit segments". In addition, any conditional speed limits that may apply to the same offset associated with the collected sensed speed limits are also encoded into the same speed limit segment. Therefore, a speed limit segment can be generated each time the value of the collected sensed speed limit changes. Starting from the box 406, the method 400 advances to the box 408.

[0061] At the box 408, the method 400 generates a "speed limit profile". The speed limit profile is generated by further associating the offset (which has been associated with the sensed speed limits collected at the box 406) with the navigation map data frame issued at the box 404. Through their common association with the offset, the navigation map data frame from the box 404 and the speed limit segments from the box 406 can be associated with each other. The two types of speed limit data thus associated with each other in the speed limit profile can be called the associated speed limit segments and the associated navigation map MPP speed limits. Now reference will be made to Figure 5 The processing operations represented by the box 408 will be discussed in detail.

[0062] Figure 5 is a flowchart showing a series of processing operations 408 represented by the Figure 4 box 408 in accordance with some embodiments of the present disclosure. The series 408 begins at the box 502, where the series 408 determines whether there are navigation map speed limits or sensed speed limits available for processing. If not, the series 408 directly advances to Figure 4Step 410, which will be described later. However, if such speed limit data exists, the speed limit profile is generated in the above manner, and series 408 proceeds to block 504, where the generated speed limit profile is added (stored) to the output buffer, which is a memory block in the memory unit that is operatively communicable with the fusion component 116 ( Figure 1 ). The list of speed limit profiles is stored in the output buffer because the vehicle 202 ( Figures 2B to 2D ) is subject to various different speed limits at different distances in the driving direction. After storing the generated speed limit profile in this way, series 408 proceeds to block 506, where it is determined whether the output buffer is full. If not, series 408 returns to block 502; if so, series 408 proceeds to block 508, where a "buffer full" warning is recorded. From block 508, series 408 proceeds to Figure 4 operation 410, as described below.

[0063] Referring again to Figure 4 , after performing the operations described above with respect to Figure 5 , method 400 proceeds from block 408 to block 410. For each speed limit profile stored in the output buffer, method 400 selects, at block 410, the speed limit to be published, on the one hand, the sensed speed limit in the relevant speed limit section and, on the other hand, the relevant navigation map MPP speed limit in the profile. This determination is made by executing one of a plurality of rules selected from the SLF rule book 411, which will be described in further detail below with reference to Figure 6 . After executing the selected rule, method 400 proceeds from block 410 to end 412, which may represent the operation of publishing the resulting speed limit data 122 to the computing system 124 ( Figure 1 ).

[0064] Figure 6 is a block diagram showing an example SLF rule book 411 stored in electronic form in the memory unit 600 according to some embodiments of the present disclosure. The SLF rule book 411 includes a plurality of stored instructions (rules), each rule instructing the fusion component 116 ( Figure 1 ) to publish, for each stored speed limit profile, either the sensed speed limit in the relevant speed limit section of the profile or the relevant navigation map MPP speed limit in the same profile. Each rule addresses the conditions or set of conditions that exist when the fusion component 116 is to execute the rule. In some embodiments, the rules of the SLF rule book 411 may be organized into groups or sets. For example, as Figure 6As shown, the rules can be organized into four sets: Set 1, including subset 1A (602) of construction area rules and subset 1B (604) of exit ramp rules; Set 2 (606), including rules for school zones, icy conditions, foggy conditions, and nighttime conditions; Set 3 (608), including unreasonable speed limit rules and navigation map matching rules; and Set 4 (610), including basic rules. For the sake of illustration, Figure 6 subset 1A (602) including three rules 612, 614, 616 and subset 1B (604) including three rules 618, 620, 622 are shown. It should be understood that the remaining sets 606, 608, 610 also each contain multiple additional corresponding rules, but for the sake of illustration, Figure 6 these rules are omitted herein. In some embodiments, sets 602, 604, 606, 608, 610 are arranged in order of priority, i.e., in the order of specificity of conditions, such that the last set 610 contains basic rules that apply only when the existing conditions do not match any of the rules in the previous sets 602, 604, 606, 608. The rules within a set are also listed in order of priority. The first-listed rule that matches a condition related to the speed limit profile in any set is executed, and the fusion component 116 ( Figure 1 ) accordingly issues speed limit data 122. Rule examples that make up the SLF rule book 411 will now be described, first with reference to subset 602 (for construction area rules), and then with reference to subset 604 (for school area rules).

[0065] In subset 602, if the sensor 106 ( Figure 1 ) detects a temporary speed limit, the construction area rule 612 instructs the fusion component 116 ( Figure 1 ) to issue the sensed speed limit in the speed limit profile; if the vehicle 202 is leaving the construction area and the speed limit profile does not include the sensed speed limit ( Figure 6 "SL" in), the construction area rule 614 instructs the fusion component 116 to issue the navigation map speed limit in the speed limit profile; and if the vehicle 202 is leaving the construction area and the speed limit profile does not include the new sensed speed limit or the new navigation map speed limit, the construction area rule 616 instructs the fusion component 116 to issue the cached navigation map speed limit in the speed limit profile.

[0066] In subset 604, the exit ramp rule 618 handles the case where the sensed speed limit in the speed limit profile does not match the navigation map speed limit in the same speed limit profile. This situation can occur, for example, when one or more sensed speed limit sensors incorrectly interpret an exit-only speed limit as a nominal speed limit. Other situations can cause such a mismatch between the sensed speed limit and the navigation map speed limit. The exit ramp rule 618 is designed to filter out the erroneously detected sensed speed limit in the event of such a speed limit mismatch by instructing the sensing component 102 and the processing component 110 ( Figure 1 ) to search for a new speed limit from nearby alternative paths having a matching speed limit or only a navigation map speed limit. If such an alternative path is found, the sensed speed limit of the speed limit profile is discarded, and the exit ramp rule 618 instructs the fusion component 116 to publish the navigation map speed limit associated with the located alternative path. The second exit ramp rule in subset 604, namely the exit ramp rule 620, handles the situation where the sensing component 102 detects an exit-only speed limit marked as "relatively uncertain" and the location of the navigation map speed limit is an exit ramp. In this case, the exit ramp rule 620 instructs the fusion component 116 to publish the sensed speed limit in the speed limit profile. In the Figure 6 illustrated embodiment, the exit ramp rule 622 appears as the final rule of subset 604. The exit ramp rule 622 illustrates the result of the fusion component 116 publishing two separate speed limits instead of one speed limit. The exit ramp rule 622 handles the following situations: (i) the sensing component 102 detects a nominal sensed speed limit and an "uncertain" exit-only sensed speed limit; and (ii) the map location associated with the navigation map speed limit is not on the exit ramp on which the vehicle 202 is traveling. Given these conditions, the exit ramp rule 622 instructs the fusion component 116 to publish the two sensed speed limits in the speed limit profile. The nominal sensed speed limit is published as "relevant" under "index 0", while the exit-only speed limit is published as "uncertain relevant" under "index 1". When the fusion component 116 publishes two speed limits, the vehicle driver can make a decision on which speed limit to apply based on the designations attached to each published sensed speed limit and / or the fusion component 116 can select between one of the speed limits using any of the techniques described herein. The exit ramp rule 622 is not the only rule in the SLF rule book 411 that requires the publication of two speed limits; such a result can occur in response to the execution of many other rules in the SLF rule book 411.

[0067] The other rules in the SLF rulebook 411 need not rely solely on the data present in the speed limit profile. Such rules can also obtain conditional information from other sources. For example, the icing condition rules in set 606 can also analyze data transmitted by an Operational Design Domain (ODD) monitor, which can be located Figure 1 in sensor 106 of. The ODD monitor can evaluate outdoor conditions, such as whether the lane or local weather indicates icing conditions. Thus, for example, one of the rules in set 606 can specify that if the ODD monitor detects environmental icing and the speed limit profile contains only the navigation map icing speed limit, the fusion component 116 will publish the navigation map icing speed limit.

[0068] As an example of a basic rule residing in set 610, if the sensed speed limit is valid, the rule can simply direct the fusion component 116 to publish the sensed speed limit in the speed limit profile. A sensed speed limit change can be considered valid if the offset associated with the sensed speed limit change is within a distance threshold corresponding to the navigation map speed limit change. In this case, the confidence in the sensed speed limit is high.

[0069] In some embodiments, the rules of the SLF rulebook 411 can be expressed in a structured protobuf schema, and a given rule set can be expressed as a TEXTPROTO file that contains rule definitions for each rule residing in that set. These formats facilitate rule definition and modification and promote set modularity. However, the foregoing format description is exemplary and not intended to be limiting. Similarly, Figure 6 the number of rules residing in subsets 602, 604 shown in need not be limited to three respectively. Additional rules can reside in each set as needed.

[0070] Now referring to Figure 7 , each block of the method 700 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The method 700 can also be embodied as computer-usable instructions stored on a computer storage medium. The method 700 can be provided by a stand-alone application, service, or hosted service (stand-alone or in combination with another hosted service) or a plug-in of another product, to name just a few. Additionally, the method 700 is described by way of example with reference to Figure 1 . However, the method 700 can alternatively or additionally be performed by any one system or any combination of systems, including but not limited to the systems described herein.

[0071] Method 700 may include, at block B702, determining a first speed limit associated with a speed limit sign based at least on sensor data obtained using one or more sensors of a machine. For example, the perception component 102 (e.g., a perception system) may receive sensor data 104 generated using the sensors 106. As described herein, the sensor data 106 may include, but is not limited to, image data, video data, LiDAR data, RADAR data, and / or any other type of sensor data. Additionally, the sensor data 104 may represent a sensor representation (e.g., an image) depicting the speed limit sign. The perception component 102 may then process the sensor data 104 and determine the first speed limit associated with the speed limit sign based at least on the processing.

[0072] Method 700 may include, at block B704, determining a second speed limit associated with a speed limit sign based at least on map data. For example, the processing component 110 may receive sensor data 104 generated using the sensors 106. As described herein, the sensor data 104 may include, but is not limited to, position data, image data, video data, LiDAR data, RADAR data, and / or any other type of data. The processing component 110 may then process the sensor data 104 and determine the position of the machine in the environment based at least on the processing. Additionally, the processing component 110 may use the position to locate the machine on a map represented by the map data 112. The processing component 110 may then use the map to determine the second speed limit associated with the speed limit sign.

[0073] Method 700 may include, at block B706, using one or more rules and determining a final speed limit associated with a speed limit sign based at least on the first speed limit and the second speed limit. For example, the fusion component 116 may use the first speed limit, the second speed limit, and the rules to determine the final speed limit associated with the speed limit sign. As described herein, the rules may indicate that if the first speed limit matches the second speed limit, the final speed limit includes the first speed limit and / or the second speed limit. Additionally, the rules may indicate that if the first speed limit does not match the second speed limit, the final speed limit includes one of the first speed limit or the second speed limit. The techniques used to determine whether the final speed limit includes the first speed limit or the second speed limit will be described in more detail in conjunction with Figures 1 to 6 and will be described in more detail.

[0074] Method 700 may include, at block B708, causing the machine to perform one or more operations based at least on the final speed limit sign. For example, the computing system 102 may use the final speed limit (which may be represented by the speed limit data 122) to cause the machine to perform operations such as navigating according to the final speed limit.

[0075] Example Autonomous Vehicle

[0076] Figure 8A FIG. 2 is an illustration of an example autonomous vehicle 800 in accordance with some embodiments of the present disclosure. The autonomous vehicle 800 (alternatively, referred to herein as "vehicle 800") can include, but is not limited to, passenger vehicles such as cars, trucks, buses, first responder vehicles, shuttle vehicles, electric or motorized bicycles, motorcycles, fire trucks, police vehicles, ambulances, boats, construction vehicles, underwater vessels, robotic vehicles, drones, airplanes, vehicles coupled to trailers (e.g., semi-trailer trucks for hauling goods), and / or another type of vehicle (e.g., driverless and / or accommodating one or more passengers). Autonomous vehicles are generally described according to the levels of automation defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) in "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (Standard No. J3016 - 201806, released on June 15, 2018, Standard No. J3016 - 201609, released on September 30, 2016, and prior and future versions of the standard). The vehicle 800 may be capable of implementing functions corresponding to one or more of Levels 3 - 5 of the autonomous driving levels. The vehicle 800 may be capable of implementing functions corresponding to one or more of Levels 1 - 5 of the autonomous driving levels. For example, depending on the embodiment, the vehicle 800 may be capable of implementing driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). As used herein, the term "autonomous" may include any and / or all types of autonomy of the vehicle 800 or other machines, such as fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, providing assisted autonomy, semi-autonomous, primarily autonomous, or other designations.

[0077] The vehicle 800 may include components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of the vehicle. The vehicle 800 may include a propulsion system 850, such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. The propulsion system 850 may be connected to a driveline of the vehicle 800 that may include a transmission to effect propulsion of the vehicle 800. The propulsion system 850 may be controlled in response to receiving a signal from the throttle / accelerator 852.

[0078] A steering system 854 that may include a steering wheel can be used to steer vehicle 800 (e.g., along a desired path or route) while the propulsion system 850 is operating (e.g., while the vehicle is in motion). The steering system 854 can receive signals from a steering actuator 856. For fully autonomous (Level 5) functionality, the steering wheel can be optional.

[0079] A brake sensor system 846 can be used to operate vehicle brakes in response to receiving signals from a brake actuator 848 and / or a brake sensor.

[0080] One or more controllers 836 that may include one or more system-on-chips (SoCs) 804 ( Figure 8C ) and / or one or more GPUs can provide signals (e.g., representing commands) to one or more components and / or systems of vehicle 800. For example, one or more controllers can send signals to operate vehicle brakes via one or more brake actuators 848, operate the steering system 854 via one or more steering actuators 856, and operate the propulsion system 850 via one or more throttles / accelerators 852. One or more controllers 836 can include one or more on-board (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operation commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving vehicle 800. One or more controllers 836 can include a first controller 836 for autonomous driving functionality, a second controller 836 for functional safety functionality, a third controller 836 for artificial intelligence functionality (e.g., computer vision), a fourth controller 836 for infotainment functionality, a fifth controller 836 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 836 can handle two or more of the above functions, two or more controllers 836 can handle a single function, and / or any combination thereof.

[0081] One or more controllers 836 may provide signals for controlling one or more components and / or systems of vehicle 800 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example and without limitation, a Global Navigation Satellite System (“GNSS”) sensor 858 (e.g., a Global Positioning System sensor), a RADAR sensor 860, an ultrasonic sensor 862, a LIDAR sensor 864, an Inertial Measurement Unit (IMU) sensor 866 (e.g., an accelerometer, a gyroscope, a magnetic compass, a magnetometer, etc.), a microphone 896, a stereo camera 868, a wide-angle camera 870 (e.g., a fish-eye camera), an infrared camera 872, a surround camera 874 (e.g., a 360-degree camera), a long-range and / or mid-range camera 898, a speed sensor 844 (e.g., for measuring the speed of vehicle 800), a vibration sensor 842, a steering sensor 840, a brake sensor (e.g., as part of a brake sensor system 846), and / or other sensor types.

[0082] One or more of the controllers 836 may receive inputs (e.g., represented by input data) from the instrument cluster 832 of vehicle 800 and provide outputs (e.g., represented by output data, display data, etc.) via the Human Machine Interface (HMI) display 834, an audible annunciator, a speaker, and / or via other components of vehicle 800. These outputs may include information such as vehicle speed, rate, time, map data (e.g., Figure 8C a high-definition (“HD”) map 822), location data (e.g., the location of vehicle 800 on a map, for example), direction, the location of other vehicles (e.g., occupancy grid), and information about objects and object states as perceived by the controller 836, and so on. For example, the HMI display 834 may display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, exiting 34B in two miles, etc.).

[0083] Vehicle 800 also includes a network interface 824, which may communicate over one or more networks using one or more wireless antennas 826 and / or a modem. For example, network interface 824 may be capable of communicating via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multi-Carrier (“CDMA2000”), etc. One or more wireless antennas 826 may also be used to enable communication between objects (such as vehicles, mobile devices, etc.) in an environment using one or more local area networks such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc. and / or one or more low power wide area networks (“LPWAN”) such as LoRaWAN, SigFox, etc.

[0084] Figure 8B For an example autonomous vehicle 800 according to some embodiments of the present disclosure for Figure 8A Example camera positions and fields of view of an example autonomous vehicle 800. The cameras and respective fields of view are one example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included, and / or these cameras may be located at different positions on vehicle 800.

[0085] The camera type for the cameras may include, but is not limited to, digital cameras that may be suitable for use with components and / or systems of vehicle 800. The cameras may operate under Automotive Safety Integrity Level (ASIL) B and / or under another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The cameras may be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a Red Clear Clear Clear (RCCC) color filter array, a Red Clear Clear Blue (RCCB) color filter array, a Red Blue Green Clear (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras such as those with RCCC, RCCB, and / or RBGC color filter arrays may be used in an effort to increase light sensitivity.

[0086] In some examples, one or more of the cameras may be used to perform Advanced Driver Assistance System (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-functional monocular camera may be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. One or more of the cameras (e.g., all cameras) may record and provide image data (e.g., video) simultaneously.

[0087] One or more of the cameras can be mounted in a mounting assembly such as a custom-designed (3D printed) component to cut off stray light and reflections from within the vehicle (such as reflections from the dashboard reflected in the windshield mirror) that may interfere with the image data capture capabilities of the camera. Regarding the wing mirror mounting assembly, the wing mirror assembly can be custom 3D printed such that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0088] A camera having a field of view that includes an environmental portion in front of the vehicle 800 (such as a front camera) can be used for surround view to help identify forward paths and obstacles and, with the help of one or more controllers 836 and / or a control SoC, assist in providing information crucial for generating an occupancy grid and / or determining a preferred vehicle path. The front camera can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. The front camera can also be used for ADAS functions and systems, including lane departure warning ("LDW"), adaptive cruise control ("ACC"), and / or other functions such as traffic sign recognition.

[0089] A variety of cameras can be used in a front-mounted configuration, including, for example, a monocular camera platform that includes a complementary metal oxide semiconductor ("CMOS") color imager. Another example can be a wide-angle camera 870, which can be used to sense objects (such as pedestrians, intersection traffic, or bicycles) entering the field of view from the periphery. Although Figure 8B only one wide-angle camera is illustrated, any number (including zero) of wide-angle cameras 870 can be present on the vehicle 800. Additionally, any number of long-range cameras 898 (such as a long-view stereo camera pair) can be used for depth-based object detection, especially for objects for which a neural network has not been trained. The long-range cameras 898 can also be used for object detection and classification and basic object tracking.

[0090] Any number of stereo cameras 868 may also be included in a front-facing configuration. In at least one embodiment, one or more stereo cameras 868 may include an integrated control unit that includes a scalable processing unit that may provide a multi-core microprocessor and programmable logic ("FPGA") with an integrated controller area network ("CAN") or Ethernet interface on a single chip. Such a unit may be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. Alternative stereo cameras 868 may include a compact stereo vision sensor that may include two camera lenses (one on the left and one on the right) and an image processing chip that may measure the distance from the vehicle to a target object and activate autonomous emergency braking and lane departure warning functions using the generated information (e.g., metadata). Other types of stereo cameras 868 may be used in addition to or in place of those described herein.

[0091] Cameras having a field of view of an environmental portion including the side of vehicle 800 (e.g., side view cameras) may be used for surround view, providing information used to create and update an occupancy grid and generate side impact collision warnings. For example, surround cameras 874 (e.g., four surround cameras 874 as shown in Figure 8B may be disposed on vehicle 800. The surround cameras 874 may include wide-angle cameras 870, fish-eye cameras, 360-degree cameras, and / or the like. By way of example, four fish-eye cameras may be disposed on the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle may use three surround cameras 874 (e.g., left, right, and rear) and may utilize one or more other cameras (e.g., a forward camera) as a fourth surround camera.

[0092] Cameras having a field of view of an environmental portion including the rear of vehicle 800 (e.g., rear view cameras) may be used for assisting with parking, surround view, rear collision warnings, and creating and updating an occupancy grid. A variety of cameras may be used, including but not limited to cameras that are also suitable as front cameras as described herein (e.g., long-range and / or mid-range cameras 898, stereo cameras 868, infrared cameras 872, etc.).

[0093] Figure 8C For use in accordance with some embodiments of the present disclosure Figure 8ABlock diagram of an example system architecture of an example autonomous vehicle 800. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, function groupings, etc.) may be used in addition to or instead of those shown, and some elements may be omitted altogether. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components, and implemented in any suitable combination and location. The various functions described herein as being performed by entities may be implemented by hardware, firmware, and / or software. For example, the various functions may be implemented by a processor executing instructions stored in memory.

[0094] Figure 8C Each of the components, features, and systems in vehicle 800 is illustrated as being connected via bus 802. Bus 802 may include a Controller Area Network (CAN) data interface (alternatively referred to herein as the "CAN bus"). CAN may be a network within vehicle 800 that is used to assist in controlling various features and functions of vehicle 800, such as driving of brakes, acceleration, braking, steering, windshield wipers, and so on. The CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus may be read to find steering wheel angle, ground speed, engine revolutions per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0095] Although bus 802 is described herein as a CAN bus, this is not intended to be limiting. For example, in addition to or instead of the CAN bus, FlexRay and / or Ethernet may be used. Further, although bus 802 is shown as a single line, this is not intended to be limiting. For example, there may be any number of buses 802, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 802 may be used to perform different functions, and / or may be used for redundancy. For example, a first bus 802 may be used for collision avoidance functions, and a second bus 802 may be used for drive control. In any example, each bus 802 may communicate with any component of vehicle 800, and two or more buses 802 may communicate with the same component. In some examples, each SoC 804, each controller 836, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors of vehicle 800), and may be connected to a common bus such as the CAN bus.

[0096] Vehicle 800 may include one or more controllers 836, such as those described herein with respect to Figure 8A the controllers described. The controller 836 may be used for a variety of functions. The controller 836 may be coupled to any other different components and systems of the vehicle 800 and may be used for the control of the vehicle 800, the artificial intelligence of the vehicle 800, the infotainment for the vehicle 800, and / or the like.

[0097] Vehicle 800 may include one or more system-on-chips (SoCs) 804. The SoC 804 may include a CPU 806, a GPU 808, a processor 810, a cache 812, an accelerator 814, a data store 816, and / or other components and features not shown. In a variety of platforms and systems, the SoC 804 may be used to control the vehicle 800. For example, one or more SoCs 804 may be combined with an HD map 822 in a system (such as the system of the vehicle 800), and the HD map may obtain map refreshes and / or updates from one or more servers (such as Figure 8D one or more servers 878) via the network interface 824.

[0098] The CPU 806 may include a CPU cluster or a CPU complex (alternatively, referred to herein as "CCPLEX"). The CPU 806 may include multiple cores and / or an L2 cache. For example, in some embodiments, the CPU 806 may include eight cores in a coherent multi-processor configuration. In some embodiments, the CPU 806 may include four dual-core clusters, each of which has a dedicated L2 cache (such as a 2MB L2 cache). The CPU 806 (such as CCPLEX) may be configured to support simultaneous cluster operation such that any combination of the clusters of the CPU 806 can be active at any given time.

[0099] The CPU 806 may implement power management capabilities including one or more of the following features: each hardware block may automatically perform clock gating when idle to save dynamic power; each core clock may be gated when the core is not actively executing instructions due to the execution of WFI / WFE instructions; each core may be independently power gated; when all cores are clock gated or power gated, each core cluster may be independently clock gated; and / or when all cores are power gated, each core cluster may be independently power gated. The CPU 806 may further implement enhanced algorithms for managing power states, where the allowed power states and the desired wake-up times are specified, and the hardware / microcode determines the best power state for the cores, clusters, and CCPLEX to enter. The processing cores may support a simplified power state entry sequence in software, and this work is offloaded to the microcode.

[0100] The GPU 808 may include an integrated GPU (alternatively referred to herein as an “iGPU”). The GPU 808 may be programmable and efficient for parallel workloads. In some examples, the GPU 808 may use enhanced tensor instruction sets. The GPU 808 may include one or more streaming microprocessors, where each streaming microprocessor may include an L1 cache (e.g., an L1 cache with at least 96 KB of storage capacity), and two or more of these streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512 KB of storage capacity). In some embodiments, the GPU 808 may include at least eight streaming microprocessors. The GPU 808 may use a compute application programming interface (API). Additionally, the GPU 808 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0101] In automotive and embedded use cases, the GPU 808 can be power optimized for optimal performance. For example, the GPU 808 may be fabricated on fin field-effect transistors (FinFETs). However, this is not intended to be limiting, and the GPU 808 may be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor may incorporate a number of mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks. In such an example, each processing block may be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA tensor cores for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64 KB register file. Additionally, the streaming microprocessor may include independent parallel integer and floating-point data paths to enable efficient execution of workloads leveraging a mix of computation and addressing computations. The streaming microprocessor may include independent thread scheduling capabilities to allow for finer-grained synchronization and collaboration between parallel threads. The streaming microprocessor may include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

[0102] The GPU 808 may include a high-bandwidth memory (HBM) and / or a 16 GB HBM2 memory subsystem that provides a peak memory bandwidth of approximately 900 GB / s in some examples. In some examples, in addition to or alternatively to HBM memory, synchronous graphics random access memory (SGRAM), such as fifth-generation graphics double data rate synchronous random access memory (GDDR5), may be used.

[0103] The GPU 808 may include unified memory technology that includes access counters to allow memory pages to be migrated more precisely to the processors that most frequently access them, thereby improving the efficiency of the memory range shared among the processors. In some examples, address translation service (ATS) support may be used to allow the GPU 808 to directly access the CPU 806 page tables. In such examples, when the GPU 808 memory management unit (MMU) experiences a miss, an address translation request may be transmitted to the CPU 806. In response, the CPU 806 may look up the virtual-physical mapping for the address in its page tables and transmit the translation back to the GPU 808. In this way, the unified memory technology may allow a single unified virtual address space for the memory of both the CPU 806 and the GPU 808, thereby simplifying GPU 808 programming and porting applications to the GPU 808.

[0104] In addition, the GPU 808 may include access counters that may track how frequently the GPU 808 accesses the memory of other processors. The access counters may help ensure that memory pages are moved to the physical memory of the processor that most frequently accesses those pages.

[0105] The SoC 804 may include any number of caches 812, including those described herein. For example, the cache 812 may include an L3 cache available to both the CPU 806 and the GPU 808 (e.g., that is connected to both the CPU 806 and the GPU 808). The cache 812 may include a write-back cache that may track the state of lines, for example, by using a cache coherence protocol (such as MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache may include 4MB or more, but smaller cache sizes may also be used.

[0106] The SoC 804 may include an arithmetic logic unit (ALU) that may be utilized in processing in performing any of a variety of tasks or operations regarding the vehicle 800, such as processing a DNN. In addition, the SoC 804 may include a floating point unit (FPU) (or other math co-processor or digital co-processor type) for performing mathematical operations within the system. For example, the SoC 804 may include one or more FPUs integrated as execution units within the CPU 806 and / or the GPU 808.

[0107] The SoC 804 may include one or more accelerators 814 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC 804 may include a hardware accelerator cluster, which may include optimized hardware accelerators and / or large on-chip memories. The large on-chip memory (e.g., 4MB SRAM) may enable the hardware accelerator cluster to accelerate neural networks and other computations. The hardware accelerator cluster may be used to supplement the GPU 808 and offload some of the tasks of the GPU 808 (e.g., freeing up more cycles of the GPU 808 for performing other tasks). As an example, the accelerator 814 may be used for targeted workloads that are stable enough to be easily controlled for acceleration (e.g., perception, convolutional neural network (CNN), etc.). When used herein, the term "CNN" may include all types of CNNs, including region-based or region convolutional neural network (RCNN) and fast RCNN (e.g., for object detection).

[0108] The accelerator 814 (e.g., the hardware accelerator cluster) may include a deep learning accelerator (DLA). The DLA may include one or more tensor processing units (TPUs) that may be configured to provide an additional one trillion operations per second for deep learning applications and inference. The TPU may be an accelerator configured to perform image processing functions (e.g., for CNN, RCNN, etc.) and optimized for performing image processing functions. The DLA may be further optimized for a specific set of neural network types and floating-point operations and inference. The design of the DLA may provide higher performance per millimeter than a general-purpose GPU and far exceed the performance of the CPU. The TPU may perform several functions, including single-instance convolution functions, supporting INT8, INT16, and FP16 data types for both features and weights, and post-processor functions.

[0109] The DLA may quickly and efficiently execute neural networks, especially CNNs, for any of a variety of functions on processed or unprocessed data, such as, but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and identification and detection using data from microphones; CNNs for face recognition and vehicle owner recognition using data from camera sensors; and / or CNNs for security and / or safety-related events.

[0110] The DLA may perform any function of the GPU 808, and by using inference accelerators, for example, the designer may target the DLA or the GPU 808 for any function. For example, the designer may focus the processing and floating-point operations of the CNN on the DLA and leave other functions to the GPU 808 and / or other accelerators 814.

[0111] The accelerator 814 (e.g., a hardware accelerator cluster) may include a Programmable Vision Accelerator (PVA), which may alternatively be referred to herein as a computer vision accelerator. The PVA may be designed and configured to accelerate computer vision algorithms for Advanced Driver Assistance Systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA may provide a balance between performance and flexibility. For example, each PVA may include, for example and without limitation, any number of Reduced Instruction Set Computer (RISC) cores, Direct Memory Access (DMA), and / or any number of vector processors.

[0112] The RISC cores may interact with an image sensor (e.g., the image sensor of any of the cameras described herein), an image signal processor, and / or the like. Each of these RISC cores may include any number of memories. Depending on the embodiment, the RISC cores may use any of several protocols. In some examples, the RISC cores may execute a Real-Time Operating System (RTOS). The RISC cores may be implemented using one or more integrated circuit devices, Application-Specific Integrated Circuits (ASICs), and / or storage devices. For example, the RISC cores may include an instruction cache and / or tightly coupled RAM.

[0113] The DMA may enable the components of the PVA to access system memory independently of the CPU 806. The DMA may support any number of features used to optimize the PVA, including but not limited to supporting multi-dimensional addressing and / or circular addressing. In some examples, the DMA may support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stride, vertical block stride, and / or depth stride.

[0114] The vector processor may be a programmable processor that may be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA and may include a Vector Processing Unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as, for example, a Single Instruction Multiple Data (SIMD), Very Long Instruction Word (VLIW) digital signal processor. The combination of SIMD and VLIW may enhance throughput and rate.

[0115] Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. As a result, in some examples, each of the vector processors may be configured to execute independently of the other vector processors. In other examples, the vector processors included in a particular PVA may be configured to employ data parallelization. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms on a sequence of images or portions of an image. Among other things, any number of PVAs may be included in a hardware accelerator cluster, and any number of vector processors may be included in each of these PVAs. Additionally, the PVA may include additional error correction code (ECC) memory to enhance overall system security.

[0116] The accelerator 814 (e.g., a hardware accelerator cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for the accelerator 814. In some examples, the on-chip memory may include at least 4MB SRAM consisting of, for example and without limitation, eight field-configurable memory blocks, which may be accessed by both the PVA and the DLA. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and the DLA may access the memory via a backbone that provides high-speed memory access to the PVA and the DLA. The backbone may include, for example using the APB, an on-chip computer vision network that interconnects the PVA and the DLA to the memory.

[0117] The on-chip computer vision network may include an interface that determines that both the PVA and the DLA provide ready and valid signals before transmitting any control signals / address / data. Such an interface may provide separate phases and separate channels for transmitting control signals / address / data, as well as burst communication for continuous data transmission. This type of interface may comply with the ISO 26262 or IEC 61508 standards, but other standards and protocols may also be used.

[0118] In some examples, the SoC 804 can include, for example, a real-time ray tracing hardware accelerator as described in U.S. Patent Application No. 16 / 101,232, filed on August 10, 2018. The real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the position and extent of objects (e.g., within a world model) in order to generate a real-time visualization simulation for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for SONAR system simulation, for general wave propagation simulation, for comparison with LIDAR data for localization and / or other functional purposes, and / or for other uses. In some embodiments, one or more tree traversal units (TTUs) can be used to perform one or more ray tracing related operations.

[0119] The accelerator 814 (e.g., a hardware accelerator cluster) has a wide range of autonomous driving applications. The PVA can be a programmable vision accelerator that can be used in key processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are a good match for algorithm domains that require predictable processing, low power, and low latency. In other words, the PVA performs well in semi-dense or dense regular computations, even on small data sets that require predictable runtimes with low latency and low power. Thus, in the context of a platform for autonomous vehicles, the PVA is designed to run classical computer vision algorithms because they are effective in object detection and integer math operations.

[0120] For example, according to one embodiment of the technology, the PVA is used to perform computer stereo vision. In some examples, an algorithm based on semi-global matching can be used, but this is not intended to be limiting. Many applications for level 3-5 autonomous driving require instant motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.). The PVA can perform computer stereo vision functions on inputs from two monocular cameras.

[0121] In some examples, the PVA can be used to perform dense optical flow. Process raw RADAR data (e.g., using a 4D fast Fourier transform) to provide processed RADAR. In other examples, the PVA is used for time-of-flight depth processing, which, for example, processes raw time-of-flight data to provide processed time-of-flight data.

[0122] DLA can be used to run any type of network to enhance control and driving safety, including, for example, a neural network that outputs a confidence metric for each object detection. Such confidence values can be interpreted as probabilities or as providing a relative "weight" of each detection compared to other detections. The confidence value enables the system to make further decisions about which detections should be considered true positive detections rather than false positive detections. For example, the system can set a threshold for the confidence and consider only detections that exceed the threshold as true positive detections. In an automatic emergency braking (AEB) system, false positive detections can cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. DLA can run a neural network for regressing confidence values. The neural network can take as its input at least some subset of parameters, such as bounding box dimensions, a ground plane estimate obtained (e.g., from another subsystem), the output of an inertial measurement unit (IMU) sensor 866 related to the vehicle 800 orientation, distance, a 3D position estimate of an object obtained from a neural network and / or other sensors (such as a LIDAR sensor 864 or a RADAR sensor 860), etc.

[0123] The SoC 804 can include one or more data stores 816 (e.g., memory). The data store 816 can be on-chip memory of the SoC 804, which can store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and safety, the data store 816 can be large enough in capacity to store multiple instances of the neural network. The data store 816 can include an L2 or L3 cache 812. References to the data store 816 can include references to memory associated with PVAs, DLAs, and / or other accelerators 814 as described herein.

[0124] The SoC 804 may include one or more processors 810 (e.g., embedded processors). The processor 810 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management functions as well as security implementation related. The boot and power management processor may be part of the SoC 804 boot sequence and may provide runtime power management services. The boot power and management processor may provide clock and voltage programming, assist system low-power state transitions, manage the SoC 804 thermal and temperature sensors, and / or manage the SoC 804 power state. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC 804 may use the ring oscillator to detect the temperature of the CPU 806, GPU 808, and / or accelerator 814. If it is determined that the temperature exceeds a threshold, then the boot and power management processor may enter a temperature fault routine and place the SoC 804 in a lower power state and / or place the vehicle 800 in a driver safety stop mode (e.g., safely stop the vehicle 800).

[0125] The processor 810 may also include a set of embedded processors that may be used as an audio processing engine. The audio processing engine may be an audio subsystem that allows for full hardware support for multi-channel audio over multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor with dedicated RAM.

[0126] The processor 810 may also include an always-on processor engine that may provide the necessary hardware features to support low-power sensor management and wake-up use cases. The always-on processor engine may include a processor core, tightly coupled RAM, support peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0127] The processor 810 may also include a security cluster engine that includes a dedicated processor subsystem for handling the security management of automotive applications. The security cluster engine may include two or more processor cores, tightly coupled RAM, support peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In a security mode, the two or more cores may operate in a lockstep mode and act as a single core with comparison logic for detecting any differences between their operations.

[0128] The processor 810 may also include a real-time camera engine that may include a dedicated processor subsystem for handling real-time camera management.

[0129] The processor 810 may also include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0130] The processor 810 may include a video image compositor that may be a processing block (e.g., implemented on a microprocessor) that implements the video post-processing functions required for a video playback application to generate the final image for the player window. The video image compositor may perform lens distortion correction on the wide-angle camera 870, the surround camera 874, and / or the in-cab monitoring camera sensor. The in-cab monitoring camera sensor is preferably monitored by a neural network running on another instance of the advanced SoC, configured to identify in-cab events and respond accordingly. The in-cab system may perform lip reading to activate mobile phone services and make calls, dictate emails, change the vehicle destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode and are disabled otherwise.

[0131] The video image compositor may include enhanced temporal noise reduction for spatial and temporal noise reduction. For example, in the case of motion in the video, the noise reduction appropriately weights the spatial information, reducing the weight of the information provided by adjacent frames. In the case where an image or a portion of the image does not include motion, the temporal noise reduction performed by the video image compositor may use information from a previous image to reduce the noise in the current image.

[0132] The video image compositor may also be configured to perform stereo correction on input stereo lens frames. When the operating system desktop is in use and the GPU 808 does not need to continuously render new surfaces, the video image compositor may be further used for user interface composition. Even when the GPU 808 is powered on and active for 3D rendering, the video image compositor may be used to relieve the burden on the GPU 808 to improve performance and responsiveness.

[0133] The SoC 804 may also include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block for receiving video and inputs from cameras and related pixel input functions. The SoC 804 may also include an input / output controller that may be software-controlled and may be used to receive I / O signals not committed to a specific role.

[0134] The SoC 804 may also include a wide range of peripheral device interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. The SoC 804 can be used to process data from cameras (connected via Gigabit Multimedia Serial Link and Ethernet), sensors (such as LIDAR sensor 864, RADAR sensor 860, etc. that can be connected via Ethernet), data from bus 802 (such as the speed of vehicle 800, steering wheel position, etc.), and data from GNSS sensor 858 (connected via Ethernet or CAN bus). The SoC 804 may also include dedicated high-performance large-capacity storage controllers, which may include their own DMA engines and can be used to free the CPU 806 from routine data management tasks.

[0135] The SoC 804 can be an end-to-end platform with a flexible architecture that spans automation levels 3 - 5, thus providing an integrated functional safety architecture for a platform that utilizes and efficiently uses computer vision and ADAS technologies to achieve diversity and redundancy, along with deep learning tools to provide a flexible and reliable driving software stack. The SoC 804 can be faster, more reliable, and even more energy-efficient and space-efficient than conventional systems. For example, when combined with the CPU 806, GPU 808, and data storage 816, the accelerator 814 can provide a fast and efficient platform for level 3 - 5 autonomous vehicles.

[0136] Thus, this technology provides capabilities and functions that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on CPUs that can be configured using high-level programming languages such as the C programming language to perform various processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for in-vehicle ADAS applications and for practical level 3 - 5 autonomous vehicles.

[0137] In contrast to conventional systems, the technology described herein allows multiple neural networks to be executed simultaneously and / or sequentially by providing a CPU complex, a GPU complex, and a hardware accelerator cluster, and combining the results to achieve level 3 - 5 autonomous driving functions. For example, a CNN executed on the DLA or dGPU (such as GPU 820) can include text and word recognition, allowing a supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA may also include a neural network capable of recognizing, interpreting, and providing semantic understanding of the signs and passing that semantic understanding to a path planning module running on the CPU complex.

[0138] As another example, multiple neural networks can run simultaneously, as required for level 3, 4, or 5 driving. For example, a warning sign indicating "Caution: Flashing lights indicate icy conditions" together with the electric lights can be interpreted independently or jointly by several neural networks. The sign itself can be recognized as a traffic sign by a first neural network deployed (e.g., a trained neural network), and the text "Flashing lights indicate icy conditions" can be interpreted by a second neural network deployed, which informs the vehicle's path planning software (preferably executed on the CPU complex) that when the flashing lights are detected, there are icy conditions. The flashing lights can be recognized by operating a third neural network deployed over multiple frames, which informs the vehicle's path planning software of the presence (or absence) of the flashing lights. All three neural networks can run simultaneously, for example, within the DLA and / or on the GPU 808.

[0139] In some examples, the CNNs for face recognition and owner recognition can use data from the camera sensors to identify the presence of an authorized driver and / or owner of the vehicle 800. A processing engine that is always on the sensor can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in a security mode, to disable the vehicle when the owner leaves the vehicle. In this way, the SoC 804 provides security against theft and / or carjacking.

[0140] In another example, the CNN for emergency vehicle detection and recognition can use data from the microphone 896 to detect and identify an emergency vehicle siren. In contrast to conventional systems that use a general classifier to detect the siren and manually extract features, the SoC 804 uses a CNN to classify environmental and urban sounds as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to recognize the relative closing rate of an emergency vehicle (e.g., by using the Doppler effect). The CNN can also be trained to recognize emergency vehicles specific to the local area in which the vehicle is operating, as recognized by the GNSS sensor 858. Thus, for example, when operating in Europe, the CNN will seek to detect European sirens, and when in the United States, the CNN will seek to identify only North American sirens. Once an emergency vehicle is detected, with the assistance of the ultrasonic sensor 862, a control program can be used to execute an emergency vehicle safety routine, slowing down the vehicle, pulling over to the side of the road, stopping the vehicle, and / or idling the vehicle until the emergency vehicle passes.

[0141] The vehicle may include a CPU 818 (e.g., a discrete CPU or dCPU) that may be coupled to the SoC 804 via a high-speed interconnect (e.g., PCIe). The CPU 818 may include, for example, an X86 processor. The CPU 818 may be used to perform any of a variety of functions, including, for example, arbitrating potentially inconsistent results between the ADAS sensors and the SoC 804, and / or monitoring the status and health of the controller 836 and / or the infotainment SoC 830.

[0142] The vehicle 800 may include a GPU 820 (e.g., a discrete GPU or dGPU) that may be coupled to the SoC 804 via a high-speed interconnect (e.g., NVIDIA's NVLINK). The GPU 820 may provide additional artificial intelligence capabilities, for example, by executing redundant and / or different neural networks, and may be used to train and / or update neural networks at least in part based on inputs (e.g., sensor data) from the sensors of the vehicle 800.

[0143] The vehicle 800 may further include a network interface 824 that may include one or more wireless antennas 826 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 824 may be used to enable wireless connections to the cloud (e.g., to the server 878 and / or other network devices), to other vehicles, and / or to computing devices (e.g., the passenger's client device) via the Internet. To communicate with other vehicles, a direct link may be established between the two vehicles, and / or an indirect link may be established (e.g., across a network and via the Internet). The direct link may be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link may provide the vehicle 800 with information about vehicles approaching the vehicle 800 (e.g., vehicles in front of, beside, and / or behind the vehicle 800). This function may be part of the cooperative adaptive cruise control function of the vehicle 800.

[0144] The network interface 824 may include an SoC that provides modulation and demodulation functions and enables the controller 836 to communicate via a wireless network. The network interface 824 may include a radio frequency front end for upconverting from baseband to radio frequency and downconverting from radio frequency to baseband. The frequency conversion may be performed by a known process and / or may be performed using a super-heterodyne process. In some examples, the radio frequency front end functions may be provided by a separate chip. The network interface may include wireless capabilities for communicating via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0145] Vehicle 800 may also include a data store 828 that may include off-chip (e.g., outside of SoC 804) storage devices. The data store 828 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard drives, and / or other components and / or devices that can store at least one bit of data.

[0146] Vehicle 800 may also include a GNSS sensor 858. The GNSS sensor 858 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used to assist mapping, perception, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 858 may be used, including, for example and without limitation, a GPS using a USB connector with an Ethernet to serial (RS-232) bridge.

[0147] Vehicle 800 may also include a RADAR sensor 860. The RADAR sensor 860 may be used by the vehicle 800 for remote vehicle detection even in dark and / or adverse weather conditions. The RADAR functional safety level may be ASIL B. The RADAR sensor 860 may use CAN and / or bus 802 (e.g., to transmit data generated by the RADAR sensor 860) for control as well as access to object tracking data and, in some examples, access Ethernet to access raw data. A variety of RADAR sensor types may be used. For example and without limitation, the RADAR sensor 860 may be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor is used.

[0148] The RADAR sensor 860 may include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, and so on. In some examples, long-range RADAR may be used for adaptive cruise control functions. The long-range RADAR system may provide a wide field of view (e.g., within 250 m) achieved through two or more independent scans. The RADAR sensor 860 may help distinguish between static and moving objects and may be used by the ADAS system for emergency braking assistance and forward collision warning. The long-range RADAR sensor may include a single station multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the central four antennas may create a focused beam pattern that is designed to record the surroundings of the vehicle 800 at a higher rate with minimal traffic interference from adjacent lanes. The other two antennas may extend the field of view, making it possible to quickly detect vehicles entering or leaving the lane of the vehicle 800.

[0149] As an example, a mid-range RADAR system can include a range of up to 860 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 850 degrees (rear). A short-range RADAR system can include, but is not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor the rear and the blind spots alongside the vehicle.

[0150] The short-range RADAR system can be used in an ADAS system for blind spot detection and / or lane change assistance.

[0151] Vehicle 800 can also include ultrasonic sensors 862. Ultrasonic sensors 862 that can be placed in the front, rear, and / or sides of vehicle 800 can be used for parking assistance and / or creating and updating an occupancy grid. A variety of ultrasonic sensors 862 can be used, and different ultrasonic sensors 862 can be used for different detection ranges (e.g., 2.5 m, 4 m). Ultrasonic sensors 862 can operate at ASIL B, a functional safety level.

[0152] Vehicle 800 can include a LIDAR sensor 864. The LIDAR sensor 864 can be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensor 864 can be at ASIL B, a functional safety level. In some examples, vehicle 800 can include multiple LIDAR sensors 864 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a gigabit Ethernet switch).

[0153] In some examples, the LIDAR sensor 864 may be able to provide a list of objects and their distances for a 360-degree field of view. Commercially available LIDAR sensors 864 can have, for example, an advertised range of approximately 800 m, an accuracy of 2 cm - 3 cm, and support for an 800 Mbps Ethernet connection. In some examples, one or more non-protruding LIDAR sensors 864 can be used. In such examples, the LIDAR sensor 864 can be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of vehicle 800. In such examples, the LIDAR sensor 864 can provide a field of view of up to 120 degrees horizontally and 35 degrees vertically, even for low-reflectivity objects, with a range of 200 m. The front-mounted LIDAR sensor 864 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0154] In some examples, LIDAR technologies such as 3D flash LIDAR can also be used. 3D flash LIDAR uses the flash of a laser as the emission source to illuminate the vehicle's surrounding environment up to approximately 200 m. The flash LIDAR unit includes a receiver that records the laser pulse transit time and the reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LIDAR can allow for the generation of highly accurate and distortion-free images of the surrounding environment using each laser flash. In some examples, four flash LIDAR sensors can be deployed, one on each side of the vehicle 800. Available 3D flash LIDAR systems include solid-state 3D staring array LIDAR cameras (e.g., non-scanning LIDAR devices) that have no moving parts other than a fan. The flash LIDAR device can use Class I (eye-safe) laser pulses of 5 nanoseconds per frame and can capture the reflected laser in the form of 3D range point clouds and co-registered intensity data. By using flash LIDAR and since flash LIDAR is a solid-state device with no moving parts, the LIDAR sensor 864 can be less susceptible to motion blur, vibration, and / or shock.

[0155] The vehicle can also include an IMU sensor 866. In some examples, the IMU sensor 866 can be located at the center of the rear axle of the vehicle 800. The IMU sensor 866 can include, for example and without limitation, accelerometers, magnetometers, gyroscopes, magnetic compasses, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 866 can include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 866 can include an accelerometer, a gyroscope, and a magnetometer.

[0156] In some embodiments, the IMU sensor 866 can be implemented as a miniature high-performance GPS-aided inertial navigation system (GPS / INS) that combines microelectromechanical system (MEMS) inertial sensors, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 866 can enable the vehicle 800 to estimate the heading without input from a magnetic sensor by directly observing the speed change from the GPS to the IMU sensor 866 and correlating it. In some examples, the IMU sensor 866 and the GNSS sensor 858 can be integrated into a single unit.

[0157] The vehicle can include microphones 896 placed in and / or around the vehicle 800. Among other things, the microphones 896 can be used for emergency vehicle detection and identification.

[0158] The vehicle may also include any number of camera types, including a stereo camera 868, a wide-angle camera 870, an infrared camera 872, a surround camera 874, a long-range and / or mid-range camera 898, and / or other camera types. These cameras can be used to capture image data around the entire periphery of the vehicle 800. The camera types used depend on the embodiment and the requirements of the vehicle 800, and any combination of camera types can be used to provide the necessary coverage around the vehicle 800. Additionally, the number of cameras can vary according to the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As an example and without limitation, these cameras can support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described in more detail herein with respect to Figure 8A and Figure 8B is described in more detail.

[0159] The vehicle 800 may also include a vibration sensor 842. The vibration sensor 842 can measure the vibration of components of the vehicle such as the axles. For example, a change in vibration can indicate a change in the road surface. In another example, when two or more vibration sensors 842 are used, the difference between the vibrations can be used to determine the friction or slip of the road surface (e.g., when there is a vibration difference between a powered drive axle and a free-spinning axle).

[0160] The vehicle 800 may include an ADAS system 838. In some examples, the ADAS system 838 may include a SoC. The ADAS system 838 may include autonomous / adaptive / automated cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0161] The ACC system can use RADAR sensors 860, LIDAR sensors 864, and / or cameras. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately in front of the vehicle 800 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Lateral ACC performs distance keeping and, when necessary, advises the vehicle 800 to change lanes. Lateral ACC is related to other ADAS applications such as LCA and CWS.

[0162] The CACC uses information from other vehicles, which can be received indirectly from other vehicles via the network interface 824 and / or the wireless antenna 826 via a wireless link or through a network connection (e.g., via the Internet). The direct link can be provided by a vehicle-to-vehicle (V2V) communication link, while the indirect link can be an infrastructure-to-vehicle (I2V) communication link. Generally, the V2V communication concept provides information about the vehicle immediately in front (e.g., the vehicle immediately in front of vehicle 800 and in the same lane as it), while the I2V communication concept provides information about traffic further ahead. The CACC system can include either or both of the I2V and V2V information sources. Given the information of the vehicle in front of vehicle 800, the CACC can be more reliable, and it has the potential to improve the smoothness of traffic flow and reduce road congestion.

[0163] The FCW system is designed to alert the driver to a hazard so that the driver can take corrective action. The FCW system uses a front camera and / or RADAR sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibration component. The FCW system can provide warnings in the form of, for example, sound, visual warnings, vibration, and / or rapid braking pulses.

[0164] The AEB system detects an impending front collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system can use a front camera and / or RADAR sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid the collision, and if the driver does not take corrective action, then the AEB system can automatically apply the brakes in an effort to prevent or at least mitigate the impact of the predicted collision. The AEB system can include technologies such as dynamic brake support and / or collision imminent braking.

[0165] The LDW system provides visual, auditory, and / or tactile warnings such as steering wheel or seat vibration to alert the driver when vehicle 800 crosses a lane marking. When the driver indicates an intentional lane departure by activating the turn signal, the LDW system is not activated. The LDW system can use a front-side-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibration component.

[0166] The LKA system is a variant of the LDW system. If the vehicle 800 starts to leave the lane, then the LKA system provides a steering input or braking to correct the vehicle 800.

[0167] The BSW system detects and warns the driver of vehicles in the vehicle's blind spot. The BSW system can provide visual, audible, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses the turn signal. The BSW system can use a rear-facing camera and / or RADAR sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0168] The RCTW system can provide visual, audible, and / or tactile notifications when an object is detected outside the rear camera range while the vehicle 800 is in reverse. Some RCTW systems include AEB to ensure that the vehicle brakes are applied to avoid a crash. The RCTW system can use one or more rear RADAR sensors 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0169] Conventional ADAS systems may be prone to false positive results, which can be annoying and distracting to the driver, but are typically not catastrophic because the ADAS system alerts the driver and allows the driver to decide if a safe condition truly exists and act accordingly. However, in an autonomous vehicle 800, in the case of conflicting results, the vehicle 800 itself must decide whether to heed the results from the primary computer or an auxiliary computer (e.g., the first controller 836 or the second controller 836). For example, in some embodiments, the ADAS system 838 can be a backup and / or auxiliary computer for providing perception information to a redundant computer sanity module. The redundant computer sanity monitor can run redundant and diverse software on hardware components to detect faults in perception and dynamic driving tasks. The output from the ADAS system 838 can be provided to the supervisory MCU. If the outputs from the primary computer and the auxiliary computer conflict, then the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0170] In some examples, the host computer can be configured to provide a confidence score to the supervisory MCU, indicating the host computer's confidence in the selected result. If the confidence score exceeds a threshold, then the supervisory MCU can follow the direction of the host computer, regardless of whether the secondary computer provides conflicting or inconsistent results. In cases where the confidence score does not meet the threshold and where the host computer and the secondary computer indicate different results (e.g., conflict), the supervisory MCU can arbitrate between these computers to determine an appropriate result.

[0171] The supervisory MCU can be configured to run a neural network that is trained and configured to determine, at least in part based on outputs from the host computer and the secondary computer, conditions under which the secondary computer provides a false alarm. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer can be trusted and when it cannot. For example, when the secondary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying a metallic object that is not in fact dangerous, such as a drainage grate or manhole cover that triggers an alarm. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to disregard the LDW when a cyclist or pedestrian is present and lane departure is actually the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU can include at least one of a DLA or a GPU suitable for running the neural network with associated memory. In a preferred embodiment, the supervisory MCU can include components of the SoC 804 and / or be included as a component of the SoC 804.

[0172] In other examples, the ADAS system 838 can include a secondary computer that performs ADAS functions using traditional computer vision rules. In this way, the secondary computer can use classical computer vision rules (if-then), and the presence of a neural network in the supervisory MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identity make the overall system more fault-tolerant, especially for faults caused by software (or software-hardware interface) functions. For example, if there is a software vulnerability or error in the software running on the host computer and the non-identical software code running on the secondary computer provides the same overall result, then the supervisory MCU can be more confident that the overall result is correct and that the vulnerability in the software or hardware on the host computer does not cause a substantial error.

[0173] In some examples, the output of the ADAS system 838 can be fed to the perception block of the main computer and / or the dynamic driving task block of the main computer. For example, if the ADAS system 838 indicates a forward collision warning due to an object being immediately in front, then the perception block can use that information when identifying the object. In other examples, the auxiliary computer can have its own neural network, which is trained and thus reduces the risk of false positives as described herein.

[0174] The vehicle 800 can also include an infotainment SoC 830 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system can not be an SoC and can include two or more discrete components. The infotainment SoC 830 can include a combination of hardware and software that can be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.), and / or information services (e.g., navigation systems, rear parking assistance, radio data systems, vehicle-related information such as fuel level, total distance covered, brake fuel level, oil level, door open / close, air filter information, etc.) to the vehicle 800. For example, the infotainment SoC 830 can include a radio, disc player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, WiFi, steering wheel audio controls, hands-free voice controls, a head-up display (HUD), an HMI display 834, a telematics device, a control panel (e.g., for controlling various components, features, and / or systems, and / or interacting with them), and / or other components. The infotainment SoC 830 can further be used to provide information (e.g., visual and / or auditory) to the users of the vehicle, such as information from the ADAS system 838, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0175] The infotainment SoC 830 can include GPU functionality. The infotainment SoC 830 can communicate with other devices, systems, and / or components of the vehicle 800 via a bus 802 (e.g., a CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 830 can be coupled to a supervisory MCU such that in the event of a failure of the main controller 836 (e.g., the main and / or standby computer of the vehicle 800), the GPU of the infotainment system can perform some autonomous driving functions. In such examples, the infotainment SoC 830 can place the vehicle 800 in a driver-safe parking mode as described herein.

[0176] Vehicle 800 may further include an instrument cluster 832 (such as a digital instrument panel, an electronic instrument cluster, a digital instrument surface panel, etc.). The instrument cluster 832 may include a controller and / or a supercomputer (such as a discrete controller or supercomputer). The instrument cluster 832 may include a set of instruments, such as a speedometer, a fuel level, an oil pressure, a tachometer, an odometer, a turn indicator, a shift position indicator, a seat belt warning light, a parking brake warning light, an engine malfunction light, a safety airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 830 and the instrument cluster 832. In other words, the instrument cluster 832 may be included as part of the infotainment SoC 830, or vice versa.

[0177] Figure 8D A system schematic diagram for communication between a cloud-based server and Figure 8A an exemplary autonomous vehicle 800 according to some embodiments of the present disclosure. The system 876 may include a server 878, a network 890, and vehicles including the vehicle 800. The server 878 may include multiple GPUs 884(A)-884(H) (collectively referred to herein as GPUs 884), PCIe switches 882(A)-882(H) (collectively referred to herein as PCIe switches 882), and / or CPUs 880(A)-880(B) (collectively referred to herein as CPUs 880). The GPUs 884, CPUs 880, and PCIe switches may be interconnected by high-speed interconnects and / or PCIe connections 886 such as, for example and without limitation, the NVLink interface 888 developed by NVIDIA. In some examples, the GPUs 884 are connected via NVLink and / or an NVSwitch SoC, and the GPUs 884 and the PCIe switches 882 are connected via a PCIe interconnect. Although eight GPUs 884, two CPUs 880, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each of the servers 878 may include any number of GPUs 884, CPUs 880, and / or PCIe switches. For example, each of the servers 878 may include eight, sixteen, thirty-two, and / or more GPUs 884.

[0178] Server 878 can receive image data via network 890 and from a vehicle, the image data representing an image showing an unexpected or changed road condition such as a recently started roadwork. Server 878 can transmit neural network 892, updated neural network 892, and / or map information 894 via network 890 and to the vehicle, including information about traffic and road conditions. Updates to map information 894 can include updates to HD map 822, such as information about construction sites, potholes, curves, floods, or other obstacles. In some examples, neural network 892, updated neural network 892, and / or map information 894 can be represented and / or generated based on data received from new training and / or data from any number of vehicles in the environment and / or experience from training performed at a data center (e.g., using server 878 and / or other servers).

[0179] Server 878 can be used to train a machine learning model (e.g., a neural network) based on training data. The training data can be generated by vehicles and / or can be generated in a simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., in cases where the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not labeled and / or preprocessed (e.g., in cases where the neural network does not require supervised learning). Training can be performed according to any one or more classes of machine learning techniques, including but not limited to the following classes: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and clustering analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, the machine learning model can be used by the vehicle (e.g., transmitted to the vehicle via network 890), and / or the machine learning model can be used by server 878 to remotely monitor the vehicle.

[0180] In some examples, server 878 can receive data from a vehicle and apply the data to a latest real-time neural network for real-time intelligent inference. Server 878 can include a deep learning supercomputer powered by GPU 884 and / or a dedicated AI computer, such as DGX and DGX Station machines developed by NVIDIA. However, in some examples, server 878 can include a deep learning infrastructure of a data center powered only by a CPU.

[0181] The deep learning infrastructure of server 878 may be capable of fast real-time inference and can use this ability to evaluate and verify the health of the processors, software, and / or associated hardware in vehicle 800. For example, the deep learning infrastructure may receive periodic updates from vehicle 800, such as an image sequence and / or objects located in the image sequence that vehicle 800 has identified (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure may run its own neural network to identify the objects and compare them with the objects identified by vehicle 800. If the results do not match and the infrastructure concludes that the AI in vehicle 800 has malfunctioned, then server 878 may transmit a signal to vehicle 800 instructing the fail-safe computer in vehicle 800 to take control, notify the passengers, and complete a safe parking operation.

[0182] For inference, server 878 may include a GPU 884 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT). The combination of a GPU-powered server and inference acceleration may enable real-time response. In other examples, such as when performance is less critical, a CPU, FPGA, and other processor-powered servers may be used for inference.

[0183] Example computing device

[0184] Figure 9 is a block diagram of an example computing device 900 suitable for implementing some embodiments of the present disclosure. Computing device 900 may include an interconnect system 902 that directly or indirectly couples the following devices: a memory 904, one or more central processing units (CPUs) 906, one or more graphics processing units (GPUs) 908, a communication interface 910, input / output (I / O) ports 912, input / output components 914, a power supply 916, one or more presentation components 918 (e.g., (one or more) displays), and one or more logic units 920. In at least one embodiment, (one or more) computing devices 900 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more of GPUs 908 may include one or more vGPUs, one or more of CPUs 906 may include one or more vCPUs, and / or one or more of logic units 920 may include one or more virtual logic units. Thus, (one or more) computing devices 900 may include discrete components (e.g., full GPUs dedicated to computing device 900), virtual components (e.g., a portion of a GPU dedicated to computing device 900), or a combination thereof.

[0185] AlthoughFigure 9 Each of the boxes is shown as being connected via circuitry through an interconnect system 902, but this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component 918 (such as a display device) may be considered an I / O component 914 (e.g., if the display is a touch screen). As another example, the CPU 906 and / or the GPU 908 may include memory (e.g., memory 904 may represent a storage device in addition to the memory of the GPU 908, the CPU 906, and / or other components). In other words, Figure 9 the computing devices are illustrative only. No distinction is made between such categories as “workstation,” “server,” “laptop computer,” “desktop computer,” “tablet computer,” “client device,” “mobile device,” “handheld device,” “gaming console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, because all are considered within the Figure 9 scope of the computing devices.

[0186] The interconnect system 902 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system 902 may include one or more bus or link types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU 906 may be directly connected to the memory 904. Further, the CPU 906 may be directly connected to the GPU 908. In cases where there are direct or point-to-point connections between components, the interconnect system 902 may include a PCIe link to effect the connection. In these examples, a PCI bus need not be included in the computing device 900.

[0187] The memory 904 may include any computer-readable medium of a variety of computer-readable media. A computer-readable medium may be any available medium that can be accessed by the computing device 900. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0188] A computer storage medium can include volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 904 can store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements such as an operating system). Computer storage media can include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic tape cartridges, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computing device 900. As used herein, a computer storage medium does not include a signal per se.

[0189] A computer storage medium can embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal such as a carrier wave or other transmission mechanism, and includes any information delivery medium. The term "modulated data signal" can refer to a signal that sets or changes one or more of its characteristics in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as a wired network or a direct wired connection) and wireless media (such as acoustic, RF, infrared, and other wireless media). Combinations of any of the above should also be included within the scope of computer-readable media.

[0190] CPU 906 can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. Each of CPU 906 can include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling numerous software threads simultaneously. CPU 906 can include any type of processor and can include different types of processors depending on the type of computing device 900 implemented (e.g., a processor with fewer cores for a mobile device and a processor with more cores for a server). For example, depending on the type of computing device 900, the processor can be an advanced RISC machine (ARM) processor implemented using reduced instruction set computing (RISC) or an x86 processor implemented using complex instruction set computing (CISC). In addition to one or more microprocessors or supplementary coprocessors (such as a math coprocessor), computing device 900 can also include one or more CPU 906.

[0191] In addition to or instead of the (one or more) CPUs 906, the (one or more) GPUs 908 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 900 to perform one or more of the methods and / or processes described herein. One or more of the GPUs 908 may be an integrated GPU (e.g., one or more of the CPUs 906) and / or one or more of the GPUs 908 may be a discrete GPU. In an embodiment, one or more of the GPUs 908 may be a coprocessor of one or more of the CPUs 906. The GPUs 908 may be used by the computing device 900 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, the GPUs 908 may be used for general-purpose computing on the GPU (GPGPU). The GPUs 908 may include hundreds or thousands of cores capable of concurrently handling hundreds or thousands of software threads. The GPUs 908 may generate pixel data of an output image in response to a rendering command (e.g., a rendering command received from the CPU 906 via a host interface). The GPUs 908 may include a graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). The display memory may be included as part of the memory 904. The GPUs 908 may include two or more GPUs operating in parallel (e.g., via a link). The link may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs through a switch (e.g., using NVSwitch). When combined, each GPU 908 may generate pixel data or GPGPU data for different parts of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0192] In addition to or instead of CPU 906 and / or GPU 908, logic unit 920 may be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. In an embodiment, (one or more) CPU 906, (one or more) GPU 908, and / or (one or more) logic unit 920 may perform any combination of methods, processes, and / or portions thereof discretely or jointly. One or more of logic units 920 may be part of one or more of CPU 906 and / or GPU 908 and / or integrated in one or more of CPU 906 and / or GPU 908 and / or one or more of logic units 920 may be discrete components or otherwise external to CPU 906 and / or GPU 908. In an embodiment, one or more of logic units 920 may be a coprocessor of one or more of CPU 906 and / or one or more of GPU 908.

[0193] Examples of logic unit 920 include one or more processing cores and / or its components, such as data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree traversal unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU), application specific integrated circuit (ASIC), floating point unit (FPU), input / output (I / O) element, peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) element, etc.

[0194] Communication interface 910 may include one or more receivers, transmitters, and / or transceivers that enable computing device 900 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communication. Communication interface 910 may include components and functionality to enable communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., over Ethernet or InfiniBand communication), low power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, one or more of logic units 920 and / or communication interface 910 may include one or more data processing units (DPUs) to directly transfer data received over the network and / or over interconnect system 902 to one or more GPUs 908 (e.g., the memory of).

[0195] The I / O port 912 can enable the computing device 900 to be logically coupled to other devices including I / O components 914, one or more presentation components 918, and / or other components, some of which may be built into (e.g., integrated in) the computing device 900. Illustrative I / O components 914 include microphones, mice, keyboards, joysticks, game pads, game controllers, dish satellite antennas, scanners, printers, wireless devices, and the like. The I / O components 914 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by the user. In some cases, the input can be transmitted to appropriate network elements for further processing. The NUI can implement any combination of speech recognition, stylus recognition, face recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of the computing device 900. The computing device 900 can include a depth camera for gesture detection and recognition, such as a stereo camera system, an infrared camera system, an RGB camera system, touch screen technology, and combinations thereof. Additionally, the computing device 900 can include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables the detection of motion. In some examples, the computing device 900 can use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0196] The power supply 916 can include hardwired power, battery power, or a combination thereof. The power supply 916 can provide power to the computing device 900 to enable the components of the computing device 900 to operate.

[0197] The presentation component 918 can include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The presentation component 918 can receive data from other components (e.g., the GPU 908, the CPU 906, the DPU, etc.) and output the data (e.g., as images, videos, sounds, etc.).

[0198] Example data center

[0199] Figure 10 An example data center 1000 that can be used in at least one embodiment of the present disclosure is shown. The data center 1000 can include a data center infrastructure layer 1010, a framework layer 1020, a software layer 1030, and / or an application layer 1040.

[0200] As Figure 10As shown, the data center infrastructure layer 1010 may include a resource coordinator 1012, grouped computing resources 1014, and node computing resources ("node C.R.") 1016(1)-1016(N), where "N" represents any whole positive integer. In at least one embodiment, node C.R.s 1016(1)-1016(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules, and / or cooling modules, and so on. In some embodiments, one or more of the node C.R.s 1016(1)-1016(N) may correspond to a server having one or more of the above computing resources. Additionally, in some embodiments, node C.R.s 1016(1)-10161(N) may include one or more virtual components, such as vGPUs, vCPUs, etc., and / or one or more of the node C.R.s 1016(1)-1016(N) may correspond to a virtual machine (VM).

[0201] In at least one embodiment, the grouped computing resources 1014 may include separate groupings of node C.R.s 1016 housed within one or more racks (not shown), or many racks within a data center located at different geographical locations (also not shown). Separate groupings of node C.R.s 1016 within the grouped computing resources 1014 may include grouped computing, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several node C.R.s 1016 including CPUs, GPUs, DPUs, and / or other processors may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any combination of any number of power modules, cooling modules, and / or network switches.

[0202] The resource coordinator 1012 may configure or otherwise control one or more of the node C.R.s 1016(1)-1016(N) and / or the grouped computing resources 1014. In at least one embodiment, the resource coordinator 1012 may include a software design infrastructure (SDI) management entity for the data center 1000. The resource coordinator 1012 may include hardware, software, or some combination thereof.

[0203] In at least one embodiment, as Figure 10As shown, the framework layer 1020 may include a job scheduler 1033, a configuration manager 1034, a resource manager 1036, and / or a distributed file system 1038. The framework layer 1020 may include a framework for software 1032 that supports the software layer 1030 and / or one or more applications 1042 of the application layer 1040. The software 1032 or the application 1042 may respectively include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1020 may be, but is not limited to, a free and open-source software web application framework (such as Apache Spark TM (hereinafter referred to as "Spark")) of the type that can utilize the distributed file system 1038 for large-scale data processing (e.g., "big data"). In at least one embodiment, the job scheduler 1033 may include a Spark driver to facilitate scheduling of workloads supported by different layers of the data center 1000. The configuration manager 1034 may be able to configure different layers, such as the software layer 1030 and the framework layer 1020 (which includes Spark and the distributed file system 1038 for supporting large-scale data processing). The resource manager 1036 may be able to manage the clustered or grouped computing resources that are mapped to the distributed file system 1038 and the job scheduler 1032 or are allocated to support the distributed file system 1038 and the job scheduler 1033. In at least one embodiment, the clustered or grouped computing resources may include the grouped computing resources 1014 in the data center infrastructure layer 1010. The resource manager 1036 may coordinate with the resource coordinator 1012 to manage these mapped or allocated computing resources.

[0204] In at least one embodiment, the software 1032 included in the software layer 1030 may include software used by at least a portion of the node C.R.s 1016(1)-1016(N), the grouped computing resources 1014, and / or the distributed file system 1038 of the framework layer 1020. One or more types of software may include, but are not limited to, Internet web search software, email virus scanning software, database software, and streaming video content software.

[0205] In at least one embodiment, the applications 1042 included in the application layer 1040 may include one or more types of applications used by at least a portion of the nodes C.R.s 1016(1)-1016(N), the grouped computing resources 1014, and / or the distributed file system 1038 of the framework layer 1020. One or more types of applications may include, but are not limited to, any number of genomic applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0206] In at least one embodiment, any one of the configuration manager 1034, the resource manager 1036, and the resource coordinator 1012 may implement any number and type of self-modifying actions based on any amount and type of data obtained in any technically feasible manner. The self-modifying actions may save the data center operator of the data center 1000 from making potentially poor configuration decisions and may avoid underutilization and / or poorly performing portions of the data center.

[0207] According to one or more embodiments described herein, the data center 1000 may include tools, services, software, or other resources to train one or more machine learning models or use one or more machine learning models to predict or infer information. For example, (one or more) machine learning models may be trained by calculating weight parameters according to a neural network architecture by using the software and / or computing resources described above with respect to the data center 1000. In at least one embodiment, the trained or deployed machine learning models corresponding to one or more neural networks may be used to infer or predict information by using the resources described above with respect to the data center 1000 by using the weight parameters calculated by one or more training techniques (such as, but not limited to, those described herein).

[0208] In at least one embodiment, the data center 1000 may use a CPU, an application specific integrated circuit (ASIC), a GPU, an FPGA, and / or other hardware (or their corresponding virtual computing resources) to perform training and / or inference by using the above resources. In addition, one or more of the software and / or hardware resources described above may be configured to allow a user to train or perform a service for inferring information, such as image recognition, speech recognition, or other artificial intelligence services.

[0209] Example Network Environment

[0210] A network environment suitable for implementing embodiments of the present disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be implemented on one or more instances of a Figure 9 computing device 900 - e.g., each device may include similar components, features, and / or functions of the (one or more) computing device 900. Additionally, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of a data center 1000, an example of which is described herein with respect to Figure 10 more detail.

[0211] The components of the network environment may communicate with each other via a network, which may be wired, wireless, or both. The network may include multiple networks or one network among multiple networks. For example, the network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the public switched telephone network (PSTN)), and / or one or more private networks. In the case where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) may provide wireless connectivity.

[0212] A compatible network environment may include one or more peer - to - peer network environments (in which case, servers may not be included in the network environment) and one or more client - server network environments (in which case, one or more servers may be included in the network environment). In a peer - to - peer network environment, the functions described herein for servers may be implemented on any number of client devices.

[0213] In at least one embodiment, the network environment may include one or more cloud - based network environments, distributed computing environments, combinations thereof, etc. A cloud - based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework that supports a software layer and / or one or more applications of an application layer. The software or application may respectively include network - based service software or applications. In an embodiment, one or more client devices may use network - based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open - source software web application framework that may use a distributed file system for large - scale data processing (e.g., "big data").

[0214] Cloud-based network environments can provide cloud computing and / or cloud storage that perform any combination of the computing and / or data storage functions (or one or more parts thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., from one or more data centers that can be in a state, region, country, globally, etc.). If the connection to a user (e.g., a client device) is relatively close to an edge server, the core server can assign at least a portion of the function to the edge server. Cloud-based network environments can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0215] (One or more) client devices can include at least some of the components, features, and functions of the (one or more) example computing devices 900 described herein. By way of example and not limitation, a client device can be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance device or system, vehicle, boat, spacecraft, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, appliance, consumer electronic device, workstation, edge device, any combination of these depicted devices, or any other suitable device. Figure 9 The present disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, executed by a computer or other machine such as a personal digital assistant or other handheld device. Generally, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs particular tasks or implements particular abstract data types. The present disclosure can be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. The present disclosure can also be practiced in a distributed computing environment where tasks are performed by remote processing devices linked through a communications network.

[0216] The present disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, executed by a computer or other machine such as a personal digital assistant or other handheld device. Generally, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs particular tasks or implements particular abstract data types. The present disclosure can be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. The present disclosure can also be practiced in a distributed computing environment where tasks are performed by remote processing devices linked through a communications network.

[0217] As used herein, the recitation of "and / or" with respect to two or more elements shall be construed to refer to only one element or a combination of elements. For example, "element A, element B, and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. Further, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Still further, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0218] The subject matter of the present disclosure is described herein in detail to meet statutory requirements. However, the description itself is not intended to limit the scope of the disclosure. On the contrary, the inventors have contemplated that the claimed subject matter may also be embodied in other ways, including steps different from or combinations of steps similar to those described herein in connection with other current or future technologies. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of a method employed, these terms should not be construed as implying any particular order among or between the various steps disclosed herein unless the order of the steps is expressly recited.

[0219] Example paragraph

[0220] A: A method comprising: determining a first speed limit associated with a speed limit sign based at least on sensor data obtained using one or more sensors of a machine; determining a second speed limit associated with the speed limit sign based at least on map data representative of a map; using one or more rules and based at least on the first speed limit and the second speed limit, to determine a final speed limit associated with the speed limit sign; and causing the machine to perform one or more operations based at least on the final speed limit associated with the speed limit sign.

[0221] B: The method of paragraph A, wherein determining the final speed limit comprises: determining that the first speed limit matches the second speed limit; and based at least on the first speed limit matching the second speed limit, determining that the final speed limit comprises at least one of the first speed limit or the second speed limit.

[0222] C: The method of paragraph A or paragraph B, wherein determining the final speed limit includes: determining that the first speed limit is different from the second speed limit; and determining, at least based on the first speed limit being different from the second speed limit, to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign.

[0223] D: The method of any one of paragraphs A - C, wherein determining to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign includes: determining the distance traveled by the machine since determining the first speed limit; determining whether the distance is less than or equal to a threshold distance; and one of the following: determining, at least based on the distance being less than or equal to the threshold distance, to use the first speed limit as the final speed limit; or determining, at least based on the distance being greater than the threshold distance, to use the second speed limit as the final speed limit.

[0224] E: The method of any one of paragraphs A - D, wherein determining to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign includes: determining the time period elapsed since determining the first speed limit; determining whether the time period is less than or equal to a threshold time period; and one of the following: determining, at least based on the time period being less than or equal to the threshold time period, to use the first speed limit as the final speed limit; or determining, at least based on the time period being greater than the threshold time period, to use the second speed limit as the final speed limit.

[0225] F: The method of any one of paragraphs A - E, wherein determining to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign includes: determining the probability associated with the first speed limit; determining whether the probability is greater than a threshold probability; and one of the following: determining, at least based on the probability being greater than the threshold probability, to use the first speed limit as the final speed limit; or determining, at least based on the probability being less than the threshold probability, to use the second speed limit as the final speed limit.

[0226] G: The method of any one of paragraphs A - F, wherein determining to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign includes: determining whether the machine has switched from traveling along a first road associated with the first speed limit to traveling along a second road; and one of the following: determining to use the first speed limit as the final speed limit at least based on the machine continuing to travel along the first road; or determining to use the second speed limit as the final speed limit at least based on the machine switching to travel along the second road.

[0227] H: The method of any one of paragraphs A - G, further comprising: generating a speed limit segment, the speed limit segment including: sensed speed limit data including at least one of a nominal sensed speed limit and a conditional sensed speed limit, the first speed limit being at least based on the sensed speed limit data, and offset data indicating an accumulated distance associated with at least one of the nominal sensed speed limit and the conditional sensed speed limit; generating map - based speed limit data from which the second speed limit can be obtained, the map - based speed limit data including: a data frame obtained from streaming map data, the data frame including an updated map - based speed limit, and historical map data of a predetermined distance behind the machine cached when the machine moves along a travel path, the historical map data including cached map - based speed limits; and generating a speed limit profile by associating the offset data of the speed limit segment with the map - based speed limit data, wherein determining the final speed limit is at least based on the speed limit profile.

[0228] I: The method of any one of paragraphs A - H, wherein the data frame undergoes pre - processing before generating the speed limit profile, the pre - processing including at least one of the following: data conversion to convert the data frame into a format conducive to further processing, duplicate data removal of speed limit values, and removal of obsolete data.

[0229] J: A system, comprising: one or more processing units for: determining a first speed limit associated with a speed limit sign at least based on sensor data obtained using one or more sensors of a machine; determining a second speed limit associated with the speed limit sign at least based on map data representing a map; using one or more rules and at least based on the first speed limit and the second speed limit, to determine a final speed limit associated with the speed limit sign; and causing the machine to perform one or more operations at least based on the final speed limit associated with the speed limit sign.

[0230] K: The system of paragraph J, wherein determining the final speed limit includes: determining that the first speed limit matches the second speed limit; and determining that the final speed limit includes at least one of the first speed limit or the second speed limit, at least based on the first speed limit matching the second speed limit.

[0231] L: The system of paragraph J or paragraph K, wherein determining the final speed limit includes: determining that the first speed limit is different from the second speed limit; and determining to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign, at least based on the first speed limit being different from the second speed limit.

[0232] M: The system of any one of paragraphs J-L, wherein determining to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign includes: determining the distance traveled by the machine since determining the first speed limit; determining whether the distance is less than or equal to a threshold distance; and one of the following: determining to use the first speed limit as the final speed limit, at least based on the distance being less than or equal to the threshold distance; or determining to use the second speed limit as the final speed limit, at least based on the distance being greater than the threshold distance.

[0233] N: The system of any one of paragraphs J-M, wherein determining to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign includes: determining the time period elapsed since determining the first speed limit; determining whether the time period is less than or equal to a threshold time period; and one of the following: determining to use the first speed limit as the final speed limit, at least based on the time period being less than or equal to the threshold time period; or determining to use the second speed limit as the final speed limit, at least based on the time period being greater than the threshold time period.

[0234] O: The system of any one of paragraphs J-N, wherein determining to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign includes: determining the probability associated with the first speed limit; determining whether the probability is greater than a threshold probability; and one of the following: determining to use the first speed limit as the final speed limit, at least based on the probability being greater than the threshold probability; or determining to use the second speed limit as the final speed limit, at least based on the probability being less than the threshold probability.

[0235] P: A system of any one of paragraphs J - O, wherein determining to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign includes: determining whether the machine has switched from traveling along a first road associated with the first speed limit to traveling along a second road; and one of the following: determining to use the first speed limit as the final speed limit based at least on the machine continuing to travel along the first road; or determining to use the second speed limit as the final speed limit based at least on the machine switching to travel along the second road.

[0236] Q: A system of any one of paragraphs J - P, wherein the one or more processing units are further configured to: generate a speed limit segment, the speed limit segment including: sensed speed limit data including at least one of a nominal sensed speed limit and a conditional sensed speed limit, the first speed limit being at least based on the sensed speed limit data, and offset data indicating a cumulative distance associated with at least one of the nominal sensed speed limit and the conditional sensed speed limit; generate map - based speed limit data from which the second speed limit can be obtained, the map - based speed limit data including: data frames obtained from streaming map data, the data frames including updated map - based speed limits, and historical map data of a predetermined distance behind the machine cached when the machine moves along a travel path, the historical map data including cached map - based speed limits; and generate a speed limit profile by associating the offset data of the speed limit segment with the map - based speed limit data, wherein determining the final speed limit is at least based on the speed limit profile.

[0237] R: A system of any one of paragraphs J - Q, wherein the system is included in at least one of the following: a control system for an autonomous or semi - autonomous machine; a sensing system for an autonomous or semi - autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing optical transmission simulations; a system for performing collaborative content creation of 3D assets; a system for performing generative AI operations; a system for performing operations using large language models; a system for performing deep learning operations; a system implemented using edge devices; a system implemented using robots; a system for performing conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0238] S: A processor, comprising: one or more processing units configured to perform a fusion of map-based speed limit data and sensed speed limit data, wherein the fusion employs at least one rule to issue an output speed limit selected from among the map-based speed limit data, the sensed speed limit data, and cached map-based speed limit data stored in a memory unit in communication with the one or more processing units.

[0239] T: The processor of paragraph S, wherein the processor is included in at least one of the following: a control system for an autonomous or semi-autonomous machine; a sensing system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing optical transmission simulations; a system for performing collaborative content creation of 3D assets; a system for performing generative AI operations; a system for performing operations using a large language model; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system comprising one or more virtual machines VM; a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

Claims

1. A method comprising: determining a first speed limit associated with the speed limit sign based at least on sensor data obtained using one or more sensors of the machine; determining a second speed limit associated with the speed limit sign based on at least map data representing a map; determining a final speed limit associated with the speed limit sign using one or more rules and based on at least the first speed limit and the second speed limit; as well as The machine is caused to perform one or more operations based at least on the final speed limit associated with the speed limit sign.

2. The method of claim 1 , wherein determining the final speed limit comprises: determining that the first speed limit matches the second speed limit; as well as Based at least on the first speed limit matching the second speed limit, determining the final speed limit includes at least one of the first speed limit or the second speed limit.

3. The method of claim 1 , wherein determining the final speed limit comprises: determining that the first speed limit is different from the second speed limit; as well as Based at least on the first speed limit being different from the second speed limit, a determination is made to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign.

4. The method of claim 3, wherein determining to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign comprises: determining a distance traveled by the machine since determining the first speed limit; determining whether the distance is less than or equal to a threshold distance; as well as One of the following: determining to use the first speed limit as the final speed limit based at least on the distance being less than or equal to the threshold distance; or Based at least on the distance being greater than the threshold distance, it is determined to use the second speed limit as the final speed limit.

5. The method of claim 3, wherein determining to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign comprises: determining a period of time that has elapsed since determining said first speed limit; determining whether the time period is less than or equal to a threshold time period; as well as One of the following: Determining to use the first speed limit as the final speed limit based at least on the time period being less than or equal to the threshold time period; or Based at least on the time period being greater than the threshold time period, a determination is made to use the second speed limit as the final speed limit.

6. The method of claim 3, wherein determining to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign comprises: determining a probability associated with the first speed limit; determining whether the probability is greater than a threshold probability; as well as One of the following: determining to use the first speed limit as the final speed limit based at least on the probability being greater than the threshold probability; or Based at least on the probability being less than the threshold probability, a determination is made to use the second speed limit as the final speed limit.

7. The method of claim 3, wherein determining to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign comprises: determining whether the machine has switched from traveling along a first road associated with the first speed limit to traveling along a second road; as well as One of the following: determining to use the first speed limit as the final speed limit based at least on the machine continuing to travel along the first road; or A determination to use the second speed limit as the final speed limit is made based at least on the machine switching to travel along the second road.

8. The method according to claim 1, further comprising: Generate a speed limit segment, the speed limit segment comprising: perceived speed limit data comprising at least one of a nominal perceived speed limit and a conditional perceived speed limit, the first speed limit being based at least on the perceived speed limit data, and offset data indicating a cumulative distance associated with at least one of the nominal perceived speed limit and the conditional perceived speed limit; generating map-based speed limit data from which the second speed limit can be derived, the map-based speed limit data comprising: a data frame obtained from the streaming map data, the data frame including an updated map-based speed limit, and historical map data cached a predetermined distance behind the machine as the machine moves along a path of travel, the historical map data including a cached map-based speed limit; and generating a speed limit profile by associating the offset data for the speed limit segment with the map-based speed limit data, Wherein, determining the final speed limit is based at least on the speed limit profile.

9. The method of claim 8, wherein the data frame undergoes pre-processing before generating the speed limit profile, the pre-processing comprising at least one of: data conversion to convert the data frame into a format that is conducive to further processing, de-duplication of speed limit values, and removal of obsolete data.

10. A system comprising: One or more processing units for: determining a first speed limit associated with the speed limit sign based at least on sensor data obtained using one or more sensors of the machine; determining a second speed limit associated with the speed limit sign based on at least map data representing a map; determining a final speed limit associated with the speed limit sign using one or more rules and based on at least the first speed limit and the second speed limit; as well as The machine is caused to perform one or more operations based at least on the final speed limit associated with the speed limit sign.

11. The system of claim 10, wherein the determination of the final speed limit comprises: determining that the first speed limit matches the second speed limit; as well as Based at least on the first speed limit matching the second speed limit, determining the final speed limit includes at least one of the first speed limit or the second speed limit.

12. The system of claim 10, wherein the determination of the final speed limit comprises: determining that the first speed limit is different from the second speed limit; as well as Based at least on the first speed limit being different from the second speed limit, a determination is made to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign.

13. The system according to claim 12, wherein: Determining to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign includes: determining a distance traveled by the machine since determining the first speed limit; determining whether the distance is less than or equal to a threshold distance; and One of the following: determining to use the first speed limit as the final speed limit based at least on the distance being less than or equal to the threshold distance; or Based at least on the distance being greater than the threshold distance, a determination is made to use the second speed limit as the final speed limit.

14. The system of claim 12, wherein determining to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign comprises: determining a period of time that has elapsed since determining said first speed limit; determining whether the time period is less than or equal to a threshold time period; as well as One of the following: Determining to use the first speed limit as the final speed limit based at least on the time period being less than or equal to the threshold time period; or Based at least on the time period being greater than the threshold time period, a determination is made to use the second speed limit as the final speed limit.

15. The system of claim 12, wherein determining to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign comprises: determining a probability associated with the first speed limit; determining whether the probability is greater than a threshold probability; as well as One of the following: determining to use the first speed limit as the final speed limit based at least on the probability being greater than the threshold probability; or Based at least on the probability being less than the threshold probability, a determination is made to use the second speed limit as the final speed limit.

16. The system of claim 12, wherein determining to use one of the first speed limit or the second speed limit as the final speed limit associated with the speed limit sign comprises: determining whether the machine has switched from traveling along a first road associated with the first speed limit to traveling along a second road; as well as One of the following: determining to use the first speed limit as the final speed limit based at least on the machine continuing to travel along the first road; or A determination to use the second speed limit as the final speed limit is made based at least on the machine switching to travel along the second road.

17. The system of claim 10, wherein the one or more processing units are further configured to: Generate a speed limit segment, the speed limit segment comprising: perceived speed limit data comprising at least one of a nominal perceived speed limit and a conditional perceived speed limit, the first speed limit being based at least on the perceived speed limit data, and offset data indicating a cumulative distance associated with at least one of the nominal perceived speed limit and the conditional perceived speed limit; generating map-based speed limit data from which the second speed limit can be derived, the map-based speed limit data comprising: a data frame obtained from the streaming map data, the data frame including an updated map-based speed limit, and historical map data cached a predetermined distance behind the machine as the machine moves along a path of travel, the historical map data including a cached map-based speed limit; and generating a speed limit profile by associating the offset data for the speed limit segment with the map-based speed limit data, Wherein, determining the final speed limit is based at least on the speed limit profile.

18. The system of claim 10, wherein the system is included in at least one of the following: control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; A system for performing simulation operations; Systems for performing digital twin operations; A system for performing light transport simulations; A system for performing collaborative content creation of 3D assets; Systems for performing generative AI operations; Systems for performing operations using large language models; Systems for performing deep learning operations; Systems implemented using edge devices; Systems implemented using robots; Systems for performing conversational AI operations; Systems for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

19. A processor comprising: One or more processing units for performing fusion of map-based speed limit data and perceived speed limit data, wherein the fusion employs at least one rule to issue an output speed limit selected from one of the map-based speed limit data, the perceived speed limit data, and cached map-based speed limit data stored in a memory unit in communication with the one or more processing units.

20. The processor of claim 19, wherein the processor is included in at least one of the following: control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; A system for performing simulation operations; Systems for performing digital twin operations; A system for performing light transport simulations; A system for performing collaborative content creation of 3D assets; Systems for performing generative AI operations; Systems for performing operations using large language models; Systems for performing deep learning operations; Systems implemented using edge devices; Systems implemented using robots; Systems for performing conversational AI operations; Systems for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

Images