Sensor extrinsic parameter correction for unified model deployment in autonomous systems and applications

Abstract

The invention relates to sensor extrinsic parameter correction for unified model deployment in autonomous systems and applications. In various examples, image data used to train a machine learning model (MLM) and / or training tags may be based on sensors having different viewing angles (e.g., field of view based on location and orientation). The image data and / or training tags may be transformed to generate transformed images and tag data, e.g., by translating, rotating, scaling, or tilting images corresponding to the image data to different fields of view of the real or virtual standard sensor. Thus, the MLM may be trained and / or deployed using transformed sensor data and training tags having the same or similar field of view. Thus, MLMs may be trained and / or deployed on any number of different vehicles equipped with cameras and / or other sensors having different viewing angles using transformed image data and / or tags having the same viewing angle as real or virtual specification sensors.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of Indian Provisional Patent Application No. 202441074176, filed on 1 October 2024, the entire contents of which are incorporated herein by reference. Background Technology

[0003] Designing a system capable of autonomously driving a vehicle with a practically acceptable level of safety in unsupervised conditions is extremely difficult. An autonomous vehicle should at least be able to perform as effectively as a focused driver, utilizing perception and action systems with the robust ability to identify and respond to moving and static obstacles in complex environments, thus avoiding collisions with other objects or structures in its path. Perception in autonomous vehicles can be achieved using scene computer vision and understanding algorithms that rely on applying images captured by the vehicle's cameras to a convolutional neural network (CNN). The accuracy of perception decreases when camera characteristics (such as camera position, orientation, field of view, or lens distortion) deviate from the values ​​of the camera used to train the neural network (NN). For example, if a neural network is trained using a camera mounted on the windshield of a low-chassis sports car, the system's accuracy may decrease, or even malfunction, if a camera mounted on a higher chassis vehicle (such as a sports utility vehicle (SUV) or truck). Furthermore, the images used to train the neural network should generally share similar camera characteristics, such as being captured by cameras of the same type and / or model.

[0004] Conventionally, this approach minimizes the variation in camera characteristics used to generate images for training and deployment by using a series of neural networks, each trained and deployed with consistent camera characteristics. For example, different neural networks could be trained and deployed using consistent camera characteristics for each vehicle year, brand, and / or model. However, this method requires a large amount of training data, which is both expensive and time-consuming to collect. Training also requires significant computational resources because each network for each vehicle / camera needs to be trained individually. Furthermore, each neural network requires separate maintenance and updates, consuming storage space and bandwidth.

[0005] Other conventional techniques can generate labeled training data for a variety of different vehicle types and transform the images or videos included in the training data to normalized camera positions and / or viewpoints without transforming the training labels associated with the training data. While transformed training data may be suitable for training a single machine learning model (MLM) to encompass multiple vehicle types, training an MLM using untransformed training labels can lead to a decrease in the performance of the trained machine learning model.

[0006] Therefore, more effective technologies are needed to improve unified model generation so that it can be deployed on different vehicle brands, models and types with different camera characteristics. Summary of the Invention

[0007] Embodiments of this disclosure relate to sensor extrinsic calibration for unified model generation in autonomous or semi-autonomous systems and applications. Systems and methods are disclosed for transforming image data, other sensor data (e.g., LiDAR, RADAR, ultrasound, etc.) and / or training labels to compensate for differences in sensor characteristics of the sensors used to capture the sensor data, and for compensating for these differences when generating training labels. The transformed sensor data can be applied to MLMs for training and / or inference, thereby improving perception.

[0008] Compared to conventional methods, the sensor data used to train an MLM (e.g., a deep neural network (DNN)) and / or the sensor data used by the DNN during deployment can be captured by sensors (e.g., cameras, LiDAR, etc.) with different viewpoints (e.g., field of view, sensor field, position (e.g., left, right, top, bottom), pose, and / or orientation relative to a specific vehicle, ground plane, or other reference point). In these examples, the sensor data can be transformed to generate transformed sensor data, for example by altering or eliminating lens distortion, shifting, rotating, cropping, and / or extracting at least one region of interest (ROI) from the image corresponding to this sensor data to the field of view of different physical or virtual sensors. Furthermore, the training labels included in the sensor data used to train the MLM can be transformed to account for differences between sensors with different viewpoints compared to different physical or virtual sensors. Therefore, MLMs can be trained and / or deployed using sensor data captured from the same or similar field of view and transformed training labels associated with the captured sensor data. Therefore, MLM can be trained and / or deployed on any number of different vehicles equipped with cameras and / or other sensors with different perspectives, using sensor data with the same viewpoint as a reference, pinhole, or ideal sensor, and training labels transformed to the viewpoints of other sensors. This process improves the scalability of the system while eliminating vehicle-specific dependencies, thereby generating machine learning models that can be deployed in any number of different vehicles. Attached Figure Description

[0009] The present disclosure’s system and method for sensor extrinsic parameter calibration for unified model generation in autonomous and semi-autonomous systems and applications will now be described in detail with reference to the accompanying drawings, wherein:

[0010] Figure 1 This is a block diagram of an example computing device applicable to implementing some embodiments of the present disclosure;

[0011] Figure 2 According to various embodiments Figure 1 A more detailed illustration of training engine 122 in the diagram;

[0012] Figure 3 A flowchart of a method for training a machine learning model according to various embodiments is shown;

[0013] Figure 4 According to various embodiments Figure 1 More detailed illustrations of the inference engine 124 in the diagram;

[0014] Figure 5 A flowchart of a method for reasoning and predicting vehicle paths according to various embodiments is shown;

[0015] Figure 6A These are illustrations of exemplary autonomous vehicles according to some embodiments of the present disclosure;

[0016] Figure 6B According to some embodiments of this disclosure Figure 6A An example of camera position and field of view for an exemplary autonomous vehicle;

[0017] Figure 6C According to some embodiments of this disclosure Figure 6A A block diagram of an exemplary system architecture for an exemplary autonomous vehicle;

[0018] Figure 6D It is one or more cloud-based servers according to some embodiments of this disclosure and Figure 6A A system diagram illustrating communication between exemplary autonomous vehicles;

[0019] Figure 7 This is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0020] Figure 8 This is a block diagram of an example data center applicable to implementing some embodiments of this disclosure. Detailed Implementation

[0021] Systems and methods related to sensor extrinsic parameter calibration for unified model generation in autonomous and semi-autonomous systems and applications are disclosed. Although this disclosure may relate to an example autonomous vehicle or semi-autonomous vehicle or machine 600 (in this document, alternatively referred to as "vehicle 600" or "self-vehicle 600"), examples are provided. Figures 6A-6DThe description herein is provided, but is not intended to be limiting. For example, the systems and methods described herein can be used with (but are not limited to) non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, ships, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles, drones, and / or other types of vehicles. Furthermore, although this disclosure describes path prediction for autonomous vehicles, it is not intended to be limiting, and the systems and methods described herein can be used in augmented reality, virtual reality, mixed reality, robotics, security and supervision, autonomous or semi-autonomous machine applications, and / or any other technological space where vehicle path prediction can be used. Additionally, although this disclosure primarily uses examples of sensors in the form of cameras for description, the disclosed techniques can be used to apply transformations (e.g., transformations of their sensing fields) to any suitable type of sensor (e.g., LiDAR, RADAR, ultrasound, etc.).

[0022] As discussed in this paper, conventional techniques can train and deploy a series of MLMs, each trained using consistent camera features and training labels specific to a particular vehicle type. These techniques require significant computational resources for training, as each network needs to be trained individually. Furthermore, each neural network requires separate maintenance and updates, consuming storage space and bandwidth. Other conventional techniques train a single MLM across various vehicle types by transforming training data captured from cameras with diverse features to simulate the viewpoint of a real or canonical camera. These techniques do not transform the training labels associated with the training data, leading to a degraded performance of the individual MLM.

[0023] To improve path prediction, the disclosed technique transforms the training data and the training labels associated with it based on the characteristic differences between one or more cameras used to capture the training data and a standard camera (e.g., a virtual camera). The transformed training data and training labels are used to train a machine learning model to generate predicted paths for the vehicle based on imagery received from cameras included in the vehicle. During inference, the disclosed technique transforms the imagery from the vehicle-included cameras, making the trained machine learning model suitable for deployment across various vehicle types.

[0024] The training engine receives training data, which includes images, identified regions of interest (ROIs) within the images, and labeled paths associated with the images, such as middle lane, right lane, left lane, left lane split, right lane split, merging from the left, and / or merging from the right. The labeled paths are defined by a set of two-dimensional (2D) coordinates in the received images. The training engine also receives external camera data describing the onboard camera associated with the images. The external camera data describes the camera's position and orientation, such as three-dimensional (3D) position coordinates referenced to its position inside the vehicle, as well as pitch, roll, and / or yaw.

[0025] The training engine transforms images, regions of interest (ROIs), and / or labeled paths based on the differences between external camera data and various external data associated with a specified canonical camera position and orientation. The canonical camera can be a real camera or a virtual camera. The transformed images, ROIs, and labeled paths simulate the appearance of the original images, ROIs, and / or labeled paths as seen from the canonical camera's viewpoint (e.g., position and orientation). The training engine trains the MLM based on the transformed training data and / or training labels.

[0026] During inference, the inference engine, including the trained MLM, generates predicted vehicle paths for vehicles including the camera based on the input image received from the camera and one or more ROIs extracted from the input image.

[0027] The inference engine receives camera data, which includes input images from the camera and external camera data associated with the camera. The external camera data describes the camera's position and orientation, such as three-dimensional (3D) position coordinates with reference to its position inside the vehicle, as well as pitch, roll, and / or yaw.

[0028] The inference engine transforms the input image based on the differences between external camera data and external data from a canonical camera. The inference engine can apply a transformation matrix to one or more pixels included in the input image to generate a transformed image. The transformed image comprises a simulated depiction of the scene included in the input image as observed from the perspective of a canonical real or virtual camera and / or sensor.

[0029] The inference engine extracts one or more regions of interest (ROIs) from the transformed image and sends these ROIs to the trained MLM. Based on these ROIs, the trained MLM predicts the vehicle's path. The predicted path may include the vehicle traveling along the middle lane, left lane, or right lane of the roadway, splitting into left or right lanes, or merging from the left or right lane.

[0030] Figure 1This is a block diagram of an example computing device applicable to implementing some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, commands, functional groups, etc.) may be used in addition to or as alternatives to the arrangements and elements shown, and certain elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and in any location. The various functions performed by the entities described herein can be implemented by hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with… Figures 6A-6D Example of autonomous vehicles 600 Figure 7 Example computing device 700 and / or Figure 8 The example data center 800 uses components, features, and / or functions similar to those of other components, features, and / or functions to perform the same task.

[0031] In one embodiment, computing device 100 includes a desktop computer, laptop computer, smartphone, personal digital assistant (PDA), tablet computer, or any other type of computing device configured to receive input, process data, and optionally display images, and is suitable for practicing one or more embodiments. Computing device 100 is configured to run a training engine 122 and an inference engine 124 residing in memory 116.

[0032] It should be noted that the computing devices described herein are illustrative, and any other technically feasible configuration falls within the scope of this disclosure. For example, multiple instances of training engine 122 and inference engine 124 may execute on a set of nodes in a distributed and / or cloud computing system to implement the functionality of computing device 100. In another example, training engine 122 and inference engine 124 may execute in various sets of hardware, device types, or environments to adapt training engine 122 or inference engine 124 to different use cases or applications. In a third example, training engine 122 and inference engine 124 may execute on different computing devices and / or different sets of computing devices.

[0033] In one embodiment, computing device 100 includes, but is not limited to, an interconnect (bus) 112 connecting one or more processors 102, an input / output (I / O) device interface 104 coupled to one or more input / output (I / O) devices 108, memory 116, storage device 114, and network interface 106. Processor 102 can be any suitable processor implemented as a central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), artificial intelligence (AI) accelerator, any other type of processing unit, or combination of different processing units, such as a CPU configured to work in conjunction with a GPU. Typically, processor 102 can be any technically feasible hardware unit capable of processing data and / or executing software applications. Furthermore, in the context of this disclosure, the computing element shown in computing device 100 can correspond to a physical computing system (e.g., a system in a data center) or can be a virtual computing instance executed in a computing cloud.

[0034] I / O device 108 includes devices capable of providing input, such as a keyboard, mouse, touchscreen, etc., and devices capable of providing output, such as a display device. Furthermore, I / O device 108 may also include devices capable of receiving input and providing output, such as a touchscreen, a Universal Serial Bus (USB) port, etc. I / O device 108 can be configured to receive various types of input from end users of computing device 100 (e.g., designers), and can also provide various types of output to end users of computing device 100, such as displayed digital images, digital video, or text. In some embodiments, one or more I / O devices 108 are configured to couple computing device 100 to network 110.

[0035] Network 110 is any technically feasible type of communication network that allows the exchange of data between computing device 100 and external entities or devices (such as a web server or another networked computing device). For example, network 110 may include a wide area network (WAN), a local area network (LAN), a wireless (WiFi) network, and / or the Internet.

[0036] Storage device 114 includes non-volatile storage for applications and data, and may include fixed or removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other magnetic, optical, or solid-state storage devices. Training engine 122 and inference engine 124 may be stored in storage device 114 and loaded into memory 116 during execution.

[0037] Memory 116 includes random access memory (RAM) modules, flash memory cells, or any other type of memory cell or combination thereof. Processor 102, I / O device interface 104, and network interface 106 are configured to read data from memory 116 and write data to memory 116. Memory 116 includes various software programs executable by processor 102 and application data associated with said software programs (including training engine 122 and inference engine 124).

[0038] Figure 2 According to various embodiments Figure 1 A more detailed illustration of the training engine 122 is provided. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, commands, function groups, etc.) may be used in addition to or as alternatives to the arrangements and elements shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and in any location. The various functions performed by these entities can be implemented by hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory.

[0039] The training engine 122 can receive a training dataset 210 and includes elements such as an ROI extractor 220, a label extractor 230, an ROI transformer 240, a label transformer 250, a machine learning model 260, and a loss calculator 270.

[0040] In summary, training engine 122 can be configured to generate, process, preprocess, enhance, and / or otherwise prepare input data for training the MLM, and to train the MLM. In embodiments including ROI extractor 220, ROI extractor 220 can be configured to extract one or more ROIs from sensor data and / or image data included in training dataset 210. In embodiments including ROI transformer 240, ROI transformer 240 can be configured to transform ROIs corresponding to the input data included in training dataset 210 to generate transformed ROIs corresponding to viewpoints of different physical or virtual sensors.

[0041] In embodiments including label extractor 230, label extractor 230 may be configured to extract one or more labels associated with sensor data and / or image data included in training dataset 210. In embodiments including label transformer 250, label transformer 250 may be configured to transform the labels extracted by label extractor 230 to generate transformed labels corresponding to viewpoints of different physical or virtual sensors.

[0042] Therefore, in embodiments using ROI transformer 240 and label transformer 250 in training engine 122, training engine 122 can train machine learning model 260 using input data (e.g., images and / or frames of sensor input) generated using multiple different sensor characteristics or parameters that can be normalized using ROI transformer 240 and label transformer 250. For example, ROI transformer 240 and label transformer 250 can normalize input data from one or more cameras and / or other sensors with different viewpoints included in training dataset 210 to generate transformed ROIs and training labels with the same viewpoint as a reference and / or ideal sensor, thereby training machine learning model 260 using at least the transformed ROIs and training labels.

[0043] In various examples, training dataset 210 may include sensor data generated using any number of sensors (physical and / or virtual or analog sensors), such as LiDAR sensor 664, RADAR sensor 660, ultrasonic sensor 662, microphone 696, and / or other types of sensors. Sensor data may represent the field of view and / or sensing field of the sensor (e.g., LiDAR sensor 664, RADAR sensor 660, etc.), and / or may represent the perception of the environment by one or more sensors (e.g., microphone 696). Sensors such as image sensors (e.g., image sensors of a camera), LiDAR sensors, RADAR sensors, SONAR sensors, ultrasonic sensors, etc., may be referred to herein as sensing sensors or sensing sensor devices, and sensor data generated by sensing sensors may be referred to herein as sensing sensor data. In some examples, instances or representations of sensor data may be represented by images captured by image sensors (e.g., image data), depth maps generated by LiDAR sensors, etc. LiDAR data, SONAR data, RADAR data, and / or other sensor data types may be related to or associated with image data generated using one or more image sensors. For example, image data representing one or more images can be updated to include data related to LiDAR sensors, SONAR sensors, RADAR sensors, etc., making the sensor data used to train machine learning model 260 and / or used as input to machine learning model 260 more informative or detailed than the image data alone. Therefore, machine learning model 260 can learn to generate predictions using this additional information from any number of sensing sensors.

[0044] In embodiments using sensor data, the sensor can be calibrated such that the sensor data is correlated with pixel coordinates in the image data. The ROI extractor 220 can perform preprocessing on the sensor data, similar to the preprocessing described herein with respect to image data. In some embodiments, such as where the sensor data indicates depth (e.g., RADAR data, LiDAR data, etc.), depth values ​​can be correlated with pixel coordinates in the image data and then used as additional (or, in some examples, alternative) input to the machine learning model 260. For example, one or more pixels can have an associated additional value representing the depth determined based on the sensor data.

[0045] As described herein, training dataset 210 may include other data types, such as training labels and / or external camera data. Training labels may include ground truth predicted vehicle paths associated with the images and / or sensor data included in training dataset 210. Predicted vehicle paths may include a set of two-dimensional (2D) locations included in the images. Training labels may also include annotations such as one of the following: middle lane, right lane, left lane, left lane split, right lane split, merging from the left, or merging from the right. External camera data includes position and orientation information associated with the cameras and / or sensors used to capture elements of the training data included in training dataset 210. In various embodiments, external camera data includes the 3D position of the camera / sensor relative to a fixed position on the vehicle, and angular information defining the pitch, yaw, and / or roll values ​​of the camera / sensor. By defining the position and orientation of the camera / sensor, the external camera data defines the viewpoint of the camera / sensor used to capture the training data.

[0046] In one or more embodiments, at least some of the data included in the training dataset 210 may be generated using a simulator, such as a simulator that renders or otherwise determines image and / or sensor data inputs from one or more virtual environments (e.g., real-world 3D representations and / or simulations). In one or more embodiments, the training dataset 210 may include all real input data, all simulated or synthetic input data, or some combination thereof.

[0047] In embodiments using ROI extractor 220, ROI extractor 220 can be configured to extract ROIs from image and / or sensor data included in training dataset 210. In some embodiments, the ROIs included in training dataset 210 may include boundaries defined by 2D locations included in the image and / or sensor data and a set of lines, where each line connects two 2D locations. For example, an ROI may include a lower boundary comprising a horizontal line of predetermined length located at a predetermined fixed distance in front of vehicle 600 in the image. An example ROI may also include an upper boundary comprising a horizontal line of predetermined length aligned with a portion of the image, such as the horizon separating the ground from the sky in the image. An example ROI may also include a line connecting the left end of the upper boundary to the left end of the lower boundary, and a line connecting the right end of the upper boundary to the right end of the lower boundary. Training engine 122 sends the extracted ROIs to ROI transformer 240.

[0048] In some embodiments, one or more portions of the ROI extractor 220 may implement an image preprocessing pipeline for processing the raw image acquired by a sensor (e.g., a camera) and included in image data to produce a preprocessed image. An example of a suitable image preprocessing pipeline could be a raw RCCB Bayer (e.g., single-channel) type image from a sensor, converted to an RCB (e.g., three-channel) planar image stored in a fixed-precision (e.g., 16 bits per channel) format. This image preprocessing pipeline may include decompression, noise reduction, demosaicing, white balance, histogram calculation, and / or adaptive global tone mapping (e.g., in this order or an alternative order).

[0049] When ROI extractor 220 employs denoising, it can include bilateral denoising in the Bayer domain. When ROI extractor 220 employs demosaicing, it can include bilinear interpolation. When ROI extractor 220 employs histogram calculation, it can include calculating the histogram of the C channels, and in some examples, this can be combined with decompression or denoising. When ROI extractor 220 employs adaptive global tone mapping, it can include performing an adaptive gamma-log transformation. This can include calculating the histogram, obtaining midtone levels, and / or estimating maximum luminance using midtone levels.

[0050] In embodiments using label extractor 230, label extractor 230 may be configured to extract training labels associated with the images and / or sensor data included in training dataset 210. As discussed herein, training labels may include ground truth predicted vehicle paths associated with the images and / or sensor data included in training dataset 210. The predicted vehicle paths may include a set of two-dimensional (2D) locations included in the images. Training labels may include annotations, such as one of the following: middle lane, right lane, left lane, left lane split, right lane split, merging from the left, or merging from the right. Training engine 122 sends the extracted training labels to label transformer 250.

[0051] In embodiments using ROI transformer 240, ROI transformer 240 may be configured to transform extracted ROIs received from ROI extractor 220 such that the transformed ROI simulates the extracted ROI as observed from different viewpoints of a real or virtual canonical camera. In some embodiments, ROI transformer 240 may apply a transformation matrix (e.g., a homography matrix) to the 2D locations included in the extracted ROI. ROI transformer 240 may rotate, scale, translate, and / or tilt the extracted ROI to generate the transformed ROI based on the difference between external data from the camera / sensor used to capture the ROI and external camera data from a real or virtual canonical camera. In other embodiments, ROI transformer 240 may generate a lookup table (LUT) that includes a correspondence between 2D locations in the extracted ROI and 2D locations in the transformed ROI. This correspondence may be based on the difference between external data from the camera / sensor used to capture the ROI and external camera data from a real or virtual canonical camera. Using this LUT, the ROI transformer can determine the corresponding 2D position in the transformed ROI for each 2D position included in the extracted ROI. The training engine 122 sends the transformed ROI to the machine learning model 260.

[0052] In embodiments using label transformer 250, label transformer 250 can be configured to transform the predicted vehicle path included in the extracted labels received from label extractor 230 into a transformed predicted vehicle path. In some embodiments, label transformer 250 does not modify annotations included in the extracted labels, such as middle lane, right lane, left lane, left lane split, right lane split, merging from the left and / or merging from the right. Similar to ROI transformer 240 discussed herein, in some embodiments, label transformer 250 can apply a transformation matrix (e.g., homography matrix) to the 2D ground truth vehicle path location included in the extracted labels to rotate, scale, translate, and / or tilt the ground truth vehicle path. The transformed labels include the transformed ground truth vehicle path and the same annotations as the extracted labels. Training engine 122 sends the transformed labels to loss calculator 270.

[0053] Machine learning model 260 can use one or more images or other data representations or instances (e.g., LiDAR data, RADAR data, SONAR data, ultrasonic data, etc.) represented by transformed ROIs and / or transformed labels as input to generate output. In a non-limiting example, machine learning model 260 can use an image represented by the input data as input (e.g., after processing with ROI extractor 220 and ROI transformor 240 to predict vehicle trajectory data).

[0054] While this paper describes examples using neural networks (particularly convolutional neural networks) as MLM 260, this is not intended to be limiting. For example, but not limited to, the MLM 260 described herein can include one or more machine learning models of any type, such as those using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutions, recurrent neural networks, perceptrons, long / short-term memory (LSTM), Hopfield, Boltzmann, deep belief, deconvolution, generative adversarial, liquid machines, etc.), and / or other types of machine learning models.

[0055] Training engine 122 can iteratively train machine learning model 260 on training data extracted and transformed as discussed herein, included in training dataset 210. During each training iteration, training engine 122 sends a transformed ROI to machine learning model 260. Based on the transformed ROI, machine learning model 260 generates a predicted vehicle path and sends the generated predicted vehicle path to loss calculator 270. Loss calculator 270 calculates a training loss based on the generated predicted vehicle path and transformed labels received from label transformer 250. In some embodiments, training engine 122 can be configured to calculate the training loss based on the difference between the corresponding 2D location included in the generated predicted vehicle path and the transformed label (e.g., calculated via mean squared error (MSE)). In some embodiments, training engine 122 can continue to iteratively train the machine learning model until the calculated training loss is below a predetermined threshold. The trained machine learning model 260 can be used by inference engine 124 discussed herein.

[0056] The inference engine 124 is described by way of example, not limitation, with respect to an MLM trained for computer vision and / or perception operations to navigate a vehicle. However, aspects of this disclosure can be more broadly applied to any form of MLM trained and / or deployed to make predictions based on sensor data. In some examples, the MLM 260 can be trained to predict trajectory points, vehicle orientation (e.g., relative to environmental features such as lane markings), and / or vehicle states (e.g., regarding object manipulation, such as lane changing, turning, merging, etc.), which can be used to control an autonomous vehicle. However, this is not intended to be limiting.

[0057] Furthermore, inference engine 124 is one example of an engine that can be used in at least one embodiment, such as for training an MLM for computer vision and / or perception operations to navigate a vehicle or for other purposes. However, inference engine 124 can vary to include more than Figure 2 Showing more or fewer components and / or processing paths and / or with Figure 2 The different components and / or processing paths are shown.

[0058] Now for reference Figure 3 and Figure 5 Each operation of methods 300 and 500 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. These methods can also be embodied as computer-usable instructions stored on a computer storage medium. These methods can be provided by standalone applications, services, or managed services (independently or in combination with other managed services), or plug-ins to another product, etc. Furthermore, this document provides examples regarding... Figures 1-2and Figure 4 The systems described herein are methods 300 and 500. However, these methods may be additionally or alternatively performed by any single system or any combination of systems, including but not limited to the systems described herein.

[0059] Figure 3 Flowcharts of methods for training machine learning models according to various embodiments are shown. Figure 3 As shown, method 300 begins at operation 302, where training engine 122 receives training data from training dataset 210, including image and / or sensor data, training labels, and external camera data. In various embodiments, the image and / or sensor data may include regions of interest (ROIs) included in the image and / or sensor data. Training labels may include a set of 2D locations included in the image and / or sensor data that define a ground truth vehicle path associated with the image and / or sensor data. External camera data may include position and orientation data associated with the camera and / or sensor used to capture the training data, such that the position and orientation data define the viewpoint of the camera and / or sensor.

[0060] Method 300 includes at operation 304: extracting regions of interest (ROIs) and training labels from training data. In some embodiments, ROI extractor 220 extracts ROIs from training data based on one or more boundaries included in the image and / or sensor training data. In some embodiments, label extractor 230 extracts training labels from the training data. Training labels include ground truth vehicle paths associated with the image and / or sensor data, and annotations describing the vehicle path, such as middle lane, right lane, left lane, left lane split, right lane split, merging from the left, and / or merging from the right.

[0061] Method 300 includes, at operation 306, transforming the extracted ROI to generate a transformed ROI. In some embodiments, the ROI transformer 240 transforms the extracted ROI by applying a homography matrix to one or more 2D locations included in the ROI. The homography matrix is ​​based on the difference between external camera data associated with training data and external camera data associated with a real or virtual canonical camera. In other embodiments, the ROI transformer 240 may transform the extracted ROI using a lookup table that includes a correspondence between 2D locations included in the extracted ROI and 2D locations included in the transformed ROI. Values ​​in the lookup table may be based on the difference between external camera data associated with training data and external camera data associated with a real or virtual canonical camera.

[0062] Method 300 includes, at operation 308, transforming the extracted training labels to generate transformed labels. In some embodiments, label transformer 250 transforms the ground truth vehicle paths included in the extracted training labels by applying a homography matrix to one or more 2D locations included in the ground truth vehicle paths associated with the training data. In other embodiments, label transformer 250 may transform the ground truth vehicle paths using a lookup table that includes a correspondence between 2D locations included in the extracted ROI and 2D locations included in the transformed ROI. The values ​​included in the lookup table may be based on the difference between external camera data associated with the training data and external camera data associated with a real or virtual canonical camera.

[0063] Method 300 includes, at operation 310, training a machine learning model based on a transformed ROI and transformed labels. In some embodiments, training engine 122 iteratively trains machine learning model 260. During each training iteration, training engine 122 sends a transformed ROI to machine learning model 260. Based on the transformed ROI, machine learning model 260 generates a predicted vehicle path and sends the generated predicted vehicle path to loss calculator 270. Loss calculator 270 calculates a training loss based on the generated predicted vehicle path and transformed labels received from label transformer 250. In some embodiments, training engine 122 may be configured to calculate the training loss based on the difference between the corresponding 2D location included in the generated predicted vehicle path and the transformed label, for example, via mean squared error (MSE). In some embodiments, training engine 122 may continue to iteratively train the machine learning model until the calculated training loss is below a predetermined threshold.

[0064] Figure 4 According to various embodiments Figure 1 A more detailed illustration of the inference engine 124 is provided. It should be understood that the arrangements and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groups, etc.) may be used in addition to or as alternatives to the arrangements and elements shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and at any location. The various functions performed by these entities can be performed by hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory.

[0065] The inference engine 124 can receive camera data 400 and generate a predicted path 430. The inference engine 124 may include elements such as a camera data transformer 410, a ROI extractor 420, and a machine learning model 260.

[0066] In summary, the inference engine 124 can be configured to generate a predicted path for the vehicle based on camera data received from the vehicle. In embodiments including a camera data transformer 410, the camera data transformer 410 can be configured to generate sensor data and / or image data included in the camera data 400 based on external camera values ​​included in the camera data 400 and external camera values ​​of a real or virtual canonical camera. In embodiments including a ROI extractor 420, the ROI extractor 420 can be configured to identify regions of interest (ROIs) corresponding to the input data included in the camera data 400. A trained machine learning model 260 generates a predicted path 430 for the vehicle based on the identified ROIs associated with the camera data 400.

[0067] Camera data 400 includes images and / or sensor data captured by, for example, cameras / sensors included in vehicle 600. Camera data 400 also includes external camera data associated with the cameras / sensors. The external camera data includes the 3D position of the camera / sensor relative to a fixed point included in vehicle 600 and one or more orientation information (including pitch angle, roll angle, and / or yaw angle) associated with the camera / sensor. The external camera data defines, for example, the viewpoint of the camera included in vehicle 600.

[0068] In embodiments including a camera data transformer 410, the camera data transformer 410 may be configured to transform image / sensor data included in camera data 400 to generate transformed camera data based on the differences between external camera data included in camera data 400 and external camera data associated with a real or virtual specification camera. The camera data transformer 410 transforms the image / sensor data included in camera data 400 such that the transformed camera data depicts camera / sensor data as viewed from the viewpoint of a real or virtual specification camera. In various embodiments, the camera data transformer 410 may apply a homography matrix to one or more 2D locations included in camera data 400 to calculate the correspondence between one or more 2D locations included in camera data 400 and one or more 2D locations included in the transformed camera data. The values ​​included in the homography matrix may be based on the differences between external camera data included in camera data 400 and external camera data associated with a real or virtual specification camera. In other embodiments, the camera data transformer 410 may refer to a lookup table (LUT) that includes a correspondence between each 2D location in the image / sensor data included in the camera data 400 and each 2D location included in the transformed camera data.

[0069] In embodiments including ROI extractor 420, ROI extractor 420 can be configured to generate a region of interest (ROI) based on transformed camera data. The generated ROI may include 2D locations included in the transformed camera data and a boundary defined by a set of lines, each line connecting two 2D locations. For example, the ROI may include a lower boundary comprising a horizontal line of predetermined length located at a predetermined fixed distance in front of vehicle 600 in the image. An example ROI may also include an upper boundary comprising a horizontal line of predetermined length aligned with a portion of the image, such as the horizon separating the ground from the sky in the image. An example ROI may also include a line connecting the left end of the upper boundary and the left end of the lower boundary, and a line connecting the right end of the upper boundary and the right end of the lower boundary. Inference engine 124 sends the generated ROI to machine learning model 260.

[0070] As discussed herein, machine learning model 260 has been previously trained by training engine 122 to generate predicted vehicle paths based on input ROIs included in transformed camera data. Machine learning model 260 processes the ROIs received from ROI extractor 420 and generates a predicted path 430. The predicted path 430 includes a set of 2D locations included in the transformed camera data and indicates the predicted movement path of vehicle 600 as observed from the perspective of a real or virtual canonical camera.

[0071] Figure 5A flowchart illustrating a method for reasoning and predicting vehicle paths according to various embodiments is shown. Figure 5 As shown, method 500 begins at operation 502, where inference engine 124 receives camera data from, for example, vehicle 600, which includes image and / or sensor data as well as external camera data. The external camera data may include position and orientation data associated with the camera and / or sensor used to capture camera data 400, such that the position and orientation data defines the viewpoint of the camera and / or sensor.

[0072] Method 500 includes, at operation 504, transforming camera data 400 to generate transformed camera data. In various embodiments, camera data transformer 410 may apply a homography matrix to each 2D location included in camera data 400, or may refer to a lookup table (LUT) that includes a correspondence between each 2D location in the image / sensor data included in camera data 400 and each 2D location included in the transformed camera data. The values ​​included in the homography matrix and / or LUT may be based on the difference between external camera data included in camera data 400 and external camera data of a real or virtual canonical camera. The transformed camera data depicts the camera and / or sensor data included in camera data 400 as viewed from the viewpoint of a real or virtual canonical camera.

[0073] Method 500 includes, at operation 506, extracting a region of interest (ROI) included in the transformed camera data. ROI extractor 420 generates the ROI based on the transformed camera data. The generated ROI may include 2D locations included in the transformed camera data and a boundary defined by a set of lines, each line connecting two 2D locations. For example, the ROI may include a lower boundary comprising a horizontal line of predetermined length located at a predetermined fixed distance in front of vehicle 600 in the image. An example ROI may also include an upper boundary comprising a horizontal line of predetermined length aligned with a portion of the image, such as the horizon separating the ground from the sky in the image. An example ROI may also include a line connecting the left end of the upper boundary and the left end of the lower boundary, and a line connecting the right end of the upper boundary and the right end of the lower boundary.

[0074] Method 500 includes at operation 508: generating a predicted path for the vehicle based on ROIs generated by ROI extractor 420 via a trained machine learning model. Machine learning model 260 (which has been previously trained by training engine 122 (as discussed herein)) generates a predicted path 430 for the vehicle (e.g., vehicle 600) based on ROIs generated from transformed camera data. The predicted path 430 includes a set of 2D locations included in the transformed camera data and indicates the predicted movement path of the vehicle as observed from the viewpoint of a real or virtual canonical camera.

[0075] The systems and methods described herein can be used, but are not limited to, non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, ships, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles, drones, and / or other vehicle types. Furthermore, the systems and methods described herein can be used for a variety of purposes, including, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, safety and supervision, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, and / or any other suitable application.

[0076] The disclosed embodiments may be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, marine systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems implementing one or more language models (e.g., one or more large language models (LLMs), one or more visual language models (VLMs), and / or one or more multimodal language models), systems for performing optical transmission simulations, systems for performing collaborative content creation of 3D assets, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0077] Example autonomous vehicles

[0078] Figure 6AThis is an illustration of an exemplary autonomous or semi-autonomous vehicle 600 according to some embodiments of this disclosure. The autonomous vehicle 600 (or referred to herein as “vehicle 600”) may include, but is not limited to, passenger vehicles such as cars, trucks, buses, first-response vehicles, shuttles, electric or motorized bicycles, motorcycles, fire trucks, police cars, ambulances, boats, construction vehicles, underwater vehicles, robotic vehicles, drones, aircraft, vehicles coupled to trailers (e.g., semi-tractor-trailer trucks for hauling goods) and / or other types of vehicles (e.g., driverless and / or vehicles accommodating one or more passengers). Autonomous vehicles are typically described according to their level of automation, as defined by the National Highway Traffic Safety Administration (NHTSA) of the U.S. Department of Transportation, and by the Society of Automotive Engineers (SAE) in its "Classification and Definition of Terms Related to Driving Automation Systems for Road Motor Vehicles" (Standard No.: J3016-201806, published June 15, 2018; Standard No.: J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 600 may implement functions according to one or more of Level 3 to Level 5 of autonomous driving. Vehicle 600 may be able to have one or more functionalities that conform to Level 1 to Level 5 of autonomous driving. For example, according to an embodiment, vehicle 600 may have driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). As used herein, the term “autonomy” can include any and / or all types of autonomy of the vehicle 600 or other machines, such as full autonomy, high autonomy, conditional autonomy, partial autonomy, provision of auxiliary autonomy, semi-autonomy, primary autonomy or other specified autonomy.

[0079] Vehicle 600 may include a chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. Vehicle 600 may include a propulsion system 650, such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or other propulsion system types. Propulsion system 650 may be connected to the drivetrain of vehicle 600, which may include a transmission, to enable propulsion of vehicle 600. Propulsion system 650 may be controlled in response to receiving signals from throttle valve / accelerator 652.

[0080] When the propulsion system 650 is in operation (e.g., when the vehicle is moving), the steering system 654, including a steering wheel, can be used to guide the vehicle 600 (e.g., along a desired path or route). The steering system 654 can receive signals from the steering actuator 656. For fully automatic (level 5) functionality, the steering wheel may be optional.

[0081] The brake sensor system 646 can be used to operate the vehicle brakes in response to signals received from the brake actuator 648 and / or the brake sensor.

[0082] One or more controllers 636 may include one or more system-on-chip (SoC) 604 ( Figure 6C One or more controllers and / or one or more GPUs may provide signals (e.g., representing commands) to one or more components and / or systems of vehicle 600. For example, one or more controllers may send signals via one or more brake actuators 648 to operate vehicle brakes, via one or more steering actuators 656 to operate steering system 654, and via one or more throttles / accelerators 652 to operate propulsion system 650. One or more controllers 636 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving vehicle 600. One or more controllers 636 may include a first controller 636 for autonomous driving functions, a second controller 636 for functional safety functions, a third controller 636 for artificial intelligence functions (e.g., computer vision), a fourth controller 636 for infotainment functions, a fifth controller 636 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 636 may handle two or more of the functions described above, and two or more controllers 636 may handle a single function and / or any combination thereof.

[0083] One or more controllers 636 may provide signals for controlling one or more components and / or systems of vehicle 600 in response to sensor data (e.g., sensor input) received from one or more sensors. Sensor data may be received from, for example, but not limited to, one or more Global Navigation Satellite System (“GNSS”) sensors 658 (e.g., one or more Global Positioning System sensors), one or more RADAR sensors 660, one or more ultrasonic sensors 662, one or more LiDAR sensors 664, one or more Inertial Measurement Unit (IMU) sensors 666 (e.g., one or more accelerometers, one or more gyroscopes, one or more magnetometers, magnetometers, etc.), one or more microphones 696, one or more stereo cameras 668, one or more wide-angle cameras 670 (e.g., fisheye cameras), one or more infrared cameras 672, one or more surround cameras 674 (e.g., 360-degree cameras), one or more long-range and / or mid-frequency cameras 698, one or more speed sensors 644 (e.g., for measuring the speed of vehicle 600), one or more vibration sensors 642, one or more steering sensors 640, one or more braking sensors (e.g., as part of braking sensor system 646) and / or other sensor types.

[0084] One or more of the controllers 636 may receive inputs (e.g., represented by input data) from the instrument cluster 632 of the vehicle 600 and provide outputs (e.g., represented by output data, displayed data, etc.) via a human-machine interface (HMI) display 634, an audio signaler, a speaker, etc., and / or via other components of the vehicle 600. Outputs may include information such as vehicle speed, rate, time, map data (e.g., ...). Figure 6C Information such as a high-definition (“HD”) map 622, location data (e.g., the location of vehicle 600, such as its location on a map), direction, the location of other vehicles (e.g., grid occupancy), and information about objects and their states perceived by one or more controllers 636. For example, the HMI display 634 may display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.), and / or information about driving actions that the vehicle has performed, is performing, or will perform (e.g., changing lanes now, exiting from exit 34B in two miles, etc.).

[0085] Vehicle 600 also includes a network interface 624, which can communicate over one or more networks using one or more wireless antennas 626 and / or a modem. For example, network interface 624 can be used to communicate via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multicarrier (“CDMA2000”), etc. One or more wireless antennas 626 can also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using one or more local area networks (e.g., Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc.) and / or Low Power Wide Area Networks (LPWANs, such as LoRaWAN, SigFox, etc.).

[0086] Figure 6B According to some embodiments of this disclosure Figure 6A An example of the camera position and field of view of an exemplary autonomous vehicle 600. The camera and its respective field of view are an example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included and / or the cameras may be located at different positions on the vehicle 600.

[0087] The camera type may include, but is not limited to, a digital camera, which may be suitable for components and / or systems of vehicle 600. One or more cameras may operate at Automotive Safety Integrity Level (ASIL) B and / or other ASILs. According to embodiments, the camera type may have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc. The camera may use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red transparent (RCCC) color filter array, a red transparent blue (RCCB) color filter array, a red blue green transparent (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or other types of color filter arrays. In some embodiments, a sharp-pixel camera, such as a camera with RCCC, RCCB, and / or RBGC color filter arrays, may be used to improve light sensitivity.

[0088] In some examples, one or more cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundancy or fail-safe design). For example, a multi-functional single camera can be installed to provide functions such as lane departure warning, traffic sign assistance, and intelligent headlight control. One or more cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0089] One or more cameras may be mounted in mounting components, such as custom-designed (3D-printed) components, to cut off stray light and interior reflections (e.g., dashboard reflections from the windshield rearview mirror) that could interfere with the camera's ability to capture image data. Regarding wing mirror mounting components, these components may be custom-3D printed so that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras may be integrated into the wing-shaped rearview mirror. For side-view cameras, one or more cameras may also be integrated into the four pillars at each corner of the cab.

[0090] A camera with a field of view including a portion of the environment ahead of the vehicle (e.g., a front-facing camera) can be used for surround view to help identify forward paths and obstacles, and, with the assistance of one or more controllers 636 and / or control SOCs, to assist in providing information crucial for generating an occupancy grid and / or determining the preferred vehicle path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The forward-facing camera can also be used in ADAS functions and systems, including lane departure warning (“LDW”), adaptive cruise control (“ACC”), and / or other functions such as traffic sign recognition.

[0091] Various cameras can be used in front-facing configurations, including, for example, monocular camera platforms that include a complementary metal-oxide-semiconductor (“CMOS”) color imager. Another example could be a wide-angle camera 670 that can be used to perceive objects entering the field of view from the periphery (e.g., pedestrians, cross traffic, or bicycles). Although Figure 6B Only one wide-angle camera is shown, but the vehicle 600 may have any number (including zero) of wide-angle cameras 670. Furthermore, any number of remote cameras 698 (e.g., a pair of long-angle stereo cameras) can be used for depth-based object detection, particularly for objects for which neural networks have not yet been trained. One or more remote cameras 698 can also be used for object detection and classification, as well as basic object tracking.

[0092] Any number of stereo cameras 668 may also be included in the front-mounted configuration. In at least one embodiment, one or more stereo cameras 668 may include an integrated control unit comprising a scalable processing unit that may provide programmable logic (“FPGA”) and a multi-core microprocessor with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. This unit can be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. One or more alternative stereo cameras 668 may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip that measures the distance from the vehicle to a target object and uses the generated information (e.g., metadata) to activate automatic emergency braking and lane departure warning functions. In addition to the stereo cameras described herein, or alternatively, other types of stereo cameras 668 may be used.

[0093] A camera with a field of view including a portion of the vehicle's side environment (e.g., a side-view camera) can be used in the surround view to provide information for creating and updating the occupancy mesh and generating side collision warnings. For example, one or more surround cameras 674 (e.g., such as...) Figure 6B The four surround cameras 674 shown may be positioned on the vehicle 600. One or more surround cameras 674 may include one or more wide-angle cameras 670, one or more fisheye cameras, one or more 360-degree cameras, etc. For example, four fisheye cameras may be located at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle may use three surround cameras 674 (e.g., left, right, and rear) and may utilize one or more other cameras (e.g., a front-facing camera) as a fourth surround-view camera.

[0094] Cameras with a view that includes a portion of the environment behind the vehicle 600 (e.g., rear-view cameras) can be used for parking assistance, surround view, rear-end collision warning, and creating and updating occupancy grids. A variety of cameras can be used, including but not limited to cameras that are also suitable as front-facing cameras (e.g., one or more long-range and / or mid-range cameras 698, one or more stereo cameras 668, one or more infrared cameras 672, etc.), as described herein.

[0095] Figure 6C According to some embodiments of this disclosure Figure 6AA block diagram of an exemplary system architecture for an exemplary autonomous vehicle 600 is provided. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groups, etc.) may be used in addition to the arrangements and elements shown, or other arrangements and elements may be used instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory.

[0096] Figure 6C Every component, feature, and system of vehicle 600 is connected via bus 602. Bus 602 may include a Controller Area Network (CAN) data interface (also referred to herein as the "CAN bus"). CAN can be a network within vehicle 600 used to help control various features and functions of vehicle 600, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, etc. The CAN bus can be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). The CAN bus can be read to locate steering wheel angle, ground speed, engine speed per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may conform to the ASIL B standard.

[0097] Although bus 602 is described herein as a CAN bus, this is not intended to be limiting. For example, in addition to or from a CAN bus, FlexRay and / or Ethernet may also be used. Furthermore, although bus 602 is represented by a single line, this is not intended to be limiting. For example, any number of buses 602 may exist, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 602 may be used to perform different functions and / or for redundancy. For example, a first bus 602 may be used for a collision avoidance function, and a second bus 602 may be used for drive control. In any example, each bus 602 may communicate with any component of vehicle 600, and two or more buses 602 may communicate with the same component. In some examples, each SoC 604, each controller 636, and / or each computer within the vehicle may access the same input data (e.g., input from sensors of vehicle 600) and may be connected to a common bus, such as a CAN bus.

[0098] Vehicle 600 may include one or more controllers 636, as described herein. Figure 6A The controller 636 is described above. Controller 636 can be used for various functions. One or more controllers 636 can be coupled to any of the various other components and systems of vehicle 600, and can be used to control vehicle 600, artificial intelligence of vehicle 600, infotainment of vehicle 600, etc.

[0099] Vehicle 600 may include one or more System-on-Chip (SoC) 604. SoC 604 may include one or more CPUs 606, one or more GPUs 608, one or more processors 610, one or more caches 612, one or more accelerators 614, one or more data storage 616, and / or other components and features not shown. One or more SoCs 604 can be used to control vehicle 600 in various platforms and systems. For example, one or more SoCs 604 may be combined with an HD map 622 in a system (e.g., the system of vehicle 600), the HD map 622 being accessible via a network interface 624 from one or more servers (e.g., [server name missing]). Figure 6D Server 678) receives map refresh and / or updates.

[0100] One or more CPUs 606 may include CPU clusters or CPU complexes (or referred to herein as “CCPLEX”). One or more CPUs 606 may include multiple cores and / or a L2 cache. For example, in some embodiments, one or more CPUs 606 may include eight cores in a coherent multiprocessor configuration. In some embodiments, one or more CPUs 606 may include four dual-core clusters, each with a dedicated L2 cache (e.g., 2MB L2 cache). One or more CPUs 606 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, such that any combination of clusters of CPUs 606 is active at any given time.

[0101] One or more CPU 606s can implement power management capabilities including one or more of the following features: automatic clock gating of a single hardware block when idle to conserve dynamic power; clock gating of each core when the core is not actively executing instructions due to executing WFI / WFE instructions; independent power gating of each core; independent clock gating of each core cluster when all cores are clock-gated or power-gated; and / or independent power gating of each core cluster when all cores are power-gated. One or more CPU 606s can further implement enhanced algorithms for managing power states, specifying allowed power states and expected wake-up times, and the hardware / microcode determines the optimal power state for the core, cluster, and CCPLEX to enter. The processing core can support simplified power state input sequences in software and offload the work to the microcode.

[0102] One or more GPUs 608 may include integrated GPUs (or referred to herein as “iGPUs”). GPUs 608 may be programmable and efficient for parallel workloads. In some examples, one or more GPUs 608 may use an enhanced tensor instruction set. One or more GPUs 608 may include one or more streaming microprocessors, wherein each streaming microprocessor may include a Level 1 cache (e.g., a Level 1 cache with at least 96KB of storage), and two or more streaming microprocessors may share a Level 2 cache (e.g., a Level 2 cache with 512KB of storage). In some embodiments, one or more GPUs 608 may include at least eight streaming microprocessors. One or more GPUs 608 may use one or more computation application programming interfaces (APIs). Furthermore, one or more GPUs 608 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA’s CUDA).

[0103] One or more GPU 608s can be power-optimized for optimal performance in automotive and embedded use cases. For example, one or more GPU 608s can be fabricated on FinFETs. However, this is not intended to limit, and other semiconductor manufacturing processes can be used to fabricate one or more GPU 608s. Each streaming microprocessor can combine multiple mixed-precision processing cores partitioned into multiple blocks. For example, but not limited to, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block could be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA tensor cores for deep learning matrix algorithms, an L0 instruction cache, a thread bundle scheduler, a dispatch unit, and / or a 64KB register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to provide efficient execution of workloads through mixed computation and addressing computation. The streaming microprocessor can include independent thread scheduling capabilities to enable finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors can include a combination of a level-one data cache and a shared memory unit to improve performance while simplifying programming.

[0104] One or more GPUs 608 may include a high-bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem to provide a peak memory bandwidth of approximately 900GB / s in some examples. In some examples, in addition to HBM memory, or optionally from HBM memory, synchronous graphics random access memory (SGRAM), such as fifth-generation graphics dual data rate synchronous random access memory (GDDR5), may be used.

[0105] The fifth-generation GPU 608 may include unified memory technology, which includes access counters to allow more accurate migration of memory pages to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, address translation service (ATS) support may be used to allow one or more GPUs 608 to directly access the page tables of one or more CPUs 606. In such examples, when one or more GPUs 608 memory management units (MMUs) experience a miss, an address translation request may be sent to one or more CPUs 606. In response, one or more CPUs 606 may look up the virtual-to-physical mapping of the address in their page tables and send the translation back to one or more GPUs 608. Therefore, unified memory technology allows for a single unified virtual address space for the memory of both one or more CPUs 606 and one or more GPUs 608, thereby simplifying the programming of one or more GPUs 608 and porting applications to one or more GPUs 608.

[0106] In addition, one or more GPUs 608 may include access counters that track the frequency with which one or more GPUs 608 access the memory of other processors. Access counters help ensure that memory pages are moved to the physical memory of the processor that accesses those pages most frequently.

[0107] One or more SoCs 604 may include any number of caches 612, including the caches 612 described herein. For example, one or more caches 612 may include an L3 cache available to one or more CPUs 606 and one or more GPUs 608 (e.g., connecting both one or more CPUs 606 and one or more GPUs 608). Caches 612 may include write-back caches with traceable thread state, for example, by using cache coherence protocols (e.g., MEI, MESI, MSI, etc.). Although a small cache size may be used, according to embodiments, the L3 cache may include 4 MB or more.

[0108] One or more SoCs 604 may include one or more arithmetic logic units (ALUs) that can be used to perform processing related to various tasks or operations of the vehicle 600, such as processing a DNN. Furthermore, one or more SoCs 604 may include one or more floating-point units (FPUs) or other mathematical coprocessors or digital coprocessor types for performing mathematical operations within the system. For example, one or more SoCs 604 may include one or more FPUs integrated as execution units within a CPU 606 and / or a GPU 608.

[0109] One or more SoCs 604 may include one or more accelerators 614 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, one or more SoCs 604 may include a hardware acceleration cluster that may include optimized hardware accelerators and / or large on-chip memory. Large on-chip memory (e.g., 4MB of SRAM) can enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster can be used to supplement one or more GPUs 608 and offload some tasks from one or more GPUs 608 (e.g., freeing up more cycles from one or more GPUs 608 to perform other tasks). For example, one or more accelerators 614 may be sufficiently stable to be suitable for accelerating target workloads (e.g., perception, convolutional neural networks (CNNs), etc.). The term "CNN" as used herein may include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0110] One or more accelerators 614 (e.g., hardware acceleration clusters) may include one or more deep learning accelerators (DLAs). One or more DLAs may include one or more tensor processing units (TPUs) configured to provide an additional trillion operations per second for deep learning applications and inference. TPUs may be accelerators configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and optimized for them. One or more DLAs may also be optimized for specific neural network types and floating-point operations and inference. One or more DLAs are designed to provide higher performance per millimeter than general-purpose GPUs and significantly outperform CPUs. One or more TPUs may perform multiple functions, including single-instance convolution functions, such as supporting INT8, INT16, and FP16 data types for features and weights, and post-processor functions.

[0111] One or more DLAs can execute neural networks, especially CNNs, quickly and efficiently on processed or unprocessed data for any of a variety of functions, including but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and recognition using microphone data; CNNs for facial recognition and vehicle owner recognition using data from camera sensors; and / or CNNs for safety and / or safety-related events.

[0112] One or more DLAs can perform any function of one or more GPUs 608. For example, by using inference accelerators, designers can perform any function for one or more DLAs or one or more GPUs 608. For example, designers can centralize the processing of CNNs and floating-point operations on one or more DLAs and leave other functions to one or more GPUs 608 and / or one or more other accelerators 614.

[0113] One or more accelerators 614 (e.g., hardware acceleration clusters) may include programmable vision accelerators (PVAs), which may also be referred to herein as computer vision accelerators. One or more PVAs may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. One or more PVAs may provide a balance between performance and flexibility. For example, each PVA may include, for example, but not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0114] The RISC core can interact with an image sensor (e.g., the image sensor of any camera described herein), one or more image signal processors, etc. Each RISC core may include any amount of memory. The RISC core can use any of a variety of protocols, depending on the implementation. In some examples, the RISC core can run a real-time operating system (RTOS). The RISC core can be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, the RISC core may include an instruction cache and / or tightly coupled RAM.

[0115] DMA enables PVA components to access system memory independently of one or more CPUs. DMA can support any features used to optimize PVA, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which may include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0116] Vector processors can be programmable processors designed to efficiently and flexibly execute computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., a VMEM). The VPU core may include a digital signal processor, such as a single-instruction, multiple-data (SIMD) or very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can improve throughput and speed.

[0117] Each vector processor may include an instruction cache and may be coupled to dedicated memory. As a result, in some examples, each vector processor may be configured to execute independently of other vector processors. In other examples, vector processors included in a particular PVA may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm on different regions of an image. In other examples, vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even different algorithms on a sequence of images or portions of an image. Among other things, any number of PVAs may be included in a hardware-accelerated cluster, and any number of vector processors may be included in each PVA. Furthermore, PVAs may include additional error-correcting code (ECC) memory to enhance overall system security.

[0118] One or more accelerators 614 (e.g., a hardware acceleration cluster) may include on-chip computer vision network and SRAM for providing high-bandwidth, low-latency SRAM for one or more accelerators 614. In some examples, the on-chip memory may include at least 4 MB of SRAM, including but not limited to eight field-configurable memory blocks accessible by the PVA and DLA. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory can be used. The PVA and DLA access memory via a backbone that provides high-speed memory access for the PVA and DLA. The backbone may include an on-chip computer vision network that interconnects the PVA and DLA to memory (e.g., using an APB).

[0119] An on-chip computer vision network may include an interface that determines that both the PVA and DLA have provided ready and valid signals before transmitting any control signals / addresses / data. Such an interface can provide independent phases and independent channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface may conform to ISO 26262 or IEC 61508 standards, but other standards and protocols may also be used.

[0120] In some examples, one or more SoC 604s may include a real-time ray tracing hardware accelerator, as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator can be used to rapidly and efficiently determine the location and extent of an object (e.g., within a world model), generate real-time visualization simulations for RADAR signal interpretation, sound propagation synthesis and / or analysis, sonar system simulation, general wave propagation simulation, comparison with LiDAR data for localization and / or other functions and / or uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations.

[0121] One or more accelerators (e.g., hardware accelerator clusters) have wide applications in autonomous driving. A PVA (Programmable Vision accelerator) could be a programmable vision accelerator used in critical processing stages of ADA (Advanced Driver Assistance Systems) and autonomous vehicles. The capabilities of a PVA are well-suited for algorithmic domains requiring predictable processing with low power consumption and low latency. In other words, PVAs perform well on semi-intensive or conventionally intensive computations, even on small datasets that require predictable runtimes with low latency and low power consumption. Therefore, in the context of autonomous vehicle platforms, PVAs are designed to run classic computer vision algorithms, as they are highly efficient in object detection and integer mathematical operations.

[0122] For example, according to one embodiment of this technology, a PVA is used to perform computer stereo vision. In some examples, a semi-global matching-based algorithm can be used, although this is not intended to limit it. Many applications of Level 3-5 autonomous driving require instantaneous motion estimation / stereo matching (e.g., motion structures, pedestrian recognition, lane detection, etc.). A PVA can perform computer stereo vision functions on input from two monocular cameras.

[0123] In some examples, PVA can be used to perform dense optical flow, providing processed RADAR data based on the raw RADAR data (e.g., using 4D Fast Fourier Transform). In other examples, PVA is used for time-of-flight depth processing, for example, providing processed time-of-flight data by processing the raw time-of-flight data.

[0124] DLA can be used to run any type of network to enhance control and driving safety, including neural networks that output a confidence measure for each object detection. Such a confidence value can be interpreted as a probability or to provide a relative “weight” for each detection relative to other detections. This confidence value allows the system to further determine which detections should be considered true positives rather than false positives. For example, the system can set a confidence threshold and only consider detections exceeding the threshold as true positives. In an Automatic Emergency Braking (AEB) system, false positives would cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most reliable detections should be considered as triggers for AEB. DLA can run a neural network to regress the confidence value. The neural network can take at least a subset of parameters as its input, such as bounding box dimensions, obtained ground plane estimates (e.g., from another subsystem), inertial measurement unit (IMU) sensor 666 outputs related to the vehicle's 600-degree orientation and distance, and three-dimensional position estimates of objects obtained from the neural network and / or other sensors (e.g., LiDAR sensor 664 or RADAR sensor 660).

[0125] One or more SoCs 604 may include one or more data storage units 616 (e.g., memory). The data storage unit 616 may be on-chip memory of the SoC 604, which may store neural networks to be executed on the GPU and / or DLA. In some examples, the capacity of the data storage unit 616 may be large enough to store multiple neural network instances for redundancy and security. Data cache 612 may include a level 2 or level 3 cache 612. As described herein, references to one or more data storage units 616 may include references to memory associated with the PVA, DLA, and / or one or more other accelerators 614.

[0126] One or more SoCs 604 may include one or more processors 610 (e.g., embedded processors). Processor 610 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management functions, as well as related security implementations. The boot and power management processor may be part of a boot sequence for one or more SoCs 604s and may provide runtime power management services. The boot power and management processor may provide clock and voltage programming, system low-power state transition assistance, management of SoC 604 thermal sensors and temperature sensors, and / or management of SoC 604 power states. Each temperature sensor may be implemented as a ring oscillator with an output frequency proportional to temperature, and one or more SoCs 604s may use the ring oscillator to detect the temperature of one or more CPUs 606s, one or more GPUs 608s, and / or one or more accelerators 614s. If a temperature is determined to exceed a threshold, the boot and power management processor may enter a temperature fault routine and place one or more SoCs 604s into a low-power state and / or place vehicle 600 into a driver-safe parking mode (e.g., safely parking vehicle 600).

[0127] One or more processors 610 may also include a set of embedded processors that can serve as an audio processing engine. The audio processing engine may be an audio subsystem capable of providing full hardware support for multi-channel audio through multiple interfaces and a wide range of flexible audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core of a digital signal processor with dedicated RAM.

[0128] One or more processors 610 may also include a normally-on processor engine that provides the necessary hardware functionality to support low-power sensor management and wake-up use cases. The normally-on processor engine may include a processor core, tightly coupled RAM, peripheral support (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0129] One or more processors 610 may also include a secure cluster engine, which includes a dedicated processor subsystem for handling security management for automotive applications. The secure cluster engine may include two or more processor cores, tightly coupled RAM, support for peripheral devices (e.g., timers, interrupt controllers, etc.), and / or routing logic. In secure mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic to detect any differences in their operations.

[0130] One or more processors 610 may also include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0131] One or more processors 610 may also include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine as part of the camera processing pipeline.

[0132] One or more processors 610 may include a video image synthesizer, which may be a processing block (e.g., implemented on a microprocessor) that implements the video post-processing functions required by the video playback application to generate the final image of the player window. The video image synthesizer may perform lens distortion correction on one or more wide-angle cameras 670, one or more surround cameras 674, and / or in-cabin monitoring camera sensors. The in-cabin monitoring camera sensors are preferably monitored by a neural network running on another instance of an advanced SoC, configured to recognize in-cabin events and respond accordingly. The in-cabin system may perform lip reading to activate cellular service and make phone calls, dictate emails, change vehicle destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. Some functions are only available to the driver when the vehicle is running in automatic mode; otherwise, they are disabled.

[0133] Video image synthesizers may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, in the case of motion in a video, noise reduction appropriately weights spatial information, thereby reducing the weight of information provided by adjacent frames. In cases where an image or part of an image does not contain motion, temporal noise reduction performed by the video image synthesizer can use information from the previous image to reduce noise in the current image.

[0134] The video image compositor can also be configured to perform stereoscopic correction on input stereoscopic shot frames. When the operating system desktop is in use, the video image compositor can also be used for user interface compositing, without requiring the GPU 608 to continuously render new surfaces. Even when one or more GPUs 608 are powered on and active during 3D rendering, the video image compositor can be used to offload one or more GPUs 608 to improve performance and responsiveness.

[0135] One or more SoCs 604 may also include a Mobile Industrial Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for receiving video and input from a camera and associated pixel input functions. One or more SoCs 604 may also include one or more input / output controllers that may be software-controlled and can be used to receive I / O signals that are not assigned a specific role.

[0136] One or more SoCs 604 may also include a wide range of peripheral interfaces for communication with peripheral devices, audio codecs, power management and / or other devices. One or more SoCs 604 may be used to process data from cameras (e.g., via gigabit multimedia serial links and Ethernet connections), sensors (e.g., one or more LiDAR sensors 664, one or more RADAR sensors 660, etc., connected via Ethernet), from bus 602 (e.g., vehicle 600 speed, steering wheel position, etc.), and from one or more GNSS sensors 658 (e.g., via Ethernet or CAN bus connections). One or more SoCs 604 may also include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine and may be used to free one or more CPUs 606 from routine data management tasks.

[0137] One or more SoCs 604 can form an end-to-end platform with a flexible architecture spanning automation levels 3-5, providing a comprehensive functional safety architecture that leverages and effectively utilizes computer vision and ADAS technologies to achieve diversity and redundancy, providing a platform for a flexible and reliable driver software stack and deep learning tools. Compared to traditional systems, one or more SoCs 604 can be faster, more reliable, and even more energy-efficient and space-saving. For example, when one or more accelerators 614 are combined with one or more CPUs 606, one or more GPUs 608, and one or more data storage units 616, a fast and efficient platform can be provided for Level 3-5 autonomous vehicles.

[0138] Therefore, this technology offers capabilities and functionalities that are unavailable in traditional systems. For example, computer vision algorithms can be executed on a CPU, which can be configured using a high-level programming language (such as C) to execute a wide variety of processing algorithms on diverse visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for automotive ADAS applications and practical Level 3-5 autonomous vehicles.

[0139] Compared to traditional systems, the techniques described in this paper, by providing CPU complexes, GPU complexes, and hardware acceleration clusters, allow for the simultaneous and / or sequential execution of multiple neural networks and the combination of results to achieve Level 3–5 autonomous driving capabilities. For example, a CNN executing on a DLA or dGPU (e.g., one or more GPU 620s) can include text and character recognition, allowing a supercomputer to read and understand traffic signs, including signs for which the neural network has not yet been specifically trained. The DLA can also include a neural network capable of recognizing, interpreting, and providing semantic understanding of symbols, and passing this semantic understanding to a path planning module running on the CPU complex.

[0140] Another example is the ability to run multiple neural networks simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of "Warning: Flashing lights indicate icing conditions" and a light can be interpreted independently or jointly by multiple neural networks. The sign itself can be recognized as a traffic sign by a first deployed neural network (e.g., a trained neural network), while the text "Flashing lights indicate icing conditions" can be interpreted by a second deployed neural network, which, when a flashing light is detected, notifies the vehicle routing software (preferably executed on a CPU complex) of the presence of icing conditions. A third deployed neural network can be used to identify the flashing light and notify the vehicle routing software of its presence by operating it across multiple frames. All three neural networks can run simultaneously, for example, within a DLA and / or on one or more GPUs 608.

[0141] In some examples, the CNN used for facial recognition and owner identification can use data from camera sensors to identify the presence of an authorized driver and / or the owner of the vehicle 600. The engine can be unlocked using a normally open sensor when the owner approaches the driver's door and turns on the lights, and the vehicle can be disabled in safe mode when the owner leaves. In this way, one or more SoCs 604 provide anti-theft and / or carjacking protection.

[0142] In another example, the CNN for emergency vehicle detection and identification can use data from microphone 696 to detect and identify emergency vehicle sirens. Unlike conventional systems that use a general classifier to detect sirens and manually extract features, one or more SoCs(s) 604 use the CNN to classify environmental and urban sounds, as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative closing speed of emergency vehicles (e.g., by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the vehicle's operating area, as identified by one or more GNSS sensors 658. Thus, for example, when operating in Europe, the CNN will seek to detect European sirens, and when operating in the United States, the CNN will seek to identify only North American sirens. Once an emergency vehicle is detected, an emergency vehicle safety routine can be executed using a control program with the aid of ultrasonic sensors 662, causing the vehicle to slow down, pull over, stop, and / or idle until one or more emergency vehicles pass.

[0143] The vehicle may include one or more CPUs 618 (e.g., one or more discrete CPUs or one or more dCPUs) coupled to one or more SoCs 604 via high-speed interconnects (e.g., PCIe). For example, one or more CPUs 618 may include x86 processors. The CPUs 618 can be used to perform any of a variety of functions, including arbitrating potentially inconsistent results between ADAS sensors and SoCs 604, and / or monitoring the status and health of one or more controllers 636 and / or infotainment SoCs 630.

[0144] Vehicle 600 may include one or more GPUs 620 (e.g., one or more discrete GPUs or one or more dGPUs) coupled to SoC 604 via high-speed interconnects (e.g., NVIDIA's NVLINK). One or more GPUs 620 may provide additional artificial intelligence capabilities, such as by executing redundant and / or different neural networks, and may be used to train and / or update neural networks based on inputs from sensors of vehicle 600 (e.g., sensor data).

[0145] Vehicle 600 may also include a network interface 624, which may include one or more wireless antennas 626 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). Network interface 624 can be used to enable wireless connectivity via the Internet to the cloud (e.g., with one or more servers 678 and / or other network devices), other vehicles, and / or computing devices (e.g., a passenger's client device). For communication with other vehicles, direct and / or indirect links can be established between the two vehicles (e.g., via a network and the Internet). A vehicle-to-vehicle communication link can provide a direct link. A vehicle-to-vehicle communication link can provide vehicle 600 with information about vehicles nearby (e.g., vehicles in front, to the side, and / or behind vehicle 600). This functionality may be part of vehicle 600's cooperative adaptive cruise control function.

[0146] Network interface 624 may include a SoC that provides modulation and demodulation functions and enables one or more controllers 636 to communicate over a wireless network. Network interface 624 may include an RF front-end for up-conversion from baseband to RF and down-conversion from RF to baseband. Frequency conversion can be performed using well-known processes and / or using superheterodyne processes. In some examples, the RF front-end functionality may be provided by a separate chip. The network interface may include wireless functions for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0147] The vehicle 600 may further include one or more data storage units 628, which may be off-chip (e.g., off-SoC). The data storage unit 628 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0148] The vehicle 600 may also include one or more GNSS sensors 658. One or more GNSS sensors 658 (e.g., GPS, assisted GPS sensors, differential GPS (DGPS) sensors, etc.) are used to assist in mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 658 can be used, including, for example, but not limited to, GPS with a USB connector having an Ethernet-to-serial (RS-232) bridge.

[0149] Vehicle 600 may also include one or more RADAR sensors 660. Even in dark and / or inclement weather conditions, vehicle 600 can use one or more RADAR sensors 660 for remote vehicle detection. The RADAR functional safety level can be ASIL B. One or more RADAR sensors 660 can use CAN and / or bus 602 (e.g., to transmit data generated by one or more RADAR sensors 660) for control and access to target tracking data; in some examples, raw data is accessed via Ethernet. Various RADAR sensor types can be used. For example, but not limited to, one or more RADAR sensors 660 can be used for front, rear, and side RADAR applications. In some examples, pulse Doppler RADAR sensors are used.

[0150] One or more RADAR sensors 660 may include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In some examples, a long-range RADAR can be used for adaptive cruise control functions. A long-range RADAR system can provide a wide field of view, for example, within a range of 250 meters, achieved through two or more independent scans. One or more RADAR sensors 660 can help distinguish between static and moving objects and can be used by ADAS systems for emergency braking assist and forward collision warning. The long-range RADAR sensor may include a monostatic multimode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record 600 views of the environment around the vehicle at high speeds with minimal traffic interference from adjacent lanes. The other two antennas can expand the field of view, enabling it to quickly detect vehicles entering or leaving the vehicle within 600 lanes.

[0151] For example, a mid-range RADAR system may include a range of up to 660 meters (front) or 80 meters (rear), and a field of view of up to 42 degrees (front) or 650 degrees (rear). A short-range RADAR system may include, but is not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system may generate two beams, continuously monitoring the blind spots behind and beside the vehicle.

[0152] ADAS systems can use near-field RADAR systems for blind spot detection and / or lane change assistance.

[0153] Vehicle 600 may also include one or more ultrasonic sensors 662. One or more ultrasonic sensors 662 may be located at the front, rear, and / or sides of vehicle 600 and may be used for parking assistance and / or creating and updating occupancy grids. Various ultrasonic sensors 662 may be used, and different ultrasonic sensors 662 may be used for different detection ranges (e.g., 2.5m, 4m). One or more ultrasonic sensors 662 may operate at the ASIL B functional safety level.

[0154] Vehicle 600 may include one or more LiDAR sensors 664. The one or more LiDAR sensors 664 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The functional safety level of the one or more LiDAR sensors 664 may be ASIL B. In some examples, vehicle 600 may include multiple LiDAR sensors 664 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0155] In some examples, one or more LiDAR sensors 664 can provide a 360-degree field of view of objects and a list of their distances. One or more commercially available LiDAR sensors 664 have an advertised range of approximately 600m, an accuracy of 2cm-3cm, and support, for example, a 600Mbps Ethernet connection. In some examples, one or more non-protruding LiDAR sensors 664 can be used. In examples of this type, one or more LiDAR sensors 664 can be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of a vehicle 600. In examples of this type, one or more LiDAR sensors 664 can provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, with a range of 200m even for low-reflectivity objects. One or more front-mounted LiDAR sensors 664 can be configured with a horizontal field of view between 45 and 135 degrees.

[0156] In some examples, LiDAR technology, such as 3D flash LiDAR, can also be used. 3D flash LiDAR uses a laser flash as the transmission source, illuminating approximately 200 meters around the vehicle. The flash LiDAR device includes a receiver that records the laser pulse transmission time and the reflected light on each pixel, with each pixel corresponding to the range from the vehicle to the object. Flash LiDAR can utilize each laser flash to generate highly accurate, distortion-free environmental images. In some examples, four flash LiDAR sensors can be deployed, one on each side of the vehicle 600. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras with no moving parts other than a fan (e.g., a non-scanning LiDAR device). The flash LiDAR device uses 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D distance point cloud and co-registered intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, one or more LiDAR sensors 664 can be less susceptible to motion blur, vibration, and / or shock.

[0157] The vehicle may also include one or more IMU sensors 666. In some examples, one or more IMU sensors 666 may be located at the center of the rear axle of the vehicle 600. One or more IMU sensors 666 may include, for example, but not limited to, one or more accelerometers, one or more magnetometers, one or more gyroscopes, one or more magnetic compasses, and / or other sensor types. In some examples, such as in a six-axis application, one or more IMU sensors 666 may include accelerometers and gyroscopes, while in a nine-axis application, one or more IMU sensors 666 may include accelerometers, gyroscopes, and magnetometers.

[0158] In some embodiments, one or more IMU sensors 666 can be implemented as a miniaturized, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, one or more IMU sensors 666 can enable vehicle 600 to estimate heading by directly observing and correlating velocity changes from GPS to one or more IMU sensors 666, without requiring input from magnetic sensors. In some examples, one or more IMU sensors 666 and one or more GNSS sensors 658 can be combined in a single integrated unit.

[0159] The vehicle may include one or more microphones 696 placed inside and / or around the vehicle 600. One or more microphones 696 may be used for emergency vehicle detection and identification, etc.

[0160] The vehicle may also include any number of camera types, including one or more stereo cameras 668, one or more wide-angle cameras 670, one or more infrared cameras 672, one or more surround cameras 674, one or more long-range and / or mid-range cameras 698, and / or other camera types. The cameras can be used to capture image data of the entire perimeter of the vehicle 600. The types of cameras used depend on the embodiment and requirements of the vehicle 600, and any combination of camera types can be used to provide the necessary coverage around the vehicle 600. Furthermore, the number of cameras can vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or other numbers of cameras. As an example, the cameras may support, but are not limited to, Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. This document will refer to... Figure 6A and Figure 6B Describe each camera in more detail.

[0161] Vehicle 600 may also include one or more vibration sensors 642. One or more vibration sensors 642 can measure vibrations of vehicle components, such as axles. For example, changes in vibration may indicate changes in road surface. In another example, when two or more vibration sensors 642 are used, differences between vibrations can be used to determine friction or slippage on the road surface (e.g., when the vibration difference is between a driven shaft and a freely rotating shaft).

[0162] Vehicle 600 may include ADAS system 638. In some examples, ADAS system 638 may include SoC. ADAS system 638 may include automatic / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic warning (RCTW), collision warning system (CWS), lane centering (LC) and / or other features and functions.

[0163] The ACC system may use one or more RADAR sensors 660, one or more LiDAR sensors 664, and / or one or more cameras. The ACC system may include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle directly in front of vehicle 600 and automatically adjusts the vehicle speed to maintain a safe distance. Lateral ACC performs distance holding and suggests lane changes to vehicle 600 if necessary. Lateral ACC is associated with other ADAS applications such as LCA and CWS.

[0164] CACC uses information from other vehicles, which can be received indirectly from other vehicles via a wireless link or a network connection (e.g., via the Internet) through network interface 624 and / or one or more wireless antennas 626. The direct link can be provided by a vehicle-to-vehicle (V2V) communication link, while the indirect link can be an infrastructure-to-vehicle (I2V) communication link. Generally, the V2V communication concept provides information about vehicles ahead (e.g., vehicles directly in front of vehicle 600 and in the same lane), while the I2V communication concept provides information about traffic ahead. A CACC system may include one or both I2V and V2V information sources. By taking into account information about vehicles ahead of vehicle 600, CACC may be more reliable and has the potential to improve traffic flow smoothness and reduce congestion on the road.

[0165] The Forward-Facing Warning (FCW) system is designed to alert the driver to hazards so that the driver can take corrective action. The FCW system uses a forward-facing camera and / or one or more RADAR sensors 660 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, electrically coupled to driver feedback, such as a display, speaker, and / or vibration components. The FCW system can provide warnings such as audible, visual, haptic, and / or rapid braking pulses.

[0166] The AEB system detects an impending forward collision with another vehicle or other object. If the driver does not take corrective action within a specified time or distance parameter, the AEB system may automatically apply the brakes. The AEB system may use one or more front-facing cameras and / or one or more RADAR sensors 660 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision. If the driver does not take corrective action, the AEB system may automatically apply the brakes to prevent or at least mitigate the effects of the anticipated collision. The AEB system may include technologies such as dynamic brake support and / or collision proximity braking.

[0167] The Lane Departure Warning (LDW) system provides visual, auditory, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle crosses lane markings. The LDW system does not activate when the driver indicates intentional lane departure by activating a turn signal. The LDW system may use a front-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to actuator feedback, such as a display, speaker, and / or vibration assembly.

[0168] The LKA system is a variant of the LDW system. If the vehicle begins to leave the lane at 60°, the LKA system provides steering input or braking to correct the vehicle's 60° deviation.

[0169] The BSW system detects and warns drivers of vehicles within the blind spot. The BSW system can provide visual, auditory, and / or tactile alerts to indicate unsafe merging or lane changing. Additional warnings may be provided when the driver uses a turn signal. The BSW system may utilize a rear-facing camera and / or RADAR sensor 660 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration assembly.

[0170] When the vehicle 600 is reversing and detects an object outside the range of the rear camera, the RCTW system can provide visual, auditory, and / or tactile notifications. Some RCTW systems include AEB (Autonomous Emergency Braking) to ensure the application of the vehicle's brakes to avoid a collision. The RCTW system may use one or more rearward RADAR sensors 660, which are coupled to a dedicated processor, DSP, FPGA, and / or ASIC, electrically coupled to actuator feedback, such as a display, speaker, and / or vibration assembly.

[0171] Traditional ADAS systems can be prone to false positives, which can be frustrating and distracting for drivers, but usually do not lead to catastrophic consequences because the ADAS system alerts the driver and allows them to determine whether a safe situation truly exists and take appropriate action. However, in an autonomous vehicle 600, in the event of conflicting results, the vehicle 600 itself must decide whether to heed the results from the main computer or the auxiliary computer (e.g., the first controller 636 or the second controller 636). For example, in some embodiments, the ADAS system 638 may be a backup and / or auxiliary computer for providing perception information to a backup computer module. A backup computer rationality monitor may run redundant different software on hardware components to detect faults in perception and dynamic driving tasks. The output from the ADAS system 638 may be provided to a monitoring MCU. If the outputs of the main computer and the auxiliary computer conflict, the monitoring MCU must determine how to reconcile the conflict to ensure safe operation.

[0172] In some examples, the master computer can be configured to provide a confidence score to the monitoring MCU, indicating the master computer's confidence level in a selected result. If the confidence score exceeds a threshold, the monitoring MCU can follow the master computer's direction regardless of whether the auxiliary computer provides conflicting or inconsistent results. If the confidence score does not meet the threshold, and the master computer and the auxiliary computer indicate different results (e.g., conflict), the monitoring MCU can arbitrate between the computers to determine the appropriate result.

[0173] The monitoring MCU can be configured to run one or more trained and configured neural networks to determine the conditions under which the auxiliary computer provides a false alarm based on the outputs of the main computer and the auxiliary computer. Therefore, one or more neural networks in the monitoring MCU can learn when the output of the auxiliary computer is reliable and when it is not. For example, when the auxiliary computer is a RADAR-based FCW system, one or more neural networks in the monitoring MCU can learn when the FCW system recognizes a metallic object that is not actually dangerous, such as a drain grille or manhole cover that triggers an alarm. Similarly, when the auxiliary computer is a camera-based LDW system, the neural network in the monitoring MCU can learn to override the LDW when a bicycle or pedestrian is present and lane departure is actually the safest maneuver. In embodiments that include one or more neural networks running on the monitoring MCU, the monitoring MCU may include at least one of a DLA or GPU adapted to run one or more neural networks with associated memory. In a preferred embodiment, the monitoring MCU may include and / or include components that are SoC604.

[0174] In other examples, ADAS system 638 may include an auxiliary computer that performs ADAS functions using conventional computer vision rules. Therefore, the auxiliary computer can use classic computer vision rules (if-then), and the presence of one or more neural networks in the monitoring MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identification make the entire system more fault-tolerant, especially to failures caused by software (or hardware / software interface) functionality. For instance, if a software defect or error exists in the software running on the main computer, and different software code running on the auxiliary computer provides the same overall result, the monitoring MCU can have greater confidence in the correctness of the overall result, and the software or hardware defect on the main computer did not lead to a major error.

[0175] In some examples, the output of ADAS system 638 may be fed into the perception block and / or the dynamic drive task block of the host computer. For example, if ADAS system 638 indicates a forward collision warning due to an object directly in front, the perception block may use this information when identifying the object. In other examples, as described herein, the secondary computer may have its own trained neural network, thereby reducing the risk of false positives.

[0176] Vehicle 600 may also include an infotainment SoC 630 (e.g., an in-vehicle infotainment system (IVI)). Although shown and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 630 may include a combination of hardware and software that can be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., television, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or provide information services to vehicle 600 (e.g., navigation system, rear parking assist, radio data system, vehicle-related information such as fuel level, total distance traveled, brake fuel level, engine oil level, door opening / closing, air filter information, etc.). For example, the infotainment SoC 630 may be a radio, disk player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, Wi-Fi, steering wheel audio controls, hands-free voice control, head-up display (HUD), HMI display 634, telecom device, control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 630 may also be used to provide information to vehicle users (e.g., visual and / or auditory), such as information from ADAS system 638, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0177] The infotainment SoC 630 may include GPU functionality. The infotainment SoC 630 can communicate with other devices, systems, and / or components of the vehicle 600 via bus 602 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 630 may be coupled to a monitoring MCU, allowing the GPU of the infotainment system to perform self-driving functions in the event of a failure of one or more main controllers 636 (e.g., the main computer and / or backup computer of the vehicle 600). In such an example, the infotainment SoC 630 may place the vehicle 600 into a driver-to-safe parking mode, as described herein.

[0178] Vehicle 600 may also include an instrument cluster 632 (e.g., a digital instrument panel, electronic instrument cluster, etc.). The instrument cluster 632 may include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 632 may include a set of instruments such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn signals, shift position indicator, one or more seatbelt warning lights, one or more parking brake warning lights, one or more engine malfunction lights, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 630 and the instrument cluster 632. In other words, the instrument cluster 632 may be included as part of the infotainment SoC 630, and vice versa.

[0179] Figure 6D It is one or more cloud-based servers according to some embodiments of this disclosure and Figure 6A A system diagram illustrating communication between exemplary autonomous vehicles 600 is provided. System 676 may include one or more servers 678, one or more networks 690, and vehicles, including vehicle 600. One or more servers 678 may include multiple GPUs 684(a)-684(H) (collectively referred to as GPU 684), PCIe switches 682(a)-682(H) (collectively referred to as PCIe switch 682), and / or CPUs 680(A)-680(B) (collectively referred to herein as CPU 680). GPUs 684, CPUs 680, and PCIe switches may be interconnected via high-speed interconnects, such as, but not limited to, NVIDIA-developed NVLink interface 688 and / or PCIe connection 686. In some examples, GPUs 684 are connected via NVLink and / or NV switch SoCs, and GPUs 684 and PCIe switches 682 are connected via PCIe interconnects. Although eight GPUs 684, two CPUs 680, and two PCIe switches are illustrated, this is not intended to be limiting. According to an embodiment, each of one or more servers 678 may include any number of GPUs 684, CPUs 680, and / or PCIe switches. For example, one or more servers 678 may each include eight, sixteen, thirty-two, and / or more GPUs 684.

[0180] One or more servers 678 may receive image data representing images from vehicles via one or more networks 690, showing unexpected or changed road conditions, such as recently started roadwork. One or more servers 678 may send neural network 692, updated neural network 692, and / or map information 694, including information about traffic and road conditions, to vehicles via one or more networks 690. Updates to map information 694 may include updates to HD map 622, such as information about construction sites, potholes, detours, floods, and / or other obstacles. In some examples, neural network 692, updated neural network 692, and / or map information 694 may originate from new training and / or experience represented in data received from any number of vehicles in the environment, and / or based on training performed in a data center (e.g., using one or more servers 678 and / or other servers).

[0181] One or more servers 678 can be used to train machine learning models (e.g., neural networks) based on training data. Training data may be generated by the vehicle and / or generated in a simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is unlabeled and / or preprocessed (e.g., the neural network does not require supervised learning). Training can be performed according to any one or more classes of machine learning techniques, including but not limited to: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component analysis and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, it can be used by the vehicle (e.g., transmitted to vehicle 690 via one or more networks, and / or the machine learning model can be used by one or more servers 678 for remote monitoring of the vehicle).

[0182] In some examples, one or more servers 678 may receive data from a vehicle and apply the data to state-of-the-art real-time neural networks for real-time intelligent inference. One or more servers 678 may include a deep learning supercomputer and / or a dedicated AI computer powered by a GPU 684, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, one or more servers 678 may include a deep learning infrastructure in a data center using only CPU power.

[0183] The deep learning infrastructure of one or more servers 678 can perform rapid real-time inference and use this capability to assess and verify the health status of the processors, software, and / or associated hardware in vehicle 600. For example, the deep learning infrastructure can receive periodic updates from vehicle 600, such as sequences of images and / or objects located by vehicle 600 in the image sequence (e.g., through computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them with objects identified by vehicle 600. If the results do not match and the infrastructure concludes that the AI ​​in vehicle 600 has malfunctioned, one or more servers 678 can send a signal to vehicle 600 instructing the fail-safe computer of vehicle 600 to take control, notify passengers, and complete a safe stopping operation.

[0184] For inference, one or more servers 678 may include one or more GPUs 684 and one or more programmable inference accelerators (such as NVIDIA's TensorRT). The combination of GPU-powered servers and inference acceleration can enable real-time responses. In other examples, such as where performance is less critical, inference can be performed using servers powered by CPUs, FPGAs, and other processors.

[0185] Example computing device

[0186] Figure 7 This is a block diagram of an example computing device 700 suitable for implementing some embodiments of the present disclosure. The computing device 700 may include an interconnect system 702 directly or indirectly coupled to: a memory 704, one or more central processing units (CPUs) 706, one or more graphics processing units (GPUs) 708, a communication interface 710, input / output (I / O) ports 712, input / output components 714, a power supply 716, one or more presentation components 718 (e.g., displays), and one or more logic units 720. In at least one embodiment, one or more computing devices 700 may include one or more virtual machines (VMs), and / or any component thereof may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 708 may include one or more vGPUs, one or more CPUs 706 may include one or more vCPUs, and / or one or more logic units 720 may include one or more virtual logic units. Therefore, one or more computing devices 700 may include discrete components (e.g., a complete GPU dedicated to computing device 700), virtual components (e.g., a portion of the GPU dedicated to computing device 700), or a combination thereof.

[0187] although Figure 7The various modules are shown as being connected to lines via interconnect system 702, but this is not intended to be limiting, but merely for clarity. For example, in some embodiments, a presentation component 718 such as a display device can be considered as I / O component 714 (e.g., if the display is a touchscreen). As another example, CPU 706 and / or GPU 708 may include memory (e.g., memory 704 may also represent a storage device in addition to the memory of GPU 708, CPU 706, and / or other components). In other words, Figure 7 The computing devices mentioned are merely illustrative. No distinction is made between "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as is the case in [the context of the previous sentence]. Figure 7 As envisioned within the scope of computing devices.

[0188] Interconnect system 702 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 702 may include one or more bus or link types, such as Industry Standard Architecture (ISA) buses, Extended Industry Standard Architecture (EISA) buses, Video Electronics Standards Association (VESA) buses, Peripheral Component Interconnect (PCI) buses, Peripheral Component Interconnect Through (PCIE) buses, and / or other types of buses or links. In some embodiments, there is a direct connection between components. For example, CPU 706 may be directly connected to memory 704. Furthermore, CPU 706 may be directly connected to GPU 708. In cases where there is a direct connection or point-to-point connection between components, interconnect system 702 may include a PCIe link for performing the connection. In these examples, a PCI bus is not required in computing device 700.

[0189] The memory 704 may include any of a variety of computer-readable media. Computer-readable media can be any available medium accessible by the computing device 700. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0190] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 704 may store computer-readable instructions (e.g., instructions representing one or more programs and / or one or more program elements), such as an operating system. Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disk (DVD) or other optical disc storage, cassette tape, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 700. As used herein, computer storage media itself does not include signals.

[0191] Computer storage media can contain computer-readable instructions, data structures, program modules, and / or other data types contained in modulated data signals (such as carrier waves or other transmission mechanisms), and include any information delivery medium. The term "modulated data signal" can refer to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media, such as wired networks or direct wired connections, and wireless media, such as acoustic, RF, infrared, and other wireless media. Any combination of the foregoing should also be included within the scope of computer-readable media.

[0192] One or more CPUs 706 may be configured to execute at least some computer-readable instructions to control one or more components of computing device 700 to perform one or more methods and / or processes described herein. Each of the one or more CPUs 706 may include one or more cores capable of processing multiple software threads simultaneously (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.). The one or more CPUs 706 may include any type of processor and may include different types of processors depending on the type of computing device 700 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 700, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors (e.g., math coprocessors), computing device 700 may also include one or more CPUs 706.

[0193] In addition to, or selected from, one or more CPUs 706, one or more GPUs 708 may be configured to execute at least some computer-readable instructions to control one or more components of computing device 700 to perform one or more methods and / or processes described herein. One or more GPUs 708 may be integrated GPUs (e.g., having one or more CPUs 706 and / or one or more GPUs 708 may be discrete GPUs). In embodiments, one or more GPUs 708 may be coprocessors of one or more CPUs 706. Computing device 700 may use one or more GPUs 708 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, one or more GPUs 708 may be used for general-purpose computing on a GPU (GPGPU). One or more GPUs 708 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. One or more GPUs 708 may generate pixel data for an output image in response to rendering commands (e.g., rendering commands received via a host interface from one or more CPUs 706). One or more GPUs 708 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 704. One or more GPUs 708 may include two or more GPUs operating in parallel (e.g., via a link). The link may directly connect the GPUs (e.g., using NVLINK) or connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 708 may generate pixel data or GPGPU data for different portions of the output or different outputs (e.g., a first GPU for a first image, a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0194] In addition to one or more CPUs 706 and / or one or more GPUs 708, or selected from one or more CPUs 706 and / or one or more GPUs 708, one or more logic units 720 may be configured to execute at least some computer-readable instructions to control one or more components of computing device 700 to perform one or more methods and / or processes described herein. In embodiments, one or more CPUs 706, one or more GPUs 708, and / or one or more logic units 720 may execute any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 720 may be part of and / or integrated into one or more CPUs 706 and / or GPUs 708, and / or one or more logic units 720 may be discrete components or otherwise located external to one or more CPUs 706 and / or one or more GPUs 708. One or more logic units 720 may be coprocessors of one or more CPUs 706 and / or one or more GPUs 708.

[0195] Examples of one or more logic units 720 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), a tensor core (TC), a tensor processing unit (TPU), a pixel vision core (PVC), a vision processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multiprocessor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application-specific integrated circuit (ASIC), a floating-point unit (FPU), input / output (I / O) elements, peripheral component interconnect (PCI) or peripheral component interconnect pass-through (PCIe) elements, and / or the like.

[0196] The communication interface 710 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 700 to communicate with other computing devices via an electronic communication network including wired and / or wireless communications. The communication interface 710 may include components and functions to support communication over any of a variety of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, one or more logic units 720 and / or the communication interface 710 may include one or more data processing units (DPUs) for directly transmitting data received via a network and / or via interconnect system 702 to one or more GPUs 708 (e.g., their memory).

[0197] I / O port 712 enables the computing device 700 to be logically coupled to other devices, including I / O components 714, one or more presentation components 718, and / or other components, some of which may be built into (e.g., integrated into) the computing device 700. Illustrative I / O components 714 include microphones, mice, keyboards, joysticks, game pads, game controllers, satellite antennas, scanners, printers, wireless devices, etc. I / O components 714 can provide a natural user interface (NUI) that processes user-generated air gestures, voice, or other physiological input. In some cases, the input can be transmitted to appropriate network elements for further processing. The NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometrics, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition associated with the display of the computing device 700 (described in more detail below). The computing device 700 may include depth cameras, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture detection and recognition. In addition, computing device 700 may include an accelerometer or gyroscope capable of detecting motion (e.g., as part of an inertial measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by computing device 700 to render immersive augmented reality or virtual reality.

[0198] Power supply 716 may include hard-wired power supply, battery power supply, or a combination thereof. Power supply 716 may supply power to computing device 700 so that components of computing device 700 can operate.

[0199] One or more presentation components 718 may include displays (e.g., monitors, touchscreens, television screens, head-up displays (HUDs), other display types, or combinations thereof), speakers, and / or other presentation components. One or more presentation components 718 may receive data from other components (e.g., one or more GPUs 708, one or more CPUs 706, DPUs, etc.) and output data (e.g., as images, videos, sounds, etc.).

[0200] Example Data Center

[0201] Figure 8 An example data center 800 that can be used in at least one embodiment of this disclosure is shown. The data center 800 may include a data center infrastructure layer 810, a framework layer 820, a software layer 830, and / or an application layer 840.

[0202] like Figure 8 As shown, the data center infrastructure layer 810 may include a resource coordinator 812, packet computing resources 814, and node computing resources (“nodes CRs”) 816(1)-816(N), where “N” represents any integer. In at least one embodiment, nodes CRs 816(1)-816(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), storage devices (e.g., dynamic read-only memory), and in some embodiments, storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules and / or cooling modules, etc. One or more nodes CRs 816(1)-816(N) may correspond to a server having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CRs 816(1)-8161(N) may include one or more virtual components, such as vGPU, vCPU and / or similar components, and / or one or more nodes CRs 816(1)-816(N) may correspond to virtual machines (VMs).

[0203] In at least one embodiment, the packet computing resource 814 may include individual packets of node CRs 816 located within one or more racks (not shown), or multiple racks located within data centers in different geographical locations (also not shown). Individual packets of node CRs 816 within the packet computing resource 814 may include packet computing, networking, memory, or storage resources, which may be configured or allocated to support one or more workloads. In at least one embodiment, multiple node CRs 816, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any combination of any number of power modules, cooling modules, and / or network switches.

[0204] Resource coordinator 812 may be configured or otherwise control one or more nodes CRs816(1)-816(N) and / or grouped computing resources 814. In at least one embodiment, resource coordinator 812 may include a Software Design Infrastructure (SDI) management entity for data center 800. Resource coordinator 812 may include hardware, software, or some combination thereof.

[0205] In at least one embodiment, such as Figure 8 As shown, framework layer 820 may include job scheduler 833, configuration manager 834, resource manager 836, and / or distributed file system 838. Framework layer 820 may include frameworks for software 832 supporting software layer 830 and / or one or more applications 842 supporting application layer 840. Software 832 or one or more applications 842 may respectively include web-based service software or applications, such as services provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 820 may be, but is not limited to, a free and open-source software web application framework, such as Apache Spark™ (hereinafter referred to as "Spark"), which can utilize distributed file system 838 for large-scale data processing (e.g., "big data"). In at least one embodiment, job scheduler 833 may include Spark drivers to facilitate the scheduling of workloads supported by the various layers of data center 800. Configuration manager 834 may configure different layers, such as software layer 830 and framework layer 820, including Spark and distributed file system 838, to support large-scale data processing. Resource manager 836 may be able to manage cluster or group computing resources mapped to or allocated to support distributed file system 838 and job scheduler 833. In at least one embodiment, the cluster or group computing resources may include group computing resources 814 at data center infrastructure layer 810. Resource manager 836 may coordinate with resource coordinator 812 to manage these mapped or allocated computing resources.

[0206] In at least one embodiment, the software 832 included in the software layer 830 may include software used in at least a portion of the distributed file system 838 of the nodes CRs 816(1)-816(N), the grouped computing resources 814, and / or the framework layer 820. One or more types of software may include, but are not limited to, internet webpage search software, email virus scanning software, database software, and streaming video content software.

[0207] In at least one embodiment, the application 842 included in the application layer 840 may include one or more types of applications used by at least a portion of the nodes CRs 816(1)-816(N), the grouped computing resources 814 and / or the distributed file system 838 of the framework layer 820, but is not limited to any number of genomics applications, perceptual computing and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.) and / or other machine learning applications used in combination with one or more embodiments.

[0208] In at least one embodiment, any of the configuration manager 834, resource manager 836, and resource coordinator 812 can implement any number and type of self-modification actions based on any amount and type of data acquired in any technically feasible manner. Self-modification actions can protect data center operators of data center 800 from making potentially erroneous configuration decisions and may prevent underutilized and / or poorly performing portions of the data center.

[0209] According to one or more embodiments described herein, data center 800 may include tools, services, software, or other resources for training one or more machine learning models or using one or more machine learning models to predict or infer information. For example, one or more machine learning models may be trained by calculating weight parameters based on a neural network architecture using the software and / or computing resources described above for data center 800. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above for data center 800 by using weight parameters calculated through one or more training techniques (e.g., but not limited to the training techniques described herein).

[0210] In at least one embodiment, the data center 800 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured as services to allow a user to train or perform information inference, such as image recognition, speech recognition, or other artificial intelligence services.

[0211] Example network environment

[0212] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may... Figure 7 Implemented on one or more instances of computing devices 700—for example, each device may include similar components, features, and / or functions of computing device 700. Additionally, backend devices (servers, NAS, etc.) may be included as part of data center 800, examples of which are referred to herein. Figure 8 To describe in more detail.

[0213] Components of a network environment can communicate with each other through one or more networks, whether wired, wireless, or a combination of both. This network can include multiple networks, or networks of networks. For example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. In cases where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (along with other components) can provide wireless connectivity.

[0214] A compatible network environment may include one or more peer-to-peer network environments—in which case the server may not be included in the network environment—and one or more client-server network environments—in which case one or more servers may be included in the network environment. In a peer-to-peer network environment, the server functionality described herein can be implemented on any number of client devices.

[0215] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, and combinations thereof. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework supporting software at the software layer and / or application at the application layer. The software or one or more applications may respectively include web-based service software or applications. In embodiments, one or more client devices may use web-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open-source software web application framework, for example, one that can use a distributed file system for large-scale data processing (e.g., "big data").

[0216] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more of them) described herein. Any of these various functions can be distributed from a central or core server (e.g., servers in one or more data centers) to multiple locations, which may be located in a state, region, country, globally, etc. If the connection to a user (e.g., a client device) is relatively close to one or more edge servers, one or more core servers may assign at least a portion of the functionality to one or more edge servers. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0217] One or more client devices may include the information described in this article. Figure 7 The one or more example computing devices 700 described herein may include at least some components, features, and functions. By way of example and not limitation, the client device may be a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, aircraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming equipment or system, entertainment system, vehicle computer system, embedded system controller, remote control, device, consumer electronics device, workstation, edge device, any combination of these described devices, or any other suitable device.

[0218] 1. A computer-implemented method comprising: obtaining image data generated using a first camera having a first field of view; generating transformed image data by applying a transformation to the image data based at least on external camera data corresponding to the first camera, the transformation converting the first field of view into a second field of view simulating a second camera; computing output data representing one or more predictions using a machine learning model; and transmitting the output data to cause a vehicle to perform one or more operations based at least on the one or more predictions.

[0219] 2. The computer-implemented method according to Clause 1, wherein the first camera is a physical camera and the second camera is a virtual camera.

[0220] 3. The computer-implemented method according to clause 1 or 2, wherein generating the transformed image data further comprises: associating one or more two-dimensional 2D locations included in the image data with one or more 2D locations included in the transformed image data based on at least one of a transformation matrix or a lookup table.

[0221] 4. The computer-implemented method according to any one of Clauses 1-3, wherein the external camera data includes at least one of position data, attitude data, location data, or orientation data associated with the first camera.

[0222] 5. The computer-implemented method according to any one of Clauses 1-4, wherein the position data includes three-dimensional 3D position coordinates with reference to the position within the vehicle.

[0223] 6. The computer-implemented method according to any one of claims 1-5, wherein the machine learning model is trained using transformed training image data and corresponding transformed label data, the transformed label data including a set of 2D locations included in the transformed training image data, the set of 2D locations defining a ground truth vehicle path associated with the transformed training image data.

[0224] 7. The computer-implemented method according to any one of Clauses 1-6, wherein the transformed label data includes annotations selected from the group consisting of middle lane, right lane, left lane, left lane, right lane, merging from the left and merging from the right.

[0225] 8. The computer-implemented method according to any one of clauses 1-7 further includes: identifying a region of interest (ROI) based at least on the transformed image data.

[0226] 9. A system comprising: one or more processors configured to perform operations including: receiving image data generated using a first camera having a first field of view, the first camera being associated with a vehicle in an environment; receiving external camera data associated with the first camera; applying a transformation to the image data based at least on the external camera data to generate transformed image data, the transformation converting the first field of view into a second field of view simulating that of a second camera associated with the vehicle in the environment; computing output data representing one or more predictions using a machine learning model; and transmitting the output data such that the vehicle performs one or more operations based at least on the one or more predictions.

[0227] 10. The system according to Clause 9, wherein the first camera is a physical camera and the second camera is a virtual camera.

[0228] 11. The system according to Clause 9 or 10, wherein generating the transformed image data further comprises: associating one or more two-dimensional 2D locations included in the image data with one or more 2D locations included in the transformed image data based on a transformation matrix or lookup table.

[0229] 12. The system according to any one of Clauses 9-11, wherein the external camera data includes one or more of position data, attitude data, location data, or orientation data associated with the first camera.

[0230] 13. The system according to any one of Clauses 9-12, wherein the position data includes three-dimensional 3D position coordinates with reference to the position within the vehicle.

[0231] 14. The system according to any one of clauses 9-13, wherein the machine learning model is trained using training image data and training label data, the training label data including a set of 2D locations included in the training image data, the set of 2D locations defining a ground truth vehicle path associated with the training image data.

[0232] 15. A system according to any one of Clauses 9-14, wherein said system comprises at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing analog operations; a system for performing digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implemented using a robot; a system for performing conversational AI operations; a system implementing one or more large language models (LLMs); a system implementing one or more visual language models (VLMs); a system implementing one or more multimodal language models; a system for generating synthetic data; a system comprising one or more virtual machines (VMs); a system at least partially implemented in a data center; or

[0233] A system that utilizes cloud computing resources at least in part.

[0234] 16. At least one processor, comprising: one or more circuits configured to: receive image data generated by a camera having a first field of view in an environment of use, the camera being associated with a vehicle; apply a transformation to the image data to generate transformed image data, the transformation converting the first field of view into a simulated second field of view; compute output data representing one or more predictions using a machine learning model; and transmit the output data to cause the vehicle to perform one or more operations based at least on the one or more predictions.

[0235] 17. At least one processor according to Clause 16, wherein the machine learning model is trained using training image data and training label data, the training label data including a set of 2D locations included in the training image data, the set of 2D locations defining a ground truth vehicle path associated with the training image data.

[0236] 18. At least one processor according to Clause 16 or 17, wherein the camera is a physical camera and the second field of view is the field of view of a virtual camera.

[0237] 19. At least one processor according to any one of clauses 16-18, wherein generating the transformed image data further comprises: associating one or more two-dimensional 2D locations included in the image data with one or more 2D locations included in the transformed image data based on at least one of a transformation matrix or a lookup table.

[0238] 20. At least one processor according to any one of clauses 16-19, wherein said processor is included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing analog operations; a system for performing digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implemented using a robot; a system for performing conversational AI operations; a system implementing one or more large language models (LLMs); a system implementing one or more visual language models (VLMs); a system implementing one or more multimodal language models; a system for generating synthetic data; a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0239] This disclosure can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions, such as program modules, that are executed by a computer or other machine (such as a personal data assistant or other handheld device). Typically, program modules include routines, programs, objects, components, data structures, etc., and refer to code that performs a specific task or implements a specific abstract data type. This disclosure can be implemented in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure can also be implemented in distributed computing environments, where tasks are performed by remote processing devices linked via a communication network.

[0240] As used herein, the phrase “and / or” relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, “element A, element B, and / or element C” may include only element A, element B, element C, element A and element B, element A and element C, element B and element C, or element A, element B, and element C. Furthermore, “at least one of element A or element B” may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Additionally, “at least one of element A and element B” may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0241] To meet legal requirements, the subject matter of this disclosure has been described in detail herein. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have envisioned that the claimed subject matter may also be embodied in other ways, including different steps or combinations of steps similar to those described herein, as well as other existing or future techniques. Furthermore, although the terms “step,” “operation,” and / or “block” may be used herein to imply different elements of the method employed, they should not be construed as implying any particular order between the steps disclosed herein unless the order of the individual steps is explicitly described.

Claims

1. A computer-implemented method, comprising: Obtain image data generated using a first camera with a first field of view; Transformed image data is generated by applying a transformation to the image data based at least on external camera data corresponding to the first camera, the transformation converting the first field of view into a second field of view simulating a second camera. Use machine learning models to compute output data representing one or more predictions; as well as The output data is sent so that the vehicle performs one or more operations based on at least one or more predictions.

2. The computer-implemented method according to claim 1, wherein the first camera is a physical camera and the second camera is a virtual camera.

3. The computer-implemented method according to claim 1, wherein generating the transformed image data further includes: Based on at least one of a transformation matrix or a lookup table, one or more two-dimensional 2D locations included in the image data are associated with one or more 2D locations included in the transformed image data.

4. The computer-implemented method of claim 1, wherein the external camera data includes at least one of position data, pose data, location data, or orientation data associated with the first camera.

5. The computer-implemented method according to claim 4, wherein the position data includes three-dimensional 3D position coordinates with reference to the position inside the vehicle.

6. The computer-implemented method of claim 1, wherein the machine learning model is trained using transformed training image data and corresponding transformed label data, the transformed label data including a set of 2D locations included in the transformed training image data, the set of 2D locations defining a ground truth vehicle path associated with the transformed training image data.

7. The computer-implemented method of claim 6, wherein the transformed label data includes annotations selected from the group consisting of middle lane, right lane, left lane, left lane, right lane, merging from the left, and merging from the right.

8. The computer-implemented method according to claim 1, further comprising: The region of interest (ROI) is identified based at least on the transformed image data.

9. A system comprising: One or more processors are used to perform operations including the following: Receive image data generated using a first camera having a first field of view, the first camera being associated with a vehicle in the environment; Receive external camera data associated with the first camera; The image data is transformed based at least on the external camera data to generate transformed image data, the transformation converting the first field of view into a second field of view that simulates the vehicle associated with the environment; Use machine learning models to compute output data representing one or more predictions; as well as The output data is sent so that the vehicle performs one or more operations based on at least one or more predictions.

10. The system of claim 9, wherein the first camera is a physical camera and the second camera is a virtual camera.

11. The system of claim 9, wherein generating the transformed image data further comprises: Based on a transformation matrix or lookup table, one or more two-dimensional 2D locations included in the image data are associated with one or more 2D locations included in the transformed image data.

12. The system of claim 9, wherein the external camera data includes one or more of position data, attitude data, location data, or orientation data associated with the first camera.

13. The system of claim 12, wherein the position data includes three-dimensional 3D position coordinates with reference to the position inside the vehicle.

14. The system of claim 9, wherein the machine learning model is trained using training image data and training label data, the training label data including a set of 2D locations included in the training image data, the set of 2D locations defining a ground truth vehicle path associated with the training image data.

15. The system of claim 9, wherein the system comprises at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system used to perform deep learning operations; Systems implemented using edge devices; A system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; Systems implemented using robots; Systems used to perform conversational AI operations; A system that implements one or more large language model LLMs; A system that implements one or more Visual Language Models (VLMs); A system that implements one or more multimodal language models; A system for generating synthetic data; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

16. At least one processor, comprising: One or more circuits, said one or more circuits being used for: Receive image data generated by a camera with a first field of view in the usage environment, the camera being associated with the vehicle; A transformation is applied to the image data to generate transformed image data, the transformation converting the first field of view into an analog second field of view; Use machine learning models to compute output data representing one or more predictions; as well as The output data is sent so that the vehicle performs one or more operations based on at least one or more predictions.

17. The at least one processor of claim 16, wherein the machine learning model is trained using training image data and training label data, the training label data including a set of 2D locations included in the training image data, the set of 2D locations defining a ground truth vehicle path associated with the training image data.

18. The at least one processor of claim 16, wherein the camera is a physical camera and the second field of view is the field of view of a virtual camera.

19. The at least one processor of claim 16, wherein generating the transformed image data further comprises: Based on at least one of a transformation matrix or a lookup table, one or more two-dimensional 2D locations included in the image data are associated with one or more 2D locations included in the transformed image data.

20. The at least one processor according to claim 16, wherein the processor is comprised of at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system used to perform deep learning operations; Systems implemented using edge devices; A system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; Systems implemented using robots; Systems used to perform conversational AI operations; A system that implements one or more large language model LLMs; A system that implements one or more Visual Language Models (VLMs); A system that implements one or more multimodal language models; A system for generating synthetic data; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

Images