Path marker detection and classification for autonomous and semi-autonomous systems and applications

Abstract

The invention relates to path marker detection and classification for autonomous and semi-autonomous systems and applications. In various examples, multi-modal image data may be used to generate a set of top-down tile images that are applied to a deep neural network generator architecture model to produce lane marker-specific heat map images corresponding to the set of top-down tile images. The multi-modal sensor data may include LIDAR captured intensity channel data, LIDAR captured feature height channel data, and optical color image channel data. The generator model may process the set of top-down tile images to automatically detect lane boundaries and navigation boundaries to generate a pixel-level heat map image that may classify lane markings by marker characteristics (e.g., linetype and / or color). The generator model may include an encoder-decoder architecture in which a multi-scale feature extraction and / or context extraction functional layer is interposed between an encoder model and a decoder model.

Description

Background Art

[0001] Advanced driver assistance systems (ADAS) represent an evolving technology in the automotive industry, providing features such as lane centering, lane departure detection, adaptive lane changing, adaptive cruise control, and more. To a large extent, these ADAS features rely on real-time or near real-time lane marking detection and recognition techniques to extract lane features and understand the structural characteristics of lane markings that define lane boundaries. Lane marking detection and recognition also plays an important role in generating high-definition (HD) maps of roads and highways. HD maps can be used for navigation and localization applications, such as by autonomous or semi-autonomous vehicles. However, accurate lane marking recognition remains challenging due to factors such as poor lighting, poor road conditions, lane marking deterioration, road repairs, and / or obstacles that interfere with the capture of lane marking data. Summary of the invention

[0002] Embodiments of the present disclosure relate to point cloud-based heat map generation for path surface lane marking detection and classification. Systems and methods for detecting and classifying lane markings from a road surface for use in map generation and / or for other applications are disclosed.

[0003] Compared to these conventional systems for lane marking detection, one or more embodiments described herein use multimodal sensor data (e.g., a combination of camera and LIDAR (laser radar) data) to generate a set of top-down tile images, and apply the set of top-down tile images to a deep neural network (DNN) generator architecture model to produce lane marking-specific heat map images corresponding to the set of top-down tile images. The multimodal sensor data may include intensity channel data captured by the LIDAR, feature height channel data captured by the LIDAR, and optical color image channel data. The set of top-down tile images can be processed by the generator model to automatically detect lane boundaries and navigation boundaries, thereby generating a pixel-level heat map image that can classify lane markings by marking characteristics (e.g., line type and / or color). BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The following describes in detail a system and method for generating a point cloud-based heat map for detecting and classifying lane markings on a path surface with reference to the accompanying drawings, wherein:

[0005] Figure 1 is a data flow diagram of a path surface lane marking heat map generator system according to some embodiments of the present disclosure;

[0006] Figure 2 is a block diagram illustrating a multimodal tile image according to some embodiments of the present disclosure;

[0007] Figure 3is a diagram illustrating the generation of a plurality of lane marking heat map images from a tile image according to some embodiments of the present disclosure;

[0008] Figure 4 is a diagram illustrating an encoder-decoder architecture for a heatmap generator model according to some embodiments of the present disclosure;

[0009] Figure 5 is a diagram illustrating a generative adversarial network (GAN) training architecture for a heatmap generator model according to some embodiments of the present disclosure;

[0010] Figure 6 is a flow chart of a method for path surface lane marking detection according to some embodiments of the present disclosure;

[0011] Fig. 7A is an illustration of an example autonomous vehicle according to some embodiments of the present disclosure;

[0012] Figure 7B According to some embodiments of the present disclosure Fig. 7A Examples of autonomous vehicle camera positions and fields of view;

[0013] Figure 7C According to some embodiments of the present disclosure Fig. 7A a block diagram of an example system architecture for an example autonomous vehicle;

[0014] Fig.7D is a cloud-based server and Fig. 7A System diagram of communication between example autonomous vehicles;

[0015] Figure 8 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0016] Fig. 9 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Systems and methods for detecting and classifying road markings on a path surface in connection with point cloud-based heat map generation are disclosed. Although the present disclosure may be described with respect to map development, perception, or other aspects of autonomous or semi-autonomous systems and applications, this is not limiting. For example, the systems and methods described herein may be used by a non-autonomous vehicle or machine, a semi-autonomous vehicle or machine (e.g., in one or more advanced driver assistance systems (ADAS)), an autonomous vehicle or machine (e.g., an autonomous or semi-autonomous vehicle or machine 700—referred to herein alternatively as “vehicle 700” or “machine 700”, examples of which may be referred to herein as “the vehicle 700” or “the machine 700”). Figures 7A-7DThe present invention may be used and / or combined with, but is not limited to, a robot or robotic platform, a warehouse vehicle, an off-road vehicle, a vehicle coupled to one or more trailers, a flying boat, a ship, a space shuttle, an emergency response vehicle, a motorcycle, an electric or motorized bicycle, an airplane, an engineering vehicle, a train, an underwater vehicle, a remotely controlled vehicle such as a drone, and / or other vehicle types. In addition, although the present disclosure may be described with respect to path surface road marking detection and classification for autonomous driving, this is not limiting, and the systems and methods described herein may be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technical field in which a map of a path surface may be used.

[0018] The present disclosure relates to generating maps of roads, highways, and / or other pathways (e.g., high-definition (HD) maps, navigation maps, etc.). More specifically, the systems and methods presented in the present disclosure provide techniques for detecting and classifying road markings from road surfaces for use in generating maps and other applications—e.g., for real-time perception, vehicle navigation and control, path planning, etc.

[0019] Conventional techniques for road marking detection may use image segmentation techniques to classify images captured by a camera pixel by pixel to detect road surface landmark types, such as solid lane lines and broken (dashed) lane lines. Example road markings include, but are not limited to, lane markings and navigation boundaries. In the present disclosure, example embodiments and descriptions involving lane markings may be expanded and understood to generally cover road markings. Due to optical phenomena such as perspective distortion, image data from a camera is less accurate for features that are farther from the camera. For example, as features become farther away, the captured features may appear smaller in the image and may therefore be represented by fewer pixels. Therefore, image segmentation becomes increasingly unreliable for classifying lane markings that are farther from the image sensor. For example, as the distance increases, a camera-based lane marking detection system becomes increasingly unable to accurately identify features such as the exact center of a lane marking to determine the relative position of the lane marking relative to other lane markings or other features. Camera-based lane marking detection may also be sensitive to poor ambient lighting conditions, which may produce shadows, glare, and / or other optical artifacts in the image data, thereby blurring or distorting the appearance of lane markings.

[0020] Other existing lane marking detection techniques can use LIDAR-based techniques. Compared to optical camera-based techniques, LIDAR-based lane marking detection is largely unaffected by ambient lighting conditions and can provide accurate measurements of the distance between the LIDAR sensor and the lane markings. However, LIDAR point cloud data becomes increasingly sparse with increasing distance, so the reliability of classifying lane markings decreases as the distance from the LIDAR sensor increases. LIDAR camera fusion technology represents another lane marking detection technique. LIDAR camera fusion captures perspective optical images to detect lane markings and back-projects these detected lane markings onto the data captured by the LIDAR sensor through a transformation matrix, thereby providing greater context information for the detected lane markings. However, due to perspective distortion, the detection of lane markings is less accurate for features that are farther away from the LIDAR and camera sensors. In addition, the back-projection process relies on accurate calibration and alignment between the camera and LIDAR sensors to relate the detected lane markings to the corresponding points of the LIDAR data, which adds another potential source of error.

[0021] Compared to these conventional systems for lane marking detection, one or more embodiments described herein use a combination of camera and LIDAR data to generate a set of top-down tile images (or other top-down or bird's eye view (BEV) representation types) generated by multimodal sensor data, and apply the set of top-down tile images to a deep neural network (DNN) generator architecture model to generate lane marking specific heat map images corresponding to the set of top-down tile images. The multimodal sensor data may include intensity channel data captured by the LIDAR, feature height channel data captured by the LIDAR, and optical color image channel data. The set of top-down tile images can be processed by the generator model to automatically detect road markings (e.g., lane boundaries and / or navigation boundaries) to generate pixel-level heat map images that can classify lane or road boundary markings by marking characteristics (e.g., line type, line color, and / or other characteristics).

[0022] To generate a set of top-down tile images from multimodal sensor data, raw path surface data can be collected using LIDAR and optical image camera sensors mounted on a mobile platform (e.g., a car or other vehicle) traveling along a path (e.g., a road or highway). The mobile platform can travel along the path as the sensor continuously captures LIDAR point cloud data and / or optical image frames of the path surface. The result is an aggregated point cloud of a segment of the path of a defined length (e.g., in kilometers) with corresponding overlapping optical image frames. For example, the LIDAR sensor can be operated at a frequency of ten hertz to generate ten scans of LIDAR point cloud data per second as the platform advances along the path. Frames of optical image data and LIDAR point cloud data can be time-stamped so that the image data can be correlated with corresponding LIDAR data for the path segment. Since the data captured by the LIDAR is aggregated into a composite point cloud as the platform moves (e.g., using self-motion compensation), the points in the point cloud corresponding to the path surface features continue to accumulate and build density over time and as the surface features get closer to the LIDAR sensor. Therefore, the resulting composite point cloud provides a dense data set of LIDAR data covering the entire passage segment. Projection transformation can then be used to project the points of the composite point cloud onto a plane corresponding to the plane of the passage surface. For example, the composite point cloud can include road surface depth data, which can be used to represent the baseline surface of the passage surface (e.g., plane). Projection transformation can be used to generate the orthogonal projection of the points of the composite point cloud on the baseline surface representation. Rotation-translation transformation can be applied to represent the rotation of the composite point cloud relative to the baseline surface, to produce a top-down view of the baseline surface representation of the passage segment seen when the composite point cloud is looked down. The data provided by each point can indicate the intensity and / or depth of the feature on the passage surface that each point is orthogonally projected to. In certain embodiments, the pixel mapping of the optical image frame captured can be similarly used to represent the coordinate system of the baseline surface representation, so as to obtain the combination of intensity, depth and color data of the feature on the passage surface, as observed from the BEV of the passage. A pathway segment covered by a composite point cloud can be subdivided into multiple portions of the pathway to define a series of top-down view tile images, wherein the data provided by each tile image corresponds to a specific surface area of ​​the pathway. Each top-down tile image can be applied to a generator model to generate one or more heat map images indicating lane markings detected within the tile image and one or more classifications based on characteristics of the lane markings, as inferred by the generator model. In some embodiments, the heat map image can be generated to have a dimension and resolution that matches the tile image, such that individual pixels of the heat map image provide data corresponding to individual pixels of the tile image, and therefore, the heat map image pixels provide data corresponding to specific locations on the pathway surface.In some embodiments, navigation data, such as satellite navigation data (e.g., global positioning system (GPS) data) and / or real-time kinematic (RTK) data, can be captured as LIDAR and / or camera image frames are captured and used to correlate and map the locations of tile images and / or heatmap images of the tile images to geographic locations. Thus, in some embodiments, heatmap images can be mapped to segments of real-world pathways to represent the characteristics of lane markings that appear on the surface of those pathway segments.

[0023] As described herein, in some embodiments, for each tile image, the input to the generator model may include intensity data determined by LIDAR, height data determined by LIDAR, and / or color data (e.g., RGB color data) determined by a camera image. The intensity data may indicate the reflectivity of the road surface at each pixel of the tile image. For example, lane markings typically include embedded reflective particles or are more reflective than other unmarked portions of the road surface to make them easier to see. By rotating the composite point cloud to a top-down view and orthogonally projecting its points onto a baseline surface representation of the pathway surface, the depth data collected by the LIDAR sensor may be interpreted as height data indicating the offset of the feature relative to the height of the baseline surface representation. Since lane markings typically include paint, thermoplastics, resins, or other materials applied to the top of the road surface, the surface of the lane markings is higher than the height of other unmarked portions of the road surface, which can be detected from the depth data collected by the LIDAR. Therefore, the height data can be used to detect and classify navigation boundaries. For example, height data can be used to easily distinguish foreground (road marking / lane marking) pixels from background (unmarked road surface) pixels. Navigation boundaries may include features such as, but not limited to, curbs and obstacles. In some embodiments, pixels representing navigation boundaries that do not belong to a known boundary type may be classified as "unknown type" road markings.

[0024] The data obtained from the optical image frame can be used to capture the color of the lane markings. In some embodiments, the camera and LIDAR sensor can be integrated together as a single device that generates a combined output containing intensity, depth and / or color data. In some embodiments, the camera and LIDAR sensor can be implemented separately. One or more sensor calibration parameters (e.g., rotational translation transformations) are used to associate the orthogonal projection of the point cloud points with the pixels of the camera image frame to (e.g., in addition to the intensity and height data) associate the color data with the individual pixels of the tile image. In some embodiments, color data (e.g., RGB color data) can be associated with the points of the LIDAR point cloud based on matching timestamps, instead of or in addition to using sensor (intrinsic and / or extrinsic) calibration parameters. Since the resulting intensity, height and / or color data sets are obtained from the aggregation of overlapping data collected on the mobile platform and are orthogonally projected onto the baseline surface representation of the passage surface, errors that may be caused by perspective distortion, lighting conditions and / or occlusions can be substantially avoided and / or at least reduced.

[0025] In some embodiments, the tile images can be preprocessed into normalized frames in terms of pixel dimensions and data channels before being applied to the generator model. For example, in some embodiments, the height data from a LIDAR sensor can include a color component (e.g., determined based on range and reflectance data), which can be processed into a single height data channel. For example, for 8-bit RGB color, the height data channel for a point of a LIDAR point cloud can be calculated based on the following expression:

[0026]

[0027] Among them, according to LIDAR data, Height Red is the detected red channel height component, Height Green is the detected green channel height component, Height Blue is the detected blue channel height component. In some embodiments, the height data of the entire tile image can be normalized, for example, based on the following expression:

[0028]

[0029] Among them, for this tile image, Height point-min is the minimum measured height, Height point-maxis the maximum measured height. The intensity data and color data can be similarly normalized across the tile image. For example, in some embodiments, each of the three channels of 8-bit RGB intensity data from the LIDAR can be normalized by dividing the data value of each channel by 256 (e.g., so that each intensity channel is scaled to a value from 0 to 1), and each of the three channels of 8-bit RGB color data from the camera image can be normalized by dividing the data value of each channel by 256 (e.g., so that each color channel is scaled to a value from 0 to 1). In one or more such embodiments, the three intensity, one height, and three color channels can therefore be concatenated into an array of seven channels corresponding to the relevant pixels of the tile image.

[0030] In one or more embodiments, where an intensity data channel, a height data channel, and a color data channel are defined for each pixel of the tile image, the tile image with these channels can be applied to the generator model to produce one or more heat map images. In some embodiments, the generator model can output a heat map image for each road marking characteristic classification that the generator model is trained to predict. As a non-limiting example, the generator model can output: a first heat map image, in which the intensity of the pixel represents the confidence of the line characteristic prediction, that is, the pixel is associated with a detected single solid lane marking; a second heat map image, in which the intensity of the pixel represents the confidence of the line characteristic prediction, that is, the pixel is associated with a detected single dashed lane marking; a third heat map image, in which the intensity of the pixel represents the confidence of the line characteristic prediction, that is, the pixel is associated with a detected yellow lane marking; and a fourth heat map image, in which the intensity of the pixel represents the confidence of the line characteristic prediction, that is, the pixel is associated with a detected yellow lane marking. In some embodiments, pixels with a prediction confidence value that at least meets a threshold are defined as lane marking pixels. In some embodiments, for each pixel predicted to define a lane marking pixel, the generator model may also generate an angle value indicating the predicted angle of the lane marking (e.g., relative to the pixel coordinate system of the tile image). The dimensions of the heat map image produced by the generator model may match the dimensions of the input tile image, such that there is a one-to-one correspondence between the value of a pixel at a given column and row coordinate of the heat map image and the value of a pixel with corresponding column and row coordinates of the tile image. In some embodiments, the heat map images generated for different road marking characteristic classifications may be combined into a composite heat map image, thereby providing a comprehensive indication of different linear lane markings detected on the pathway surface covered by the input tile image.

[0031] In some embodiments, the generator model includes a semantic segmentation model based on a deep neural network (DNN) trained using a generative adversarial network (GAN) training method. As discussed in more detail below, the generator model and the discriminator model can participate in a two-play zero-sum task, in which the generator model iteratively learns to generate more accurate heat map images, and the discriminator model iteratively learns to better distinguish the heat map images generated by the generator model from the heat map images annotated with the true value. Training can continue, for example, until the generator model reaches the optimal heat map generation quality threshold and / or until the discriminator model can no longer distinguish the heat map images generated by the generator model from the heat map images annotated with the true value. A set of one or more loss functions (e.g., cross entropy loss, perceptual loss, L1 loss, and / or generative adversarial (GAN) loss) can be calculated in each training iteration, and these loss functions are calculated by the loss optimizer function. During the training iteration, the generator model can be adjusted based on (one or more) loss functions to try to minimize the loss function, while the discriminator model is adjusted to try to maximize (one or more) loss functions.

[0032] For the generator model, the machine learning model may include an encoder-decoder architecture with a multi-scale feature extraction and context extraction functional layer between the encoder model and the decoder model. The encoder model and the decoder model may be implemented using one or more machine learning models, such as a recurrent neural network (RNN) or other form of neural network. The encoder model may operate to compress one or more data channels of an input tile image into an encoded fixed-length representation in the form of an encoder vector (e.g., which may represent a set of random potential embeddings of the tile image). In some embodiments, the encoder model includes a downsampling machine learning encoder, and the decoder model includes an upsampling machine learning decoder for semantic segmentation.

[0033] For example, the encoder model may receive an input tile image having an intensity data channel, a height data channel, and / or a color data channel, and downsample the tile image into layers of different resolutions to extract image features of different resolutions. The multi-scale feature extraction function layer of the generator model receives an encoder vector representing a multi-scale image layer from the encoder model, and performs multi-scale feature extraction and feature fusion to estimate the segmentation of road marking landmarks from multiple resolution layers. For example, the multi-scale feature extraction function may be implemented based on a high-resolution network (e.g., a HRNetV2 network), which may include a general convolutional neural network and is often used in human pose estimation applications.

[0034] In some embodiments, the context extraction function layer of the generator model operates to encode long-range context information as context features. For example, the context extraction function layer can extract context features that indicate the association of non-adjacent pixels in the entire tile image, such as pixels belonging to separate dashed lines of dashed lane markings, pixels of different lines of a two-line lane marker, or pixels of lane markings at opposite ends of the tile image, which are too far apart to be associated by standard convolution techniques. In some embodiments, the context extraction function layer can use an attention mechanism (e.g., a self-attention-based technique) that determines contextual interactions between non-adjacent pixels based on an attention map. In some embodiments, the context detection mechanism is implemented using the Lambda layer of a LambdaNetwork, which captures long-range interactions in the data. More details about the Lambda layer can be found in "LAMBDANETWORKS: MODELING LONG-RANGE INTERACTIONS WITHOUT ATTENTION" by Bello, Irwan, which was published at the 2021 International Conference on Learning Representations, and its contents are incorporated herein by reference in their entirety. In some embodiments, the context extraction function layer may apply an attention mechanism and / or a lambda layer to dynamically allocate context features to produce context features that are output to the decoder model.

[0035] The decoder model can generate an initial prediction based on the lane marking landmark segmentation estimate from the multi-scale feature extraction and the contextual features of the pixels extracted by the context extraction function layer, which includes multiple heat map images, where each heat map image corresponds to a different road marking feature. For each pixel, the prediction can include, for example, a binary classifier for the overall foreground (e.g., lane marking foreground pixels vs. unmarked surface background pixels), a multi-class classifier for foreground pixels for lane type classification (e.g., solid, dashed, single and / or double lane markings), a foreground multi-class classifier for lane color classification, and / or a classifier for lane marking direction (e.g., angle) indication. In some embodiments, the marker direction angle can be predicted by an additional regression layer within the decoder model. In some embodiments, the decoder model can upsample the initial prediction of the heat map image using a bilinear upsampling operation combined with a convolutional layer to match the dimensional size of the input tile image.

[0036] With respect to the training of the generator model, in some embodiments, the discriminator model may include a convolutional classifier, such as a PatchGAN classifier. A PatchGAN classifier is a discriminator that treats the input as a collection of local image patches and penalizes structure at the scale of these local image patches. For example, the discriminator model, when implemented using a PatchGAN classifier, may attempt to classify each N×N pixel local image patch in a heat map image as a real image (e.g., a true value heat map image), or a predicted heat map image generated by a generator model.

[0037] In some embodiments, data augmentation may be used to enhance the training data, including random scaling, random rotation, random color jitter, and / or random cropping. The dimensions of the training data input to the generator model during training may be different from the dimensions of the tile images input to the generator model for predicting the heat map image. For example, a cropped training data array of 768x 768 pixels may be used for training to generate a generator model that can predict from 1024x 1024, 2048x 2048, and / or rectangular (e.g., 2816x 2304) tile images. In some embodiments, the discriminator model may include one or more downsampling blocks (e.g., four downsampling block discriminator models), wherein a single downsampling block may include a convolutional layer, followed by a spectral normalization layer, an instance normalization layer, a leaky rectified linear unit (ReLU) layer, and / or an average pooling layer. In some embodiments, the first downsampling block may not include a spectral normalization layer. In some embodiments, the last downsampling block may not include an average pooling layer. The output of one or more downsampling blocks may be processed by a convolutional layer (eg, a 1x1 convolutional layer) to produce an output of a predetermined dimension.

[0038] For example, in some embodiments, the first input of the discriminator model may include a heat map image of size 768x 768x 1 generated by the generator model, where the third dimension of size "1" is a data channel that can convey the confidence that the pixel is a lane marker of the line feature of the heat map image (e.g., indicating whether the pixel is predicted to be a foreground pixel). The second input of the discriminator model may include a true value heat map image of size 768x 768x 1 corresponding to the line feature of the input heat map image of the first input. These inputs can be processed by one or more downsampling blocks of the discriminator model and applied to the convolution layer to produce an output data array of 96x 96x 1 dimensions. The discriminator model can evaluate the output data array to predict which of the two inputs was generated by the generator model. The loss optimizer function can calculate one or more loss functions to generate one or more losses based on whether the prediction of the discriminator model is correct. Using one or more losses as feedback, the generator model iteratively learns to generate more accurate heat map images, and the discriminator model iteratively learns to better distinguish the heat map images generated by the generator model from the heat map images annotated with the true value. In some embodiments, one or more loss functions may calculate one or more of the cross entropy loss, the perceptual loss, the L1 loss, and / or the generative adversarial (GAN) loss. For example, the loss function loss can quantify the representation of the distance and / or difference between the feature embedding of the heat map generated by the generator and the feature embedding of the true value heat map image. In some embodiments, the L1 loss can be used to adjust the lane marking angle prediction. In one or more embodiments, the perceptual loss can be applied to the L1 loss norm between the feature extraction of the penalty prediction and the true value feature extraction. For the cross entropy loss, a smaller weight (e.g., 0.01) can be assigned to background pixels (e.g., pixels of unmarked road surface) to compensate for data imbalance. The GAN loss may include a loss for training one of the generator model and the discriminator model, and in some embodiments may be implemented as a least squares GAN loss. In some embodiments, one or more of these losses may be calculated independently and fed back as a composite loss to train the generator model and / or the discriminator model.

[0039] In one or more embodiments, the training data may be preprocessed to address the problem of missing true value labels that may adversely affect recall. For example, sometimes the training data may contain valid images in which one or more lane markings are missing labels. When such training images are input to the generator model during training, the generator model may learn incorrect reasoning due to the ambiguity of having two substantially similar but not similarly labeled lane marking features. Therefore, in some embodiments, a training mask can be generated for each training image, which includes the labeled lane marking features on which the generator model is to be trained within the mask and excludes the unlabeled lane marking features that the generator model is to ignore during training from the mask. For example, a training mask can be created for a cross entropy loss using the true value labels and the height image derived from LIDAR. The training data preprocessor can apply a binary expansion method to expand the labeled true value heat map and create a first mask. Based on defining pixels with a height of zero as not belonging to the path, the preprocessor uses this information to create a second pixel mask belonging to the path. The first mask and the second mask are added to generate the final training mask. During training with the cross-entropy loss, backpropagation is limited to pixels that lie within this training mask.

[0040] In some embodiments, the heat map images generated by the generator model can be used to generate high-definition (HD) maps of roads and highways, which can be used for navigation and positioning applications of autonomous or semi-autonomous vehicles and machines. For example, a set of heat map images generated from tile images can be used to define a traffic control map that indicates the location and characteristics of lane markings for a passage segment corresponding to the tile image. A set of traffic control maps obtained in this manner can be sequentially linked to generate a comprehensive HD map of the passage, which can be used, for example, to plan vehicle routes and / or control real-time or near-real-time navigation (e.g., positioning) of vehicles traveling on the passage represented by the HD map. In some embodiments, vector-based lane lines can be generated for the HD map based on lane marking predictions provided by the heat map images. For example, vector-based lane lines can be generated that link lane marking predictions for one tile image with lane marking predictions of the same line type for adjacent tile images. In some embodiments, vectorization based on heat map images can be used to generate a graphical skeleton of lane markings, wherein the lines of the graphical skeleton can be projected onto the pixel positions of the tile image to align the lane markings of the tile image and / or generate an HD map. In various embodiments, the HD map can be used to generate simulated environments, such as for gaming, vehicle traffic simulation, and / or autonomous or semi-autonomous vehicle training. In one or more embodiments, the heat map images generated from the tile images can be used as ground truth training data for training other machine learning models.

[0041] As described in more detail herein, one or more aspects of the generator model and / or the discriminator model may be executed at least in part on one or more graphics processing units, which may operate in conjunction with software executed on a central processing unit coupled to a memory. The graphics processing unit is programmed to execute a kernel to implement one or more features and functions of the generator model and / or the discriminator model. In some embodiments, aspects of the generator model and / or the discriminator model may be executed in parallel on different GPUs and / or other parallel processing units (PPUs). In some embodiments, some features and functions of the generator model and / or the discriminator model may be distributed and executed by a combination of processors and cloud computing resources. For example, in some embodiments, the generator model and / or the discriminator model may be implemented at least in part as a virtual function in a cloud computing environment and / or as a component of a virtualized machine learning model.

[0042] refer to Figure 1 , Figure 1 is a data flow diagram of an example path surface lane marking detection system 100 according to some embodiments of the present disclosure. It should be understood that this arrangement and other arrangements described herein are presented only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may also be used in addition to or in place of the arrangements and elements shown, and some elements may be omitted entirely. In addition, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components, and may be implemented in any suitable combination and location. The various functions performed by the entities described herein may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in a memory. In some embodiments, the systems, methods, and processes described herein may be used in conjunction with Figures 7A-7D Example of autonomous vehicle 700, Figure 8 The example computing device 800 and / or Fig. 9 The example data center 900 may be implemented with similar components, features and / or functions.

[0043] like Figure 1As shown, the path surface lane marking detection system 100 may include a path surface feature processor 105, which receives composite surface point cloud data 107 captured from the path surface by one or more point cloud sensors 106 and / or surface color data 111 captured from the path surface by one or more optical image sensors 110. The point cloud sensor 106 may include a RADAR sensor, a LIDAR sensor, and / or other sensors that generate depth data in the form of a point cloud and / or in other forms. Therefore, even if it is referred to as a point cloud sensor 106, this is not intended to be limiting, and in some embodiments, the point cloud sensor 106 may generate an output representation different from a point cloud. The surface color data 111 can be derived from optical image data captured by one or more optical image sensors 110, and the optical image sensor 110 can be positioned on a mobile platform (e.g., vehicle 700). The optical image sensor 110 may include, for example, a camera or other optical sensor that captures RGB, IR, and / or RGB-IR image frames (as non-limiting examples). In some embodiments, the point cloud sensor 106 and / or the optical image sensor 110 may include a sensor and / or a camera, such as Fig. 7A and / or Figure 7B In some embodiments, the composite surface point cloud data 107 may include point cloud data including a fusion of images from multiple point cloud sensors 106. In some embodiments, the surface color data 111 may include image data including a fusion of images from multiple optical image sensors 110. Based on the composite surface point cloud data 107 and the surface color data 111, the path surface feature processor 105 generates one or more lane marking heat map images 122 using a heat map generator model 120.

[0044] In one or more embodiments, the path surface feature processor 105 generates a set of top-down view tile images 116 from the composite surface point cloud data 107 and the surface color data 111. As described above, the mobile platform can move along the path while the point cloud sensor 106 and the optical image sensor 110 capture the point cloud data and / or optical image frames of the path surface. The points of the composite point cloud data 107 may include depth data 108 and / or surface intensity data 109. The depth data 108 may include depth information of the path surface area where lane markings have been applied and the unmarked area of ​​the path surface. Since lane markings typically include paint, thermoplastics, resins, or other materials applied on top of the road surface, the surface of the lane markings may be higher in height than the unmarked sections of the path surface. The height data of the lane markings can be detected based on the relative difference between the depth data of the marked sections and the unmarked sections of the path surface. The height data can be used by the heat map generator model 120 to detect and classify navigation boundaries and can be used to easily distinguish foreground (lane marking) pixels from background (unmarked road surface) pixels. The surface intensity data 109 may indicate the reflectivity of different areas of the roadway. For example, lane markings often include embedded reflective particles or are more reflective than other unmarked sections of the road surface to make them easier to see.

[0045] In some embodiments, the image frames of the surface color data 111 and the composite point cloud data 107 may be time stamped or otherwise marked so that the color data may be associated with the corresponding point cloud data for the pathway segment. As the point cloud sensor 106 moves, the data captured by the LIDAR is aggregated into a composite point cloud such that points in the composite point cloud data 107 corresponding to pathway surface features accumulate and build density over time and as the surface features get closer to the point cloud sensor 106. Thus, the resulting composite point cloud data 107 provides a dense dataset of depth and intensity data that may be combined with the surface color data 111 to produce aggregated pathway surface data 112 covering the pathway segment.

[0046] The path surface feature processor 105 may then apply one or more projection transformations 114 to project the resulting aggregated path surface data 112 onto a plane corresponding to the path surface. For example, in some embodiments, the composite point cloud data 107 may include road surface depth data that may be used by the path surface feature processor 105 to define a baseline surface representation 113 (e.g., a plane) of the path surface. The projection transformation 114 may generate an orthogonal projection of the points and / or color data of the aggregated path surface data 112 onto the baseline surface representation 113. In some embodiments, the projection transformation 114 may apply a rotational translation transformation to rotate the composite point cloud data 107 and / or surface color data 111 relative to the baseline surface representation to generate a top-down view of the baseline surface representation 113 of the path segment as seen when looking down through the composite point cloud data 107. The aggregated path surface data 112 may be subdivided into portions of the path to define a sequence of top-down view tile images 116, wherein the data channel provided by each tile image corresponds to a specific surface area of ​​the path.

[0047] For example, now refer to Figure 2 , Figure 2 A section 214 of a passage 210 is shown. For example, the passage 210 may include a section of a road or highway, for example, one kilometer or more in length. The passage 210 includes a surface 212, which includes lane markings 208 and unmarked surface areas 209. In some embodiments, aggregated path surface data 112 may be collected by the point cloud sensor 106 and / or the optical image sensor 110 as they move along the section 214 of the passage 210. The point cloud sensor 106 and / or the optical image sensor 110 may scan the surface 212 at a certain frequency so that multiple overlapping scans are obtained on the section as the platform with the sensor moves. The points of the aggregated path surface data 112 corresponding to the features of the passage surface 212 accumulate and build up density over time and as the surface features get closer to the sensor. The collected data may then be rotated and / or orthogonally projected onto a baseline surface representation 113, which represents the baseline plane of the surface 212 of the passage 210. In some embodiments, the path surface feature processor 105 may generate a tile image 116 by subdividing the segment 214 into subdivisions 216, each subdivision corresponding to a length of the passage 210. That is, the tile image 116 may include a data channel including point clouds and color data on the baseline surface representation 113 projected by the projection transformation 114 into the boundaries of the subdivision 216 of the segment 214. Figure 2As shown, each tile image 116 may include one or more of a surface intensity channel 220, a surface height channel 222, and a surface color channel 224. Thus, the data provided by each pixel channel of the tile image 116 may indicate the intensity, height, and / or color of features on the surface of the pathway onto which each point of the aggregated path surface data 112 is orthogonally projected. Thus, each tile image 116 may include a combination of intensity, depth, and / or color data features on the surface of the pathway 210, as observed from a bird's-eye view of the pathway 210.

[0048] Each top-down tile image 116 may be applied to a heat map generator model 120 to generate one or more lane marking heat map images 122 indicating lane markings detected within the tile image 116. The lane marking heat map images 122 may indicate one or more classifications based on the characteristics of the lane markings inferred by the heat map generator model 120. For example, Figure 3 As shown, a tile image 116 including a surface intensity channel 220 (which may be represented as an intensity image 310), a surface height channel 222 (which may be represented as a height image 320), and a surface color channel 224 (which may be represented as a color image 330) may be provided as an input to a heat map generator model 120. Based on the tile image 116, the heat map generator model 120 may infer characteristics of lane marking features from the tile image 116 and generate predictions including one or more lane marking heat map images 122. In some embodiments, the lane marking heat map images 122 may be generated to have dimensions and resolutions that match the dimensions and resolutions of the tile image 116, such that each pixel of the lane marking heat map image 122 provides data corresponding to each pixel of the tile image 116, and thus, each pixel of the heat map image provides data corresponding to a particular location on the surface 212 of the pathway 210.

[0049] In some embodiments, the generator model may output a prediction comprising a set of heat map images, where each heat map image corresponds to a classification of road marking characteristics that the heat map generator model 120 is trained to predict. As an example, the prediction may include a binary classifier for the overall foreground heat map 342 (e.g., lane marking foreground pixels versus unmarked surface background pixels). The prediction may include a multi-class classifier for foreground pixels (e.g., identified by the overall foreground heat map 342) for lane type classification, such as, but not limited to, a double solid lane marking heat map 344, a long dashed (dashed) lane marking heat map 346, a single solid lane marking heat map 348. The prediction may include a multi-class classifier for foreground pixels for lane marking color classification, such as, but not limited to, a white lane marking heat map 350 and a yellow lane marking heat map 352. In some cases, the heat map generator model 120 may identify features extracted from the tile image 116 as having characteristics of a lane marking, but inconsistent with the learned lane characteristic classification. In this case, the predictions from heat map generator model 120 may include an undefined lane marking heat map 354 that indicates pixels where undefined lane markings are located.

[0050] In some embodiments, the heat map generator model 120 can calculate a confidence value indicating the confidence of the predicted classification of the pixel. Those pixels having a confidence value that at least meets a threshold are defined as lane marking pixels within a particular classification associated with the heat map image. In some embodiments, for one or more pixels of the lane marking heat map image 122 predicted as defining a lane marking pixel, the generator model can further generate an angle value indicating the predicted angle of the lane marking (e.g., relative to the pixel coordinate system of the tile image).

[0051] The dimensions of the lane marking heat map image 122 generated by the heat map generator model 120 can match the dimensions of the input tile image 116, so that there is a one-to-one correspondence between the value of a pixel at a given column and row coordinate of the heat map image 122 and the value of a pixel having the corresponding column and row coordinate of the tile image 116. In some embodiments, the lane marking heat map images 122 generated for different road marking characteristic classifications can be combined into a composite heat map image to provide a more comprehensive indication of the different line characteristics (e.g., type and / or color) of lane markings detected on the pathway surface covered by the input tile image. As an example, the location of a yellow dashed lane marker within the tile image 116 can be determined based on identifying pixels of lane markers that appear in the long dashed (broken line) lane marker heat map 346 and the yellow lane marker heat map 352.

[0052] In some embodiments, navigation data may be captured when the point cloud and / or camera image frames are captured, such as satellite navigation data (e.g., global positioning system (GPS) data) and / or real-time kinematic (RTK) data, and used to associate and map the location of the tile image 116 and / or the heat map image 122 of the tile image 116 to the geographic location of the corresponding sub-portion 216 of the pathway segment 214. Thus, in some embodiments, the lane marking heat map image 122 may be mapped to a segment of the real-world pathway 210 to represent the characteristics of the lane markings present on the surface 212 of the pathway segment 214.

[0053] Regarding the architecture of the heat map generator model 120, the heat map generator model 120 may include an encoder-decoder architecture machine learning model. Figure 4 As shown, the heat map generator model 120 may include a downsampling encoder model 410 and an upsampling decoder model 450, wherein a multi-scale feature extraction function layer 420 and a context extraction function layer 430 are interposed between the encoder model 410 and the decoder model 450. The encoder model 410 and the decoder model 450 may be implemented using one or more machine learning models, such as a recurrent neural network (RNN) or other forms of neural networks. The encoder model 410 may be used to compress one or more data channels (e.g., a surface intensity channel 220, a surface height channel 222, and / or a surface color channel 224) of the input tile image 116 into an encoded fixed-length representation in the form of an encoder vector (e.g., which may represent a set of random potential embeddings of the tile image 116). In some embodiments, the encoder model 410 includes a downsampling machine learning encoder, and the decoder model 450 includes an upsampling machine learning decoder for semantic segmentation.

[0054] In some embodiments, the encoder model 410 can receive the input tile image 116 and downsample the tile image to layers of different resolutions to extract image features of different resolutions. The multi-scale feature extraction function layer 420 receives the encoder vector representing the multi-scale image layer from the encoder model 410 and performs multi-scale feature extraction and feature fusion to estimate the segmentation of road marking landmarks from multiple resolution layers. For example, the multi-scale feature extraction function layer 420 can be implemented based on a high-resolution network (e.g., a HRNetV2 network), which can include a general convolutional neural network commonly used for human pose estimation applications.

[0055] In some embodiments, the context extraction function layer 430 is used to encode long-range context information as context features. For example, the context extraction function layer 430 can extract context features that indicate the association of non-adjacent pixels across the entire tile image 116, such as pixels belonging to separate dashed lines of dashed lane markings, pixels of different lines of a two-line lane marker, or pixels of lane markings at opposite ends of the tile image 116. These features may be too far apart to be associated by standard convolution techniques. In some embodiments, the context extraction function layer 430 can use an attention mechanism (e.g., a self-attention-based technique) that determines contextual interactions between non-adjacent pixels based on an attention map. In some embodiments, the context detection mechanism is implemented using a Lambda layer of a LambdaNetwork that captures long-range interactions in the data. More details about Lambda layers can be found in Bello, Irwan, "LAMBDANETWORKS: MODELING LONG-RANGE INTERACTIONS WITHOUT ATTENTION", presented at the 2021 International Conference on Learning Representations, the contents of which are incorporated herein by reference in their entirety. In some embodiments, the context extraction function layer 420 can apply an attention mechanism and / or a lambda layer to dynamically allocate context features to produce context features that are output to the decoder model 450.

[0056] The decoder model 450 may generate a downsampled prediction based on the lane marking landmark segmentation estimates from the multi-scale feature extraction functional layer 420 and the contextual features of the pixels extracted by the context extraction functional layer 430, which includes a plurality of lane marking heat map images 122, wherein each heat map image corresponds to one or more different road marking characteristics, as described herein. For each pixel, the prediction 460 may include the heat map image 122, such as, but not limited to, a binary classifier for the overall foreground (e.g., lane marking foreground pixels vs. unmarked surface background pixels), a multi-class classifier for foreground pixels for lane type classification (e.g., solid, dashed, single and / or double lane markings), a foreground multi-class classifier for lane color classification, and / or a classifier for lane marking direction (e.g., angle) indication. In some embodiments, the marker direction angle may be predicted by an additional regression layer within the decoder model 450. In some embodiments, the decoder model 450 may upsample the downsampled predictions of the heat map image 122 to produce predictions 460 of the lane marking heat map image 122 that match the dimensional size of the input tile image 116 , such as using a bilinear upsampling operation in combination with a convolutional layer.

[0057] Reference now Figure 5 , Figure 5 is a diagram illustrating a generative adversarial network (GAN) training architecture 500 for the heatmap generator model 120 according to some embodiments of the present disclosure. Figure 5 As shown, the heatmap generator model 120 and the discriminator model 540 can engage in a double-zero-sum task, where the heatmap generator model 120 iteratively learns to generate more accurate heatmap image predictions 530 (e.g., using the input training data tile images 518), and the discriminator model 540 iteratively learns to better distinguish the heatmap images 530 produced by the generator model from the ground-truth labeled training data images 510. Training of the heatmap generator model 120 can continue, for example, until the heatmap generator model 120 achieves an optimal heatmap generation quality threshold, and / or the discriminator model 540 can no longer distinguish between the heatmap images produced by the generator model 120 and the ground-truth labeled heatmap images.

[0058] like Figure 5 As shown, during training, the heat map generator model 120 inputs a training data temporal image 505 (e.g., from ground truth labeled training data 510). As previously described with respect to the tile image 116, the training data temporal image 505 here may include a surface intensity channel 506 (e.g., as described with respect to the surface intensity channel 220), a surface intensity channel 507 (e.g., as described with respect to the surface height channel 222), and / or a surface intensity channel 508 (e.g., as described with respect to the surface color channel 224). A training iteration of the heat map generator model 120 may include processing the training data temporal image 505 to generate one or more predicted heat maps 520 (e.g., as described with respect to the lane marking heat map image 122). The discriminator model 540 evaluates one or more predicted heat maps 520 against the ground truth labeled training data 510 and attempts to predict which one was generated by the heat map generator model 120. The prediction accuracy of the discriminator model 540 may be represented by a prediction score 542 , which is used by the loss optimizer 500 to adjust the heatmap generator model 120 and / or the discriminator model.

[0059] In some embodiments, the discriminator model 540 may include a convolutional classifier, such as a PatchGAN classifier. A PatchGAN classifier is a type of discriminator that treats the input as a collection of local image patches and penalizes structure at the scale of these local image patches. For example, a discriminator model 540 implemented using a PatchGAN classifier may attempt to classify one or more N×N pixel local image patches of a heat map image as a true value image (e.g., an image derived from the training data 510 labeled with true value) or a predicted heat map image 530 generated by the heat map generator model 120.

[0060] In each training iteration performed using the GAN training architecture 500, the loss optimizer 550 may, for example, calculate a set of one or more loss functions based on the prediction scores 542. The loss optimizer 550 may calculate one or more loss functions, such as, but not limited to, cross entropy loss, perceptual loss, L1 loss, and / or generative adversarial (GAN) loss. The loss optimizer 550 may generate one or more loss feedback signals based on the one or more loss functions to adjust the heat map generator model 120 and / or the discriminator model 540. During the training iterations performed by the GAN training architecture 500, the heat map generator model 120 may be adjusted based on the loss feedback signal to attempt to minimize (one or more) loss functions. The discriminator model 540 may be adjusted based on (one or more) loss feedback signals to attempt to maximize (one or more) loss functions.

[0061] In some embodiments, data augmentation may be used to enhance the training data 510 of the ground truth annotations, including random scaling, random rotation, random color jittering, and / or random cropping. As described herein, the dimensions of the images of the training data input to the heat map generator model 120 during training using the GAN training architecture 500 may be different from the dimensions of the tile images input to the heat map generator model 120 at runtime to predict the lane marking heat map image 122. In some embodiments, the discriminator model 540 may include one or more downsampling blocks (e.g., four downsampling block discriminator models), wherein a single downsampling block may include a convolutional layer, followed by a spectral normalization layer, an instance normalization layer, a leaky rectified linear unit (ReLU) layer, and / or an average pooling layer. In some embodiments, the first downsampling block may not include a spectral normalization layer. In some embodiments, the last downsampling block may not include an average pooling layer. The output of one or more downsampling blocks may be processed by a convolutional layer (e.g., a 1x 1 convolutional layer) to produce an output of a predetermined dimension.

[0062] For example, in some embodiments, the first input of the discriminator model 540 may include a predicted heat map 530 of size 768x 768x 1 generated by the generator model 120. The third dimension of size "1" may include a data channel for conveying the confidence of the pixel as a lane marker of the line feature of the heat map image (e.g., indicating whether the pixel is predicted as a foreground pixel). The second input of the discriminator model 540 may include a true value heat map image of size 768x768x 1 corresponding to the line feature of the input heat map image of the first input. These inputs can be processed by one or more downsampling blocks of the discriminator model and applied to the convolution layer to produce an output data array of 96x 96x 1 dimensions. The discriminator model 540 can evaluate the output data array to predict which of the two inputs was generated by the generator model 120. The loss optimizer 550 can calculate one or more loss functions based on whether the prediction of the discriminator model 540 is correct. Using one or more losses as feedback, the heatmap generator model 120 iteratively learns to generate more accurate heatmap images, and the discriminator model 540 iteratively learns to better distinguish the heatmap images produced by the generator model 120 from the heatmap images with true value annotations.

[0063] In one or more embodiments, the training data may be preprocessed by a training data preprocessor 520. For example, sometimes the training data may include valid images in which one or more true value labels of lane markings are missing. Therefore, in some embodiments, the training data preprocessor 520 may generate and apply a training mask 522, which is used with the training images for training the heat map generator model 120. The training mask 522 may include the labeled lane marking features on which the generator model 120 is to be trained, and exclude the unlabeled lane marking features that the generator model 120 is to ignore during training from the mask. As an example, in some embodiments, a first mask may be created for the cross entropy loss using the true value labels and the height image derived from LIDAR. The training data preprocessor 520 may apply a binary expansion method to expand the labeled true value heat map and create a first mask. Based on defining pixels with a height of zero as not belonging to the path, the preprocessor 520 uses this information to create a second pixel mask belonging to the path. The first mask and the second mask are added to generate the final training mask 522. During training with the cross entropy loss, backpropagation is limited to pixels that lie within this training mask 522.

[0064] Reference now Figure 6 , Figure 6 6 is a flow chart showing a method 600 for detecting lane markings on a path surface according to some embodiments of the present disclosure. Figure 6The features and elements described in method 600 may be used in conjunction with, in combination with, or in place of elements of any other embodiment discussed herein, and vice versa. In addition, it should be understood that Figure 6 The functional, structural, and other descriptions of the elements of the embodiments described in the accompanying drawings may apply to the same or similarly named or described elements in any figures and / or embodiments described herein, and vice versa.

[0065] Each block of the method 600 described herein includes a computing process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The methods can also be embodied as computer usable instructions stored on a computer storage medium. The methods can be provided by a stand-alone application, a service or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. In addition, the method 600 is directed to Figure 1 The path surface lane marking detection system 100 is described by way of example. However, the methods may additionally or alternatively be performed by any one system or any combination of systems, including but not limited to the systems described herein.

[0066] As discussed in more detail herein, the method may include generating at least one heat map image for at least one road marking characteristic (e.g., a lane marking line characteristic) based on applying at least one tile image of a path surface to a generator model to perform semantic segmentation to classify pixels of at least one tile image representing a lane marking, wherein the at least one tile image of the path surface is based on an orthogonal projection of aggregated path surface data onto a baseline surface representation of the path surface, wherein the aggregated path surface data includes at least one of LIDAR intensity point cloud data, LIDAR depth point cloud data, and path surface color data.

[0067] Method 600 includes at box B602: generating at least one tile image of the path surface based on at least orthogonal projections of aggregated path surface data on a baseline surface representation of the path surface, the aggregated path surface data being obtained based on at least sensor data associated with the path surface. In some embodiments, the path surface feature processor 105 may project the points of the composite point cloud onto a plane corresponding to the plane of the passage surface using a projection transformation 114. For example, the composite point cloud may include road surface depth data, which may be used to establish a baseline surface representation (e.g., a plane) of the passage surface. The projection transformation 114 may be used to generate orthogonal projections of the points of the composite point cloud on the baseline surface representation. The projection transformation 114 may include a rotational translation transformation to rotate the composite point cloud relative to the baseline surface representation to generate a top-down view of the baseline surface representation of the passage segment seen when looking down through the composite point cloud. The data provided by each point may indicate the intensity and / or depth of features on the passage surface to which each point is orthogonally projected. In some embodiments, the projective transformation 114 can be similarly used to map pixels of the captured optical image frame to the coordinate system of the baseline surface representation in order to obtain a combination of intensity data, depth data, and color data for features on the pathway surface as observed from a bird's-eye view of the pathway. The pathway segment covered by the composite point cloud can be subdivided into portions of the pathway to define a series of top-down view tile images, where the data provided by each tile image corresponds to a particular surface area of ​​the pathway.

[0068] Aggregated path surface data of at least one section of the path surface can be received based at least on sensor data depicting the path surface. As described herein, when one or more point cloud sensors and / or one or more optical image sensors capture point cloud data and / or optical image frames of the path surface, the mobile platform can travel along the path. Point cloud data can include depth data and / or surface intensity data. Depth data can include depth information of the path surface area where lane markings have been applied and the unmarked area of ​​the path surface. The height data of the lane marking can be detected based on the relative difference between the depth data of the marked section and the unmarked section of the path surface. The height data can be used by the heat map generator model to detect and classify navigation boundaries, and can be used to easily distinguish foreground (lane and / or road marking) pixels from background (unmarked road surface) pixels. Surface intensity data can indicate the reflectivity of different areas of the path. For example, road markings usually include embedded reflective particles, or are more reflective than other unmarked sections of the road surface to make them more easily visible.

[0069] In some embodiments, the image frames of the surface color data and the composite point cloud data can be time stamped or otherwise marked so that the color data can be associated with the corresponding point cloud data of the passage segment. When the point cloud sensor moves, the data captured by the LIDAR is aggregated into a composite point cloud so that the points of the composite point cloud data corresponding to the features of the passage surface accumulate and build density over time and as the surface features get closer and closer to (one or more) point cloud sensors. The resulting composite point cloud data provides a dense data set of depth and intensity data, which can be combined with the surface color data to generate aggregated path surface data covering the passage segment. Therefore, the aggregated path surface data can include at least one of the LIDAR intensity point cloud data, the LIDAR depth point cloud data, and the path surface color data. The aggregated path surface data can include at least a composite point cloud of LIDAR data captured from at least one sensor on a mobile platform at multiple locations along the segment of the path surface. In some embodiments, the aggregated path surface data also includes position data (e.g., navigation data), which indicates the location where the sensor data is captured. Navigation data, such as satellite navigation data (e.g., global positioning system (GPS) data) and / or real-time kinematic (RTK) data, can be captured when the LIDAR and / or camera image frames are captured and used to associate and map the location of the tile image and / or the heat map image of the tile image to a geographic location. Thus, in some embodiments, the heat map image can be mapped to segments of a real-world pathway to represent the characteristics of road markings that appear on the surface of those pathway segments.

[0070] The method 600 includes, at block B604, applying the at least one tile image to a generator model to perform semantic segmentation on the at least one tile image, the semantic segmentation classifying one or more pixels of the at least one tile image based at least on one or more road marking characteristics. In some embodiments, the generator model may include an encoder model and a decoder model, e.g. Figure 4The model shown in . The encoder model of the generator model can be used to downsample at least one tile image into a multi-scale image representation including multiple resolution layers. In some embodiments, the multi-scale feature extraction function layer of the generator model can perform one or more multi-scale feature extractions to estimate one or more segmentations of road marking landmarks from multiple resolution layers. The decoder model of the generator model can be used to upsample one or more segmentations of road marking landmarks to generate at least one heat map image. In some embodiments, the generator model may include a context extraction function layer of the generator model, which can be used to extract one or more context features, which indicate at least one context association between one or more segmentations of road marking landmarks. The context extraction function layer can use at least one attention mechanism based on one or more attention maps to determine at least one context association and / or use one or more Lambda layers of LambdaNetwork to determine at least one context association. In some embodiments, the generator model can be trained using feedback including at least one loss determined by a discriminator model based on a generative adversarial network (GAN) training architecture. At least one loss can be calculated based on a first input including at least one heat map image and a second input including at least one true value labeled heat map image. In some embodiments, the method may include generating at least one training mask based at least on at least one ground-truth annotated heatmap image, and training a generator model based at least on road marking features of training data images located within the at least one training mask.

[0071] Method 600 includes at box B606: generating at least one heat map image based on at least one or more pixels of at least one tile image being classified as representing one or more road marking characteristics. One or more pixels of at least one heat map image can represent the road marking position of the road marking characteristics, and the road marking characteristics include but are not limited to at least one of the following: solid lane markings, dashed lane markings, white lane markings, or yellow lane markings. In some embodiments, at least one heat map image may include an indication of the line marking angle of at least one lane marking. The method may include generating a map of the path surface based at least on at least one heat map image.

[0072] The systems and methods described herein may be used by and / or in conjunction with non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, boats, space shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, airplanes, engineering vehicles, trains, underwater vehicles, remotely controlled vehicles (e.g., drones), and / or other vehicle types, but are not limited to such. In addition, the systems and methods described herein may be used for a variety of purposes, such as, but not limited to, for machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or participant simulation and / or digital twins, data center processing, conversational AI (artificial intelligence), light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, and / or any other suitable application.

[0073] The disclosed embodiments may be included in a variety of different systems, such as an automotive system (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), a system implemented using a robot, an aviation system, a medical system, a navigation system, an intelligent area monitoring system, a system performing deep learning operations, a system performing simulation operations, a system for performing digital twin operations, a system implemented using an edge device, a system including one or more virtual machines (VMs), a system for performing generative AI operations using a language model (e.g., a large language model (LLM), a system for performing synthetic data generation operations, a system implemented at least in part in a data center, a system performing conversational AI operations, a system performing light transport simulation, a system performing collaborative content creation of 3D assets, a system implemented at least in part using cloud computing resources, and / or other types of systems.

[0074] Example Autonomous Vehicle

[0075] Fig. 7A7 is an illustration of an example autonomous vehicle 700 according to some embodiments of the present disclosure. Autonomous vehicle 700 (alternatively referred to herein as "vehicle 700") may include, but is not limited to, a passenger vehicle such as a car, a truck, a bus, an emergency vehicle, a shuttle, an electric or motorized bicycle, a motorcycle, a fire truck, a police car, an ambulance, a boat, an engineering vehicle, an underwater vessel, a robotic vehicle, a drone, an airplane, a vehicle coupled to a trailer (e.g., a semi-tractor semi-trailer truck for hauling freight), and / or other types of vehicles (e.g., unmanned and / or capable of accommodating one or more passengers). Autonomous vehicles are generally described in terms of automation levels defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (Standard No. J3016-201806, issued on June 15, 2018, Standard No. J3016-201609, issued on September 30, 2016, and previous and future versions of the standard). The vehicle 700 is capable of implementing one or more of the functions of Level 3-5 that meet the autonomous driving level. The vehicle 700 is capable of implementing one or more of the functions of Level 1-5 that meet the autonomous driving level. For example, depending on the embodiment, the vehicle 700 is capable of driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). The term "autonomy" as used herein may include any and / or all types of autonomy of 700 or other machines, such as full autonomy, high autonomy, conditional autonomy, partial autonomy, assisted autonomy, semi-autonomy, primary autonomy, or other names.

[0076] The vehicle 700 may include components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of the vehicle. The vehicle 700 may include a propulsion system 750, such as an internal combustion engine, a hybrid power plant, an all-electric engine, and / or another type of propulsion system. The propulsion system 750 may be connected to a drive train of the vehicle 700, which may include a transmission, to achieve propulsion of the vehicle 700. The propulsion system 750 may be controlled in response to receiving a signal from a throttle / accelerator 752.

[0077] A steering system 754, which may include a steering wheel, may be used to steer the vehicle 700 (e.g., along a desired path or route) when the propulsion system 750 is operating (e.g., when the vehicle is in motion). The steering system 754 may receive signals from a steering actuator 756. For fully automated (Level 5) functionality, a steering wheel may be optional.

[0078] Brake sensor system 746 may be used to operate vehicle brakes in response to receiving signals from brake actuator 748 and / or brake sensors.

[0079] May include one or more system on chip (SoC) 704 ( Figure 7C ) and / or one or more controllers 736 of one or more GPUs can provide signals (e.g., representing commands) to one or more components and / or systems of the vehicle 700. For example, the one or more controllers can send signals to operate vehicle brakes via one or more brake actuators 748, operate the steering system 754 via one or more steering actuators 756, and operate the propulsion system 750 via one or more throttles / accelerators 752. The one or more controllers 736 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to achieve autonomous driving and / or assist a human driver in driving the vehicle 700. The one or more controllers 736 may include a first controller 736 for autonomous driving functions, a second controller 736 for functional safety functions, a third controller 736 for artificial intelligence functions (e.g., computer vision), a fourth controller 736 for infotainment functions, a fifth controller 736 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 736 may handle two or more of the above functions, two or more controllers 736 may handle a single function, and / or any combination thereof.

[0080] The one or more controllers 736 may provide signals for controlling one or more components and / or systems of the vehicle 700 in response to sensor data (eg, sensor input) received from one or more sensors. Sensor data may be received from, for example and without limitation, global navigation satellite system (“GNSS”) sensors 758 (e.g., global positioning system sensors), RADAR sensors 760 , ultrasonic sensors 762 , LIDAR sensors 764 , inertial measurement unit (IMU) sensors 766 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 796 , stereo cameras 768 , wide angle cameras 770 (e.g., fisheye cameras), infrared cameras 772 , surround cameras 774 (e.g., 360 degree cameras), long-range and / or mid-range cameras 798 , speed sensors 744 (e.g., for measuring the velocity of the vehicle 700 ), vibration sensors 742 , steering sensors 740 , brake sensors (e.g., as part of a brake sensor system 546 ), one or more occupant monitoring system (OMS) sensors 701 (e.g., one or more interior cameras), and / or other sensor types. In some embodiments, controller 736 may provide signals for controlling one or more components and / or systems of vehicle 700 based on lane marking heat map image 122 generated by heat map generator model 120 and / or other maps derived at least in part from lane marking heat map image 122 .

[0081] One or more of the controllers 736 may receive inputs (e.g., represented by input data) from the instrument cluster 732 of the vehicle 700 and provide outputs (e.g., represented by output data, display data, etc.) via a human machine interface (HMI) display 734, an audible annunciator, a speaker, and / or via other components of the vehicle 700. These outputs may include information such as vehicle speed, velocity, time, map data (e.g., Figure 7C The HMI display 734 may include information such as a high-definition (HD) map 722 of the vehicle 700, location data (e.g., the location of the vehicle 700 on the map), directions, locations of other vehicles (e.g., an occupancy grid), information about objects and states of objects as sensed by the controller 736, etc. For example, the HMI display 734 may display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, leaving 34B in two miles, etc.).

[0082] The vehicle 700 further includes a network interface 724 that can communicate via one or more networks using one or more wireless antennas 726 and / or a modem. For example, the network interface 724 may be capable of communicating via Long Term Evolution (LTE), Wideband Code Division Multiple Access (WCDMA), Universal Mobile Telecommunications System (UMTS), Global System for Mobile Communications (GSM), IMT-CDMA Multi-Carrier (CDMA2000), etc. The one or more wireless antennas 726 may also enable communication between objects (e.g., vehicles, mobile devices, etc.) in the environment using one or more local area networks such as Bluetooth, Bluetooth Low Energy (LE), Z-wave, ZigBee, etc. and / or one or more low power wide area networks (LPWAN) such as LoRaWAN, SigFox, etc.

[0083] Figure 7B For use according to some embodiments of the present disclosure Fig. 7A 700. The cameras and respective fields of view are an example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included, and / or the cameras may be located at different locations on the vehicle 700. In some embodiments, the point cloud sensor 106 and / or the optical image sensor 110 may be used Fig. 7A and Figure 7B This can be achieved using one or more sensors and / or cameras.

[0084] The camera type for the camera may include, but is not limited to, a digital camera that may be suitable for use with components and / or systems of the vehicle 700. The camera may operate at an automotive safety integrity level (ASIL) B and / or at another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 120fps, 240fps, etc., depending on the embodiment. The camera may be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red-white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a clear pixel camera such as a camera with an RCCC, RCCB, and / or RBGC color filter array may be used in an effort to improve light sensitivity.

[0085] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. One or more of the cameras (e.g., all of the cameras) can simultaneously record and provide image data (e.g., video).

[0086] One or more of the cameras may be mounted in a mounting assembly such as a custom designed (three-dimensional (3D) printed) assembly to cut off stray light and reflections from within the car (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capabilities. With respect to the wing mirror mounting assembly, the wing mirror assembly may be custom 3D printed so that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras may be integrated into the wing mirror. For side view cameras, one or more cameras may also be integrated into the four pillars at each corner of the cab.

[0087] A camera (e.g., a front-facing camera) having a field of view that includes a portion of the environment in front of the vehicle 700 can be used for surround view to help identify the forward path and obstacles, as well as assist in providing information critical to generating an occupancy grid and / or determining a preferred vehicle path with the help of one or more controllers 736 and / or control SoCs. The front-facing camera can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used for ADAS functions and systems, including lane departure warning (LDW), autonomous cruise control (ACC), and / or other functions such as traffic sign recognition.

[0088] A variety of cameras may be used in the front-facing configuration, including, for example, a monocular camera platform including a complementary metal oxide semiconductor (CMOS) color imager. Another example may be a wide-angle camera 770, which may be used to sense objects (e.g., pedestrians, intersection traffic, or bicycles) entering the field of view from the periphery. Although Figure 7B Only one wide-angle camera is shown in the figure, but there may be any number (including zero) of wide-angle cameras 770 on the vehicle 700. In addition, any number of long-range cameras 798 (e.g., a long-view stereo camera pair) may be used for depth-based object detection, especially for objects for which a neural network has not been trained. Long-range cameras 798 may also be used for object detection and classification and basic object tracking.

[0089] Any number of stereo cameras 768 may also be included in the front configuration. In at least one embodiment, one or more stereo cameras 768 may include an integrated control unit including an expandable processing unit that may provide a multi-core microprocessor and programmable logic (FPGA) with an integrated controller area network (CAN) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. Alternative stereo cameras 768 may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip that can measure the distance from the vehicle to the target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. In addition to those described herein or alternatively, other types of stereo cameras 768 may be used.

[0090] A camera having a field of view of a portion of the environment including the sides of the vehicle 700 (e.g., a side view camera) can be used for surround viewing, providing information used to create and update the occupancy grid and generate side impact collision warnings. Figure 7B The four surround cameras 774 shown in FIG. 7 can be placed on the vehicle 700. The surround cameras 774 can include a wide-angle camera 770, a fisheye camera, a 360-degree camera, and / or the like. For example, the four fisheye cameras can be placed in front, behind, and on the sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 774 (e.g., left, right, and rear), and can utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround camera.

[0091] A camera having a field of view that includes a portion of the environment behind the vehicle 700 (e.g., a rear view camera) can be used to assist with parking, surround view, rear collision warning, and creating and updating occupancy grids. A variety of cameras can be used, including but not limited to cameras that are also suitable as front cameras as described herein (e.g., long-range and / or mid-range cameras 798, stereo cameras 768, infrared cameras 772, etc.).

[0092] A camera (e.g., one or more OMS sensors 701) whose field of view includes a portion of the interior environment in the cabin of the vehicle 700 may be used as part of an occupant monitoring system (OMS), such as, but not limited to, a driver monitoring system (DMS). For example, an OMS sensor (e.g., OMS sensor 701) may be used (e.g., by controller 736) to track the gaze direction, head posture, and / or blinking of an occupant and / or driver. The gaze information may be used to determine the attention level of an occupant or driver (e.g., to detect drowsiness, fatigue, and / or distraction), and / or to take responsive actions to prevent harm to an occupant or operator. In some embodiments, data from the OMS sensor may be used to implement gaze control operations triggered by a driver and / or non-driver passenger, such as, but not limited to, adjusting cabin temperature and / or airflow, opening and closing windows, controlling cabin lighting, controlling an entertainment system, adjusting rearview mirrors, adjusting seat positions, and / or other operations. In some embodiments, the OMS may be used for applications such as determining when an object and / or passenger remains in the cabin (e.g., by detecting the presence of a passenger after the driver leaves the vehicle).

[0093] Figure 7C For use according to some embodiments of the present disclosure Fig. 7A Block diagram of an example system architecture for an example autonomous vehicle 700. It should be understood that this arrangement and other arrangements described herein are set forth merely as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components, and in any appropriate combination and location. The various functions described herein as being performed by an entity may be implemented by hardware, firmware, and / or software. For example, the various functions may be implemented by a processor executing instructions stored in a memory.

[0094] Figure 7C Each of the components, features, and systems of the vehicle 700 is illustrated as being connected via a bus 702. The bus 702 may include a controller area network (CAN) data interface (alternatively, referred to herein as a "CAN bus"). The CAN may be a network inside the vehicle 700 that assists in controlling various features and functions of the vehicle 700, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, and the like. The CAN bus may be configured to have tens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus may be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button position, and / or other vehicle status indicators. The CAN bus may be ASIL B compatible.

[0095] Although bus 702 is described as a CAN bus here, this is not intended to be limiting. For example, in addition to or alternatively to the CAN bus, FlexRay and / or Ethernet can be used. In addition, although bus 702 is represented by a single line, this is not intended to be limiting. For example, there can be any number of buses 702, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 702 can be used to perform different functions, and / or can be used for redundancy. For example, a first bus 702 can be used for a collision avoidance function, and a second bus 702 can be used for drive control. In any example, each bus 702 can communicate with any component of the vehicle 700, and two or more buses 702 can communicate with the same component. In some examples, each SoC 704, each controller 736, and / or each computer in the vehicle can have access to the same input data (e.g., input from a sensor of the vehicle 700), and can be connected to a common bus such as a CAN bus.

[0096] The vehicle 700 may include one or more controllers 736, such as those described herein. Fig. 7A Those controllers described. Controller 736 can be used for a variety of functions. Controller 736 can be coupled to any other different components and systems of vehicle 700, and can be used for control of vehicle 700, artificial intelligence of vehicle 700, infotainment for vehicle 700, and / or the like.

[0097] The vehicle 700 may include one or more system on chip (SoC) 704. The SoC 704 may include a CPU 706, a GPU 708, a processor 710, a cache 712, an accelerator 714, a data store 716, and / or other components and features not shown. The SoC 704 may be used to control the vehicle 700 in a variety of platforms and systems. For example, one or more SoCs 704 may be combined with an HD map 722 in a system (e.g., a system of the vehicle 700), and the HD map may be downloaded from one or more servers (e.g., a server) via a network interface 724. Fig.7D In some embodiments, one or more components of the SoC 704 may execute one or more elements of the path surface feature processor 105 described herein. For example, the heat map generator model 120 may be executed by at least one GPU 708 to generate one or more lane marking heat map images 122.

[0098] The CPU 706 may include a CPU cluster or a CPU complex (alternatively, referred to herein as a "CCPLEX"). The CPU 706 may include multiple cores and / or L2 caches. For example, in some embodiments, the CPU 706 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 706 may include four dual-core clusters, each of which has a dedicated L2 cache (e.g., a 2MB L2 cache). The CPU 706 (e.g., CCPLEX) may be configured to support simultaneous cluster operations, such that any combination of clusters of the CPU 706 can be active at any given time.

[0099] CPU 706 may implement power management capabilities including one or more of the following features: each hardware block may be automatically clock gated when idle to save dynamic power; each core clock may be gated when the core is not actively executing instructions due to the execution of WFI / WFE instructions; each core may be independently power gated; each core cluster may be independently clock gated when all cores are clock gated or power gated; and / or each core cluster may be independently power gated when all cores are power gated. CPU 706 may further implement an enhanced algorithm for managing power states, in which allowed power states and expected wake-up times are specified, and hardware / microcode determines the optimal power state to enter for the core, cluster, and CCPLEX. The processing core may support a simplified power state entry sequence in software, with the work being offloaded to the microcode.

[0100] The GPU 708 may include an integrated GPU (alternatively, referred to herein as an "iGPU"). The GPU 708 may be programmable and efficient for parallel workloads. In some examples, the GPU 708 may use an enhanced tensor instruction set. The GPU 708 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with at least 96KB storage capacity), and two or more of these streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB storage capacity). In some embodiments, the GPU 708 may include at least eight streaming microprocessors. The GPU 708 may use a computing application programming interface (API). In addition, the GPU 708 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0101] In the case of automotive and embedded use, GPU 708 can be power optimized to achieve optimal performance. For example, GPU 708 can be manufactured on fin field effect transistors (FinFETs). However, this is not intended to be limiting, and GPU 708 can be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor can merge several mixed precision processing cores divided into multiple blocks. For example and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed precision NVIDIA tensor cores for deep learning matrix arithmetic, L0 instruction cache, thread beam (warp) scheduler, dispatch unit and / or 64KB register file. In addition, the streaming microprocessor can include independent parallel integer and floating point data paths to provide efficient execution of workloads using a mix of calculations and addressing calculations. The streaming microprocessor may include independent thread scheduling capabilities to allow for finer-grained synchronization and collaboration between parallel threads. A streaming microprocessor may include a combined L1 data cache and shared memory unit to increase performance while simplifying programming.

[0102] GPU 708 can include high bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem that provides a peak memory bandwidth of approximately 900 GB / s in some examples. In some examples, synchronous graphics random access memory (SGRAM), such as fifth generation graphics double data rate synchronous random access memory (GDDR5), can be used in addition to or in lieu of HBM memory.

[0103] The GPU 708 may include unified memory technology that includes access counters to allow memory pages to be more accurately migrated to the processor that accesses them most frequently, thereby improving the efficiency of memory ranges shared between processors. In some examples, address translation service (ATS) support can be used to allow the GPU 708 to directly access the CPU 706 page table. In such an example, when the GPU 708 memory management unit (MMU) experiences a miss, an address translation request can be transmitted to the CPU 706. In response, the CPU 706 can look up the virtual-physical mapping for the address in its page table and transmit the translation back to the GPU 708. In this way, unified memory technology can allow a single unified virtual address space to be used for the memory of both the CPU 706 and the GPU 708, thereby simplifying GPU 708 programming and porting applications to the GPU 708.

[0104] In addition, GPU 708 may include access counters that can track how often GPU 708 accesses the memory of other processors. Access counters can help ensure that memory pages are moved to the physical memory of the processor that accesses them most frequently.

[0105] SoC 704 may include any number of caches 712, including those described herein. For example, cache 712 may include an L3 cache available to both CPU 706 and GPU 708 (e.g., connected to both CPU 706 and GPU 708). Cache 712 may include a write-back cache that may track the state of a line, for example, by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache may include 4MB or more, although smaller cache sizes may also be used.

[0106] The SoC 704 may include one or more arithmetic logic units (ALUs) that may be used to perform processing for any of a variety of tasks or operations with respect to the vehicle 700, such as processing a DNN. Additionally, the SoC 704 may include a floating point unit (FPU) or other math coprocessor or digital coprocessor type for performing mathematical operations within the system. For example, the SoC 704 may include one or more FPUs integrated as execution units within the CPU 706 and / or GPU 708.

[0107] SoC 704 may include one or more accelerators 714 (e.g., hardware accelerators, software accelerators, or combinations thereof). For example, SoC 704 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. The large on-chip memory (e.g., 4MB SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other calculations. The hardware acceleration cluster may be used to supplement GPU 708 and offload some tasks of GPU 708 (e.g., free up more cycles of GPU 708 for performing other tasks). As an example, accelerator 714 may be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are sufficiently stable to be easily controlled for acceleration. When used herein, the term "CNN" may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0108] The accelerator 714 (e.g., a hardware acceleration cluster) may include a deep learning accelerator (DLA). The DLA may include one or more tensor processing units (TPUs) that may be configured to provide an additional 10 trillion operations per second for deep learning applications and reasoning. The TPU may be an accelerator configured to perform image processing functions (e.g., for CNN, RCNN, etc.) and optimized for performing image processing functions. The DLA may be further optimized for a specific set of neural network types and floating point operations and reasoning. The design of the DLA may provide higher performance per millimeter than a general-purpose GPU, and far exceeds the performance of the CPU. The TPU may perform several functions, including a single instance convolution function, support, for example, INT8, INT16, and FP16 data types for both features and weights, and post-processor functions.

[0109] The DLA can quickly and efficiently execute neural networks, particularly CNNs, on processed or unprocessed data for any of a wide variety of functions, such as, but not limited to: a CNN for object recognition and detection using data from a camera sensor; a CNN for distance estimation using data from a camera sensor; a CNN for emergency vehicle detection and identification and detection using data from a microphone; a CNN for facial recognition and vehicle owner identification using data from a camera sensor; and / or a CNN for safety and / or security-related events.

[0110] The DLA can perform any function of the GPU 708, and by using an inference accelerator, for example, the designer can target any function to either the DLA or the GPU 708. For example, the designer can focus the processing of CNNs and floating point operations on the DLA, and leave other functions to the GPU 708 and / or other accelerators 714.

[0111] The accelerator 714 (e.g., a hardware acceleration cluster) may include a programmable vision accelerator (PVA), which may be alternatively referred to herein as a computer vision accelerator. The PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA may provide a balance between performance and flexibility. For example, each PVA may include, for example and without limitation, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0112] The RISC core can interact with an image sensor (e.g., an image sensor of any camera described herein), an image signal processor, and / or the like. Each of these RISC cores can include any amount of memory. Depending on the embodiment, the RISC core can use any of a number of protocols. In some examples, the RISC core can execute a real-time operating system (RTOS). The RISC core can be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and / or storage devices. For example, the RISC core can include an instruction cache and / or a tightly coupled RAM.

[0113] The DMA may enable components of the PVA to access system memory independently of the CPU 706. The DMA may support any number of features used to provide optimizations for the PVA, including, but not limited to, support for multi-dimensional addressing and / or circular addressing. In some examples, the DMA may support addressing in up to six or more dimensions, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0114] The vector processor can be a programmable processor that can be designed to efficiently and flexibly perform programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA can include a PVA core and two vector processing subsystem partitions. The PVA core can include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem can operate as the main processing engine of the PVA, and can include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., VMEM). The VPU core can include a digital signal processor, such as, for example, a single instruction multiple data (SIMD), a very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and rate.

[0115] Each of the vector processors may include an instruction cache and may be coupled to a dedicated memory. As a result, in some examples, each of the vector processors may be configured to execute independently of the other vector processors. In other examples, the vector processors included in a particular PVA may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms on sequence images or portions of an image. Among other things, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each of these PVAs. In addition, the PVA may include additional error correction code (ECC) memory to enhance overall system security.

[0116] The accelerator 714 (e.g., a hardware acceleration cluster) may include an on-chip computer vision network and SRAM to provide high bandwidth, low latency SRAM for the accelerator 714. In some examples, the on-chip memory may include at least 4MB of SRAM consisting of, for example and not limited to, eight field-configurable memory blocks, which may be accessed by both the PVA and the DLA. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, a configuration circuit system, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA may access the memory via a backbone that provides high-speed memory access to the PVA and DLA. The backbone may include an on-chip computer vision network that interconnects the PVA and DLA to the memory (e.g., using APB).

[0117] The on-chip computer vision network may include an interface that determines that both the PVA and DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such an interface may provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface may comply with ISO 26262 or IEC 61508 standards, but other standards and protocols may also be used.

[0118] In some examples, SoC 704 may include a real-time ray tracing hardware accelerator such as described in U.S. Patent Application No. 16 / 101,232 filed on August 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the position and range of objects (e.g., within a world model) in order to generate real-time visualization simulations for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for SONAR system simulations, for general wave propagation simulations, for comparison with LIDAR data for the purpose of positioning and / or other functions, and / or for other purposes. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing related operations.

[0119] The accelerator 714 (e.g., a hardware accelerator cluster) has a wide range of uses in autonomous driving. The PVA can be a programmable visual accelerator that can be used in key processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are a good match for algorithmic domains that require predictable processing, low power, and low latency. In other words, the PVA performs well on semi-dense or dense rule computations, and even on small data sets that require predictable runtimes with low latency and low power. Therefore, in the context of a platform for autonomous vehicles, the PVA is designed to run classic computer vision algorithms because they are effective at object detection and integer math.

[0120] For example, according to one embodiment of the technology, PVA is used to perform computer stereo vision. In some examples, semi-global matching based algorithms can be used, but this is not intended to be limiting. Many applications for Level 3-5 autonomous driving require instant motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions on inputs from two monocular cameras.

[0121] In some examples, PVA can be used to perform dense optical flow. According to the process raw RADAR data (e.g., using 4D fast Fourier transform) to provide processed RADAR. In other examples, PVA is used for time-of-flight depth processing, which is for example by processing raw time-of-flight data to provide processed time-of-flight data.

[0122] DLA can be used to run any type of network to enhance control and driving safety, including, for example, a neural network that outputs a confidence measure for each object detection. Such a confidence value can be interpreted as a probability, or as providing a relative "weight" of each detection compared to other detections. This confidence value enables the system to make further decisions about which detections should be considered true positive detections rather than false positive detections. For example, the system can set a threshold for confidence and only consider detections that exceed the threshold as true positive detections. In an automatic emergency braking (AEB) system, a false positive detection will cause the vehicle to automatically perform emergency braking, which is obviously undesirable. Therefore, only the most confident detection should be considered a trigger for AEB. DLA can run a neural network for regressing confidence values. The neural network can take at least some subset of parameters as its input, such as bounding box dimensions, ground plane estimates obtained (e.g., from another subsystem), inertial measurement unit (IMU) sensor 766 output related to vehicle 700 orientation and distance, 3D position estimates of objects obtained from neural networks and / or other sensors (e.g., LIDAR sensor 764 or RADAR sensor 760), etc.

[0123] SoC 704 may include one or more data stores 716 (e.g., memory). Data store 716 may be on-chip memory of SoC 704 that may store neural networks to be executed on the GPU and / or DLA. In some examples, data store 716 may be large enough to store multiple instances of a neural network for redundancy and safety. Data store 716 may include an L2 or L3 cache 712. References to data store 716 may include references to memory associated with a PVA, DLA, and / or other accelerator 714 as described herein.

[0124] SoC 704 may include one or more processors 710 (e.g., embedded processors). Processor 710 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management functions and related safety implementations. The boot and power management processor may be part of the SoC 704 boot sequence and may provide runtime power management services. The boot power and management processor may provide clock and voltage programming, auxiliary system low power state transitions, SoC 704 thermal and temperature sensor management, and / or SoC 704 power state management. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and SoC 704 may use the ring oscillator to detect the temperature of CPU 706, GPU 708, and / or accelerator 714. If it is determined that the temperature exceeds a threshold, the boot and power management processor may enter a temperature fault routine and place SoC 704 in a lower power state and / or place the vehicle 700 in a driver safety parking mode (e.g., parking the vehicle 700 safely).

[0125] Processor 710 may further include a set of embedded processors that may be used as an audio processing engine. The audio processing engine may be an audio subsystem that allows for full hardware support for multi-channel audio through multiple interfaces and a wide range of flexible audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

[0126] The processor 710 may further include an always-on processor engine that may provide the necessary hardware features to support low-power sensor management and wake-up use cases. The always-on processor engine may include a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0127] The processor 710 may further include a safety cluster engine, which includes a dedicated processor subsystem that handles safety management of automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In safety mode, the two or more cores may operate in lockstep mode and act as a single core with comparison logic to detect any differences between their operations.

[0128] Processor 710 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0129] Processor 710 may further include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0130] The processor 710 may include a video image compositer, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required for a video playback application to produce a final image for a player window. The video image compositer may perform lens distortion correction for the wide-angle camera 770, the surround camera 774, and / or for an in-cab surveillance camera sensor. The in-cab surveillance camera sensor is preferably monitored by a neural network running on another instance of the advanced SoC, configured to recognize in-cab events and respond accordingly. The in-cab system may perform lip reading to activate mobile phone service and place calls, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode and are disabled in other circumstances.

[0131] The video image compositer may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, in the case of motion in the video, the noise reduction appropriately weights the spatial information and reduces the weight of information provided by adjacent frames. In the case where an image or portion of an image does not include motion, the temporal noise reduction performed by the video image compositer may use information from previous images to reduce noise in the current image.

[0132] The video image compositor may also be configured to perform stereoscopic rectification on the input stereoscopic footage frames. The video image compositor may further be used for user interface composition when the operating system desktop is in use and the GPU 708 does not need to continuously render new surfaces. Even when the GPU 708 is powered on and active for 3D rendering, the video image compositor may be used to offload the GPU 708 to improve performance and responsiveness.

[0133] The SoC 704 may further include a mobile industry processor interface (MIPI) camera serial interface for receiving video and input from a camera, a high-speed interface, and / or a video input block that may be used for camera and related pixel input functions. The SoC 704 may further include an input / output controller that may be controlled by software and may be used to receive I / O signals that are not committed to a specific role.

[0134] SoC 704 may further include a wide range of peripheral interfaces to enable communication with peripherals, audio codecs, power management and / or other devices. SoC 704 may be used to process data from cameras (connected via Gigabit multimedia serial links and Ethernet), sensors (e.g., LIDAR sensor 764, RADAR sensor 760, etc., which may be connected via Ethernet), data from bus 702 (e.g., speed of vehicle 700, steering wheel position, etc.), data from GNSS sensor 758 (connected via Ethernet or CAN bus). SoC 704 may further include dedicated high-performance mass storage controllers, which may include their own DMA engines, and which may be used to free up CPU 706 from routine data management tasks.

[0135] SoC 704 can be an end-to-end platform with a flexible architecture that spans automation levels 3-5, providing a comprehensive functional safety architecture that leverages and efficiently uses computer vision and ADAS technologies to achieve diversity and redundancy, together with deep learning tools to provide a platform for a flexible and reliable driving software stack. SoC 704 can be faster, more reliable, and even more energy efficient and space efficient than conventional systems. For example, when combined with CPU 706, GPU 708, and data storage 716, accelerator 714 can provide a fast and efficient platform for level 3-5 autonomous vehicles.

[0136] The technology thus provides capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on CPUs, which can be configured using high-level programming languages ​​such as the C programming language to execute a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often fail to meet the performance requirements of many computer vision applications, such as those related to, for example, execution time and power consumption. In particular, many CPUs are unable to execute complex object detection algorithms in real time, which is a requirement for in-vehicle ADAS applications and a requirement for practical Level 3-5 autonomous vehicles.

[0137] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, the techniques described herein allow multiple neural networks to be executed simultaneously and / or sequentially, and the results to be combined together to achieve Level 3-5 autonomous driving functions. For example, a CNN executed on a DLA or dGPU (e.g., GPU 720) may include text and word recognition, allowing the supercomputer to read and understand traffic signs, including signs for which a neural network has not been specifically trained. The DLA may further include a neural network capable of recognizing, interpreting, and providing semantic understanding of the signs, and passing that semantic understanding to a path planning module running on the CPU complex.

[0138] As another example, as required for Level 3, 4, or 5 driving, multiple neural networks may be running simultaneously. For example, a warning sign consisting of "Caution: Flashing Lights Indicate Icing Conditions" along with electric lights may be interpreted by several neural networks independently or collectively. The sign itself may be recognized as a traffic sign by a deployed first neural network (e.g., a trained neural network), and the text "Flashing Lights Indicate Icing Conditions" may be interpreted by a deployed second neural network that informs the vehicle's path planning software (preferably executing on a CPU complex) that icing conditions exist when the flashing lights are detected. The flashing lights may be identified by operating a deployed third neural network over multiple frames that informs the vehicle's path planning software of the presence (or absence) of the flashing lights. All three neural networks may be running simultaneously, for example, within the DLA and / or on the GPU 708.

[0139] In some examples, a CNN for facial recognition and owner recognition can use data from the camera sensor to identify the presence of an authorized driver and / or owner of the vehicle 700. The always-on sensor processing engine can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in security mode, disable the vehicle when the owner leaves the vehicle. In this way, the SoC 704 provides security against theft and / or carjacking.

[0140] In another example, a CNN for emergency vehicle detection and identification can use data from microphone 796 to detect and identify emergency vehicle sirens. In contrast to conventional systems that use general classifiers to detect sirens and manually extract features, SoC 704 uses CNN to classify environmental and urban sounds and to classify visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative closing rate of emergency vehicles (e.g., by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the local area in which the vehicle is operating, as identified by the GNSS sensor 758. Thus, for example, when operating in Europe, the CNN will seek to detect European sirens, and when in the United States, the CNN will seek to identify only North American sirens. Once an emergency vehicle is detected, with the assistance of the ultrasonic sensor 762, the control program can be used to execute emergency vehicle safety routines to slow the vehicle, drive to the side of the road, stop the vehicle, and / or idle the vehicle until the emergency vehicle passes.

[0141] The vehicle may include a CPU 718 (e.g., a discrete CPU or dCPU) that may be coupled to the SoC 704 via a high-speed interconnect (e.g., PCIe). The CPU 718 may include, for example, an X86 processor. The CPU 718 may be used to perform any of a variety of functions, including, for example, arbitrating potentially inconsistent results between ADAS sensors and the SoC 704, and / or monitoring the status and health of the controller 736 and / or the infotainment SoC 730.

[0142] The vehicle 700 may include a GPU 720 (e.g., a discrete GPU or dGPU) that may be coupled to the SoC 704 via a high-speed interconnect (e.g., NVIDIA's NVLINK). The GPU 720 may provide additional artificial intelligence functionality, for example, by executing redundant and / or different neural networks, and may be used to train and / or update the neural network based on inputs (e.g., sensor data) from sensors of the vehicle 700.

[0143] The vehicle 700 may further include a network interface 724, which may include one or more wireless antennas 726 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 724 can be used to enable wireless connections with the cloud (e.g., with a server 778 and / or other network devices), with other vehicles, and / or with computing devices (e.g., a passenger's client device) via the Internet. In order to communicate with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across a network and through the Internet). The direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide the vehicle 700 with information about vehicles approaching the vehicle 700 (e.g., vehicles in front of, to the side of, and / or behind the vehicle 700). This functionality can be part of the collaborative adaptive cruise control functionality of the vehicle 700.

[0144] The network interface 724 may include a SoC that provides modulation and demodulation functions and enables the controller 736 to communicate over a wireless network. The network interface 724 may include a radio frequency front end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. The frequency conversion may be performed by a known process and / or may be performed using a super-heterodyne process. In some examples, the radio frequency front end function may be provided by a separate chip. The network interface may include a wireless function for communicating via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0145] The vehicle 700 may further include a data store 728, which may include off-chip storage (e.g., outside the SoC 704). The data store 728 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, a hard disk, and / or other components and / or devices that can store at least one bit of data.

[0146] The vehicle 700 may further include a GNSS sensor 758. The GNSS sensor 758 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used to assist in mapping, perception, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 758 may be used, including, for example and without limitation, GPS using a USB connector with an Ethernet to serial (RS-232) bridge.

[0147] The vehicle 700 may further include a RADAR sensor 760. The RADAR sensor 760 may be used by the vehicle 700 for remote vehicle detection even in darkness and / or inclement weather conditions. The RADAR functional safety level may be ASIL B. The RADAR sensor 760 may use CAN and / or bus 702 (e.g., to transmit data generated by the RADAR sensor 760) for control and access to object tracking data, accessing Ethernet in some examples to access raw data. A variety of RADAR sensor types may be used. For example and without limitation, the RADAR sensor 760 may be suitable for front, rear, and side RADAR use. In some examples, a pulse Doppler RADAR sensor is used.

[0148] The RADAR sensor 760 may include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, and the like. In some examples, the long-range RADAR may be used for adaptive cruise control functions. The long-range RADAR system may provide a wide field of view (e.g., within a range of 250m) achieved through two or more independent browsing. The RADAR sensor 760 may help distinguish between static objects and moving objects, and may be used by the ADAS system for emergency braking assistance and forward collision warnings. The long-range RADAR sensor may include a single-station multimode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the central four antennas may create a focused beam pattern designed to record the surroundings of the vehicle 700 at a higher rate with minimal traffic interference from adjacent lanes. The other two antennas may expand the field of view, making it possible to quickly detect vehicles entering or leaving the lane of the vehicle 700.

[0149] As an example, a medium-range RADAR system may include a range of up to 760m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 750 degrees (rear). A short-range RADAR system may include, but is not limited to, a RADAR sensor designed to be mounted on both ends of a rear bumper. When mounted on both ends of a rear bumper, such a RADAR sensor system may create two beams that continuously monitor the blind spots behind and beside the vehicle.

[0150] Short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0151] The vehicle 700 may further include an ultrasonic sensor 762. The ultrasonic sensor 762, which may be placed on the front, rear, and / or sides of the vehicle 700, may be used for parking assistance and / or creating and updating an occupancy grid. A variety of ultrasonic sensors 762 may be used, and different ultrasonic sensors 762 may be used for different detection ranges (e.g., 2.5 m, 4 m). The ultrasonic sensor 762 may operate at a functional safety level of ASIL B.

[0152] The vehicle 700 may include a LIDAR sensor 764. The LIDAR sensor 764 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensor 764 may be ASIL B for functional safety level. In some examples, the vehicle 700 may include multiple LIDAR sensors 764 (e.g., two, four, six, etc.) that may use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0153] In some examples, the LIDAR sensor 764 may be able to provide a list of objects and their distances for a 360-degree field of view. Commercially available LIDAR sensors 764 may have, for example, an advertised range of approximately 700m, an accuracy of 2cm-3cm, and support for 700Mbps Ethernet connections. In some examples, one or more non-protruding LIDAR sensors 764 may be used. In such examples, the LIDAR sensor 764 may be implemented as a small device that can be embedded in the front, back, side, and / or corner of the vehicle 700. In such examples, the LIDAR sensor 764 may provide a field of view of up to 120 degrees horizontally and 35 degrees vertically, with a range of 200m, even for low reflectivity objects. The front-mounted LIDAR sensor 764 may be configured for a horizontal field of view between 45 and 135 degrees.

[0154] In some examples, LIDAR technologies such as 3D flash LIDAR may also be used. 3D flash LIDAR uses a flash of laser as an emission source to illuminate the vehicle's surroundings up to about 200m. The flash LIDAR unit includes a receiver that records the laser pulse transmission time and the reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LIDAR can allow highly accurate and distortion-free images of the surrounding environment to be generated with each laser flash. In some examples, four flash LIDAR sensors can be deployed, one on each side of the vehicle 700. Available 3D flash LIDAR systems include solid-state 3D staring array LIDAR cameras (e.g., non-browsing LIDAR devices) with no moving parts other than fans. The flash LIDAR device can use 5 nanosecond Class I (eye-safe) laser pulses per frame, and can capture the reflected laser light in the form of a 3D range point cloud and co-registered intensity data. By using flash LIDAR, and because flash LIDAR is a solid-state device with no moving parts, the LIDAR sensor 764 may be less susceptible to motion blur, vibration, and / or shock.

[0155] The vehicle may further include an IMU sensor 766. In some examples, the IMU sensor 766 may be located at the center of the rear axle of the vehicle 700. The IMU sensor 766 may include, for example and without limitation, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, for example, in a six-axis application, the IMU sensor 766 may include an accelerometer and a gyroscope, and in a nine-axis application, the IMU sensor 766 may include an accelerometer, a gyroscope, and a magnetometer.

[0156] In some embodiments, the IMU sensor 766 can be implemented as a miniature high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a micro-electromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filter algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 766 can enable the vehicle 700 to estimate heading by directly observing and correlating velocity changes from the GPS to the IMU sensor 766 without the need for input from a magnetic sensor. In some examples, the IMU sensor 766 and the GNSS sensor 758 can be combined into a single integrated unit.

[0157] The vehicle may include a microphone 796 positioned in and / or around the vehicle 700. The microphone 796 may be used for, among other things, emergency vehicle detection and identification.

[0158] The vehicle may further include any number of camera types, including a stereo camera 768, a wide angle camera 770, an infrared camera 772, a surround camera 774, a long-range and / or mid-range camera 798, and / or other camera types. These cameras may be used to capture image data around the entire periphery of the vehicle 700. The type of camera used depends on the embodiment and the requirements of the vehicle 700, and any combination of camera types may be used to provide the necessary coverage around the vehicle 700. In addition, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As an example and not limitation, the cameras may support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras described herein may include a 10GbE ... Fig. 7A and Figure 7B Described in more detail.

[0159] The vehicle 700 may further include a vibration sensor 742. The vibration sensor 742 may measure vibrations of components of the vehicle, such as axles. For example, changes in vibration may indicate changes in the road surface. In another example, when two or more vibration sensors 742 are used, the difference between the vibrations may be used to determine friction or slippage of the road surface (e.g., when there is a vibration difference between a powered drive shaft and a free-spinning shaft).

[0160] The vehicle 700 may include an ADAS system 738. In some examples, the ADAS system 738 may include a SoC. The ADAS system 738 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0161] The ACC system may use a RADAR sensor 760, a LIDAR sensor 764, and / or a camera. The ACC system may include a longitudinal ACC and / or a lateral ACC. The longitudinal ACC monitors and controls the distance to the vehicle immediately in front of the vehicle 700, and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle in front. The lateral ACC performs distance keeping and suggests that the vehicle 700 change lanes when necessary. The lateral ACC is related to other ADAS applications such as LCA and CWS.

[0162] CACC uses information from other vehicles, which can be received from other vehicles indirectly via a wireless link or through a network connection (e.g., through the Internet) via a network interface 724 and / or a wireless antenna 726. A direct link can be provided by a vehicle-to-vehicle (V2V) communication link, while an indirect link can be an infrastructure-to-vehicle (I2V) communication link. Typically, the V2V communication concept provides information about the vehicle immediately ahead (e.g., the vehicle immediately ahead of the vehicle 700 and in the same lane as it), while the I2V communication concept provides information about traffic farther ahead. The CACC system may include either or both of the I2V and V2V information sources. Given information about the vehicle ahead of the vehicle 700, CACC can be more reliable, and it is possible to improve the smoothness of traffic flow and reduce road congestion.

[0163] The FCW system is designed to alert the driver to hazards so that the driver can take corrective action. The FCW system uses a front camera and / or RADAR sensor 760 coupled to a dedicated processor, DSP, FPGA and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker and / or vibrating components. The FCW system can provide warnings in the form of, for example, sound, visual warnings, vibrations and / or rapid brake pulses.

[0164] The AEB system detects an impending front collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within specified time or distance parameters. The AEB system can use a front camera and / or RADAR sensor 760 coupled to a dedicated processor, DSP, FPGA and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid the collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes in an effort to prevent or at least mitigate the effects of the predicted collision. The AEB system may include technologies such as dynamic brake support and / or collision approach braking.

[0165] The LDW system provides visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle 700 crosses a lane marking. When the driver indicates an intention to leave the lane, by activating a turn signal, the LDW system is not activated. The LDW system may use a front-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0166] The LKA system is a variation of the LDW system. If the vehicle 700 begins to leave the lane, the LKA system provides steering input or braking to correct the vehicle 700.

[0167] The BSW system detects and warns the driver of vehicles in the car's blind spot. The BSW system can provide visual, auditory and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses a turn signal. The BSW system can use a rear-facing camera and / or RADAR sensor 760 coupled to a dedicated processor, DSP, FPGA and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker and / or vibration components.

[0168] The RCTW system can provide visual, audible, and / or tactile notifications when an object is detected outside the range of the rear camera while the vehicle 700 is in reverse. Some RCTW systems include AEB to ensure that the vehicle brakes are applied to avoid a crash. The RCTW system can use one or more rear RADAR sensors 760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0169] Conventional ADAS systems may be prone to false positive results, which may annoy and distract the driver, but are typically not catastrophic because the ADAS system alerts the driver and allows the driver to decide whether the safety condition really exists and take action accordingly. However, in the autonomous vehicle 700, in the case of conflicting results, the vehicle 700 itself must decide whether to pay attention to the results from the main computer or the auxiliary computer (e.g., the first controller 736 or the second controller 736). For example, in some embodiments, the ADAS system 738 can be a backup and / or auxiliary computer for providing perception information to the backup computer rationality module. The backup computer rationality monitor can run redundant and diverse software on hardware components to detect faults in perception and dynamic driving tasks. The output from the ADAS system 738 can be provided to the supervisory MCU. If the outputs from the main computer and the auxiliary computer conflict, the supervisory MCU must determine how to coordinate the conflict to ensure safe operation.

[0170] In some examples, the master computer can be configured to provide a confidence score to the supervisory MCU, indicating the master computer's confidence in the selected result. If the confidence score exceeds a threshold, the supervisory MCU can follow the direction of the master computer, regardless of whether the auxiliary computer provides conflicting or inconsistent results. In the case where the confidence score does not meet the threshold and the master computer and the auxiliary computer indicate different results (e.g., conflicts), the supervisory MCU can arbitrate between these computers to determine the appropriate result.

[0171] The supervisory MCU may be configured to run a neural network that is trained and configured to determine conditions under which the auxiliary computer provides a false alarm based on outputs from the primary computer and the auxiliary computer. Thus, the neural network in the supervisory MCU may learn when the output of the auxiliary computer may be trusted and when it may not. For example, when the auxiliary computer is a RADAR-based FCW system, the neural network in the supervisory MCU may learn when the FCW system is identifying a metal object that is not actually dangerous, such as a drainage grate or manhole cover that triggers an alarm. Similarly, when the auxiliary computer is a camera-based LDW system, the neural network in the supervisory MCU may learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU may include at least one of a DLA or GPU suitable for running the neural network with associated memory. In a preferred embodiment, the supervisory MCU may include and / or be included as a component of the SoC 704.

[0172] In other examples, the ADAS system 738 may include an auxiliary computer that uses traditional computer vision rules to perform ADAS functions. In this way, the auxiliary computer can use classic computer vision rules (if-then), and the presence of a neural network in the supervisory MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identity make the entire system more fault-tolerant, especially for failures caused by software (or software-hardware interface) functions. For example, if there is a software vulnerability or error in the software running on the main computer and the non-identical software code running on the auxiliary computer provides the same overall result, the supervisory MCU can be more confident that the overall result is correct and that the vulnerability in the software or hardware on the main computer does not cause a substantial error.

[0173] In some examples, the output of the ADAS system 738 can be fed to the perception block of the main computer and / or the dynamic driving task block of the main computer. For example, if the ADAS system 738 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information in identifying the object. In other examples, the auxiliary computer can have its own neural network that is trained and thus reduces the risk of false positives as described herein.

[0174] The vehicle 700 may further include an infotainment SoC 730 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 730 may include a combination of hardware and software that may be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), phone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation system, rear parking assistance, radio data system, vehicle-related information such as fuel level, total distance covered, brake fuel level, oil level, door open / closed, air filter information, etc.) to the vehicle 700. For example, the infotainment SoC 730 may include a radio, a disc player, a navigation system, a video player, USB and Bluetooth connectivity, an onboard computer, onboard entertainment, Wi-Fi, steering wheel audio controls, hands-free voice controls, a head-up display (HUD), an HMI display 734, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 730 may further be used to provide information (e.g., visual and / or auditory) to a user of the vehicle, such as information from an ADAS system 738, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0175] The infotainment SoC 730 may include GPU functionality. The infotainment SoC 730 may communicate with other devices, systems, and / or components of the vehicle 700 via a bus 702 (e.g., a CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 730 may be coupled to a supervisory MCU so that in the event of a failure of a main controller 736 (e.g., a main and / or backup computer of the vehicle 700), the GPU of the infotainment system may perform some self-driving functions. In such an example, the infotainment SoC 730 may place the vehicle 700 in a driver safety parking mode as described herein.

[0176] The vehicle 700 may further include an instrument cluster 732 (e.g., a digital instrument panel, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 732 may include a controller and / or a supercomputer (e.g., a separate controller or a supercomputer). The instrument cluster 732 may include a set of instruments, such as a speedometer, a fuel level, an oil pressure, a tachometer, an odometer, a turn indicator, a shift position indicator, a seat belt warning light, a parking brake warning light, an engine fault light, an airbag (SRS) system information, a lighting control, a safety system control, navigation information, and the like. In some examples, information may be displayed and / or shared between the infotainment SoC 730 and the instrument cluster 732. In other words, the instrument cluster 732 may be included as part of the infotainment SoC 730, or vice versa.

[0177] Fig.7D For a cloud-based server and Fig. 7A Schematic diagram of a system for communicating between an example autonomous vehicle 700. System 776 may include a server 778, a network 790, and a vehicle including vehicle 700. Server 778 may include multiple GPUs 784 (A)-784 (H) (collectively referred to as GPUs 784 here), PCIe switches 782 (A)-782 (H) (collectively referred to as PCIe switches 782 here), and / or CPUs 780 (A)-780 (B) (collectively referred to as CPUs 780 here). GPUs 784, CPUs 780, and PCIe switches may be interconnected with high-speed interconnects and / or PCIe connections 786 such as, for example and without limitation, NVLink interfaces 788 developed by NVIDIA. In some examples, GPUs 784 are connected via NVLink and / or NVSwitch SoCs, and GPUs 784 and PCIe switches 782 are connected via PCIe interconnects. Although eight GPUs 784, two CPUs 780, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each of the servers 778 may include any number of GPUs 784, CPUs 780, and / or PCIe switches. For example, each of the servers 778 may include eight, sixteen, thirty-two, and / or more GPUs 784.

[0178] Server 778 may receive image data over network 790 and from a vehicle, the image data representing images showing unexpected or changed road conditions, such as recently begun road work. Server 778 may transmit neural network 792, updated neural network 792, and / or map information 794, including information about traffic and road conditions, over network 790 and to the vehicle. Updates to map information 794 may include updates to HD map 722, such as information about construction sites, potholes, curves, flooding, or other obstacles. In some examples, neural network 792, updated neural network 792, and / or map information 794 may have been generated from new training and / or data received from any number of vehicles in the environment and / or based on experience from training performed at a data center (e.g., using server 778 and / or other servers).

[0179] Server 778 can be used to train a machine learning model (e.g., a neural network) based on training data. The training data can be generated by the vehicle, and / or can be generated in simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., when the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not labeled and / or preprocessed (e.g., when the neural network does not require supervised learning). Training can be performed according to any one or more classes of machine learning techniques, including but not limited to the following classes: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal components and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternate dictionary learning), rule-based machine learning, anomaly detection, and any variants or combinations thereof. Once the machine learning model is trained, the machine learning model can be used by the vehicle (e.g., transmitted to the vehicle via network 790), and / or the machine learning model can be used by server 778 to remotely monitor the vehicle.

[0180] In some examples, server 778 can receive data from the vehicle and apply the data to the latest real-time neural network for real-time intelligent reasoning. Server 778 may include a deep learning supercomputer and / or a dedicated AI computer powered by GPU 784, such as DGX and DGX Station machines developed by NVIDIA. However, in some examples, server 778 may include a deep learning infrastructure of a data center powered only by CPUs.

[0181] The deep learning infrastructure of server 778 may be capable of fast real-time inference, and may use this capability to assess and verify the health of the processors, software, and / or associated hardware in vehicle 700. For example, the deep learning infrastructure may receive periodic updates from vehicle 700, such as a sequence of images and / or objects located in the sequence of images that vehicle 700 has located (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure may run its own neural network to identify objects and compare them to the objects identified by vehicle 700, and if the results do not match and the infrastructure concludes that the AI ​​in vehicle 700 has failed, then server 778 may transmit a signal to vehicle 700 instructing the fail-safe computer of vehicle 700 to take control, notify passengers, and complete a safe parking maneuver.

[0182] For reasoning, the server 778 may include a GPU 784 and one or more programmable reasoning accelerators (e.g., NVIDIA's TensorRT 3). The combination of GPU-powered servers and reasoning acceleration can make real-time responses possible. In other examples, such as when performance is not so important, CPU, FPGA, and other processor-powered servers can be used for reasoning.

[0183] Example computing device

[0184] Figure 88 is a block diagram of an example computing device 800 suitable for implementing some embodiments of the present disclosure. The computing device 800 may include an interconnect system 802 that directly or indirectly couples the following devices: a memory 804, one or more central processing units (CPUs) 806, one or more graphics processing units (GPUs) 808, a communication interface 810, an input / output (I / O) port 812, an input / output component 814, a power supply 816, one or more presentation components 818 (e.g., a display), and one or more logic units 820. In at least one embodiment, the computing device 800 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). For non-limiting examples, the one or more GPUs 808 may include one or more vGPUs, the one or more CPUs 806 may include one or more vCPUs, and / or the one or more logic units 820 may include one or more virtual logic units. Thus, the computing device 800 may include discrete components (e.g., a complete GPU dedicated to the computing device 800), virtual components (e.g., a portion of a GPU dedicated to the computing device 800), or a combination thereof. In some embodiments, the CPU 806 and / or the GPU 808 may execute one or more elements of the path surface feature processor 105 described herein, such as, but not limited to, the heat map generator model 120.

[0185] although Figure 8 The various blocks of are shown as being connected via an interconnect system 802 having wires, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, a presentation component 818 such as a display device may be considered an I / O component 814 (e.g., if the display is a touch screen). As another example, the CPU 806 and / or the GPU 808 may include memory (e.g., the memory 804 may represent a storage device other than the memory of the GPU 808, the CPU 806, and / or other components). In other words, Figure 8 The computing devices described herein are illustrative only. No distinction is made between categories such as "workstations," "servers," "laptops," "desktops," "tablets," "client devices," "mobile devices," "handheld devices," "game consoles," "electronic control units (ECUs)," "virtual reality systems," and / or other device or system types, as all are considered Figure 8 within the range of computing devices.

[0186] The interconnection system 802 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 802 may include one or more links or bus types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a video electronics standard association (VESA) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, the CPU 806 may be directly connected to the memory 804. In addition, the CPU 806 may be directly connected to the GPU 808. In the case where there is a direct or point-to-point connection between components, the interconnection system 802 may include a PCIe link to perform the connection. In these examples, the computing device 800 does not need to include a PCI bus.

[0187] Memory 804 may include any of a variety of computer-readable media. Computer-readable media may be any available media that can be accessed by computing device 800. Computer-readable media may include volatile and nonvolatile media and removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0188] Computer storage media may include volatile and nonvolatile media and / or removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 804 may store computer-readable instructions (e.g., representing programs and / or program elements, such as an operating system). Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by computing device 800. As used herein, computer storage media does not include the signals themselves.

[0189] Computer storage media may contain computer readable instructions, data structures, program modules, and / or other data types in a modulated data signal such as a carrier wave or other transmission mechanism, and include any information delivery media. The term "modulated data signal" may refer to a signal that has one or more of its characteristics set or changed in such a way that information is encoded into the signal. By way of example and not limitation, computer storage media may include wired media such as a wired network or a direct wired connection, and wireless media such as sound, RF, infrared, and other wireless media. Any combination of the above should also be included within the scope of computer readable media.

[0190] The CPU 806 may be configured to execute at least some of the computer-readable instructions so as to control one or more components of the computing device 800 to perform one or more of the methods and / or processes described herein. Each of the CPUs 806 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of simultaneously processing a large number of software threads. The CPU 806 may include any type of processor, and may include different types of processors, depending on the type of computing device 800 implemented (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers). For example, depending on the type of computing device 800, the processor may be an Advanced RISC Mechanism (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors such as mathematical coprocessors, the computing device 800 may also include one or more CPUs 806.

[0191] In addition to or in place of the CPU 806, the GPU 808 may also be configured to execute at least some computer-readable instructions to control one or more components of the computing device 800 to perform one or more methods and / or processes described herein. One or more GPUs 808 may be integrated GPUs (e.g., with one or more CPUs 806) and / or one or more GPUs 808 may be discrete GPUs. In an embodiment, one or more GPUs 808 may be coprocessors of one or more CPUs 806. The computing device 800 may use the GPU 808 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, the GPU 808 may be used for general-purpose computing on the GPU (GPGPU). The GPU 808 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The GPU 808 may generate pixel data for outputting an image in response to a rendering command (e.g., a rendering command from the CPU 806 received via a host interface). The GPU 808 may include a graphics memory such as a display memory for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 804. GPU 808 may include two or more GPUs operating in parallel (e.g., via a link). The link may connect the GPUs directly (e.g., using NVLINK) or may connect the GPUs through a switch (e.g., using NVSwitch). When combined together, each GPU 808 may generate pixel data or GPGPU data for different portions of output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0192] In addition to or in lieu of the CPU 806 and / or the GPU 808, the logic unit 820 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 800 to perform one or more methods and / or processes described herein. In an embodiment, the CPU 806, the GPU 808, and / or the logic unit 820 may discretely or jointly perform any combination of methods, processes, and / or portions thereof. The one or more logic units 820 may be a part of and / or integrated into the one or more CPUs 806 and / or the one or more GPUs 808, and / or the one or more logic units 820 may be discrete components of the CPU 806 and / or the GPU 808 or otherwise external thereto. In an embodiment, the one or more logic units 820 may be a processor of the one or more CPUs 806 and / or the one or more GPUs 808.

[0193] Examples of logic unit 820 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), a tensor core (TC), a tensor processing unit (TPU), a pixel vision core (PVC), a vision processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multiprocessor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application specific integrated circuit (ASIC), a floating point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI) or a peripheral component interconnect express (PCIe) element, etc.

[0194] The communication interface 810 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 800 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communications. The communication interface 810 may include components and functions that enable communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communicating over Ethernet or InfiniBand), low power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 820 and / or the communication interface 810 may include one or more data processing units (DPUs) to transmit data received over a network and / or over the interconnect system 802 directly to one or more GPUs 808 (e.g., memory in a GPU 808).

[0195] The I / O port 812 can enable the computing device 800 to be logically coupled to other devices including I / O components 814, presentation components 818, and / or other components, some of which can be built into (e.g., integrated into) the computing device 800. Illustrative I / O components 814 include microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dishes, browsers, printers, wireless devices, and the like. The I / O components 814 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by the user. In some instances, the input can be transmitted to an appropriate network element for further processing. The NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition on and adjacent to the screen, air gestures, head and eye tracking, and touch recognition associated with the display of the computing device 800 (as described in more detail below). The computing device 800 may include a depth camera such as a stereo camera system, an infrared camera system, an RGB camera system, touch screen technology, and a combination of these for gesture detection and recognition. Additionally, computing device 800 may include an accelerometer or gyroscope to enable motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by computing device 800 to render immersive augmented reality or virtual reality.

[0196] The power supply 816 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 816 may provide power to the computing device 800 to enable the components of the computing device 800 to operate.

[0197] The presentation component 818 may include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other display types, or a combination thereof), a speaker, and / or other presentation components. The presentation component 818 may receive data from other components (e.g., GPU 808, CPU 806, DPU, etc.) and output the data (e.g., as an image, video, sound, etc.).

[0198] Sample Data Center

[0199] Fig. 9 An example data center 900 is shown, which may be used in at least one embodiment of the present disclosure. The data center 900 may include a data center infrastructure layer 910, a framework layer 920, a software layer 930, and an application layer 940.

[0200] like Fig. 9As shown, the data center infrastructure layer 910 may include a resource coordinator 912, grouped computing resources 914, and node computing resources ("node CRs") 916 (1)-916 (N), where "N" represents any complete positive integer. In at least one embodiment, the node CRs 916 (1)-916 (N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state drives or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules and cooling modules, etc. In some embodiments, one or more of the node CRs 916 (1)-916 (N) may correspond to a server having one or more of the above computing resources. In addition, in some embodiments, the nodes CR916(1)-916(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more of the nodes CR916(1)-916(N) may correspond to a virtual machine (VM). In some embodiments, one or more of the CR916(1)-916(N) may execute one or more elements of the path surface feature processor 105 described herein, such as but not limited to the heat map generator model 120.

[0201] In at least one embodiment, the computing resources 914 of the grouping can include the separate grouping (not shown) of the node CR916 contained in one or more racks, or many racks (not shown) contained in the data center of each geographical location. The separate grouping of the node CR916 in the computing resources 914 of the grouping can include the computing, network, memory or storage resources that can be configured or assigned to support the grouping of one or more workloads. In at least one embodiment, several node CR916 including CPU, GPU, DPU and / or other processors can be grouped in one or more racks to provide computing resources to support one or more workloads. One or more racks can also include any number of power modules, cooling modules and / or network switches in any combination.

[0202] Resource coordinator 912 may configure or otherwise control one or more nodes CR 916(1)-916(N) and / or grouped computing resources 914. In at least one embodiment, resource coordinator 912 may include a software design infrastructure (SDI) management entity for data center 900. Resource coordinator 912 may include hardware, software, or some combination thereof.

[0203] In at least one embodiment, Fig. 9 As shown, the framework layer 920 may include a job scheduler 933, a configuration manager 934, a resource manager 936, and a distributed file system 938. The framework layer 920 may include a framework that supports the software 932 of the software layer 930 and / or one or more applications 942 of the application layer 940. The software 932 or the application 942 may include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 920 may be, but is not limited to, a free and open source software network application framework, such as Apache Spark, which may utilize the distributed file system 938 for large-scale data processing (e.g., "big data"). TM (hereinafter referred to as "Spark"). In at least one embodiment, the job scheduler 933 may include a Spark driver to facilitate scheduling of workloads supported by various layers of the data center 900. In at least one embodiment, the configuration manager 934 may be able to configure different layers, such as the software layer 930 and the framework layer 920 including Spark and a distributed file system 938 for supporting large-scale data processing. The resource manager 936 can manage cluster or group computing resources mapped to or allocated to support the distributed file system 938 and the job scheduler 933. In at least one embodiment, the cluster or group computing resources may include grouped computing resources 914 at the data center infrastructure layer 910. The resource manager 936 can coordinate with the resource coordinator 912 to manage these mapped or allocated computing resources.

[0204] In at least one embodiment, software 932 included in software layer 930 may include software used by at least a portion of node CRs 916(1)-916(N), grouped computing resources 914, and / or distributed file system 938 of framework layer 920. One or more types of software may include, but are not limited to, Internet web page search software, email virus scanning software, database software, and streaming video content software.

[0205] In at least one embodiment, one or more applications 942 included in the application layer 940 may include one or more types of applications used by at least a portion of the node CRs 916(1)-916(N), the grouped computing resources 914, and / or the distributed file system 938 of the framework layer 920. The one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0206] In at least one embodiment, any of configuration manager 934, resource manager 936, and resource coordinator 912 can implement any number and type of self-modification actions based on any number and type of data acquired in any technically feasible manner. The self-modification actions can relieve the data center operator of data center 900 from making potentially bad configuration decisions and can avoid underutilized and / or underperforming portions of the data center.

[0207] The data center 900 may include tools, services, software, or other resources for training one or more machine learning models or using one or more machine learning models to predict or infer information according to one or more embodiments described herein. For example, a machine learning model may be trained by calculating weight parameters according to a neural network architecture using the software and computing resources described above with respect to the data center 900. In at least one embodiment, by using weight parameters calculated by one or more training techniques, the resources described above with respect to the data center 900 may be used to infer or predict information using a trained machine learning model corresponding to one or more neural networks, such as but not limited to those described herein.

[0208] In at least one embodiment, the data center 900 can use a CPU, an application-specific integrated circuit (ASIC), a GPU, an FPGA, and / or other hardware (or virtual computing resources corresponding thereto) to use the above resources to perform training and / or reasoning. In addition, one or more of the above software and / or hardware resources can be configured as a service to allow users to train or perform information reasoning, such as image recognition, speech recognition, or other artificial intelligence services.

[0209] Example network environment

[0210] A network environment suitable for implementing embodiments of the present disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be configured to communicate with the client device, server, or other device types. Figure 8 The backend device 900 may be implemented on one or more instances of the computing device 800 of the embodiment of the present invention—for example, each device may include similar components, features and / or functions of the computing device 800. In addition, in the case of implementing a backend device (e.g., a server, NAS, etc.), the backend device may be included as part of the data center 900, examples of which are described herein with respect to Fig. 9 Describe in more detail.

[0211] The components of the network environment can communicate with each other through the network, which can be wired, wireless, or both. The network can include multiple networks, or a network of multiple networks. For example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. In the case where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connections.

[0212] Compatible network environments may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment), and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein with respect to the server may be implemented on any number of client devices.

[0213] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. A cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework for supporting software at the software layer and / or one or more applications at the application layer. The software or application may include network-based service software or application programs, respectively. In an embodiment, one or more client devices may use network-based service software or application programs (e.g., by accessing the service software and / or application programs via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open source software network application framework, such as a distributed file system that may be used for large-scale data processing (e.g., "big data").

[0214] A cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these various functions can be distributed across multiple locations from a central or core server (e.g., one or more data centers that can be distributed across states, regions, countries, the world, etc.). If the connection to the user (e.g., client device) is relatively close to an edge server, the core server can assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0215] The client device may include Figure 8 At least some of the components, features, and functionality of the described example computing device 800. By way of example and not limitation, the client device may be embodied as a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smart watch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality head mounted display, a global positioning system (GPS) or device, a video player, a camera, a surveillance device or system, a vehicle, a vessel, an aircraft, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or system, an entertainment system, an in-vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of these described devices, or any other suitable device.

[0216] The present disclosure may be described in the general context of machine-usable instructions or computer codes executed by a computer or other machine such as a personal digital assistant or other handheld device, including computer-executable instructions such as program modules. Typically, program modules including routines, programs, objects, components, data structures, etc. refer to codes that perform specific tasks or implement specific abstract data types. The present disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. The present disclosure may also be practiced in a distributed computing environment in which tasks are performed by remote processing devices linked through a communication network.

[0217] As used herein, the description of "and / or" about two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B and C. In addition, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0218] The subject matter of the present disclosure is described in detail herein to meet statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. On the contrary, the inventors have contemplated that the claimed subject matter may also be embodied in other ways to include steps different from the steps described herein in conjunction with other current or future technologies or combinations of similar steps. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of the method employed, these terms should not be interpreted as implying any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

Claims

1. A system comprising: One or more processing units for: generating at least one tile image of a path surface based at least on an orthogonal projection of aggregated path surface data onto a baseline surface representation of the path surface, wherein the aggregated path surface data is obtained based at least on sensor data associated with the path surface; applying the at least one tile image to a generator model to perform semantic segmentation on the at least one tile image, the semantic segmentation classifying one or more pixels of the at least one tile image based at least on one or more road marking characteristics; and At least one heat map image is generated based at least on the one or more pixels of the at least one tile image being classified as representing the one or more road marking characteristics.

2. The system according to claim 1, wherein: The aggregated path surface data includes at least one of LIDAR intensity point cloud data, LIDAR depth point cloud data, or path surface color data.

3. The system according to claim 1, wherein: The aggregated path surface data includes at least a composite point cloud of LIDAR data captured from at least one sensor on a mobile platform at a plurality of locations along a segment of the path surface.

4. The system according to claim 1, wherein: The aggregated path surface data also includes location data indicating a location at which the sensor data was captured.

5. The system according to claim 1, wherein: The one or more processing units are further configured to: downsampling the at least one tile image into a multi-scale image representation comprising a plurality of resolution layers using an encoder model of the generator model; performing one or more multi-scale feature extractions to estimate one or more segmentations of road marking landmarks from the plurality of resolution layers using a multi-scale feature extraction functional layer of the generator model; as well as The one or more segmentations of road marking landmarks are upsampled using a decoder model of the generator model to generate the at least one heat map image.

6. The system according to claim 5, wherein: The one or more processing units are further configured to: One or more context features are extracted using a context extraction function layer of the generator model, the context features indicating at least one contextual association between the one or more segmentations of road marking landmarks.

7. The system according to claim 6, wherein: The context extraction functional layer determines the at least one context association using at least one attention mechanism based on one or more attention maps.

8. The system according to claim 6, wherein: The context extraction functional layer determines the at least one context association using one or more Lambda layers of a Lambda Network.

9. The system according to claim 1, wherein: The one or more processing units are further configured to: Training the generator model using feedback comprising at least one loss determined using a discriminator model based on a generative adversarial network (GAN) training architecture; and The at least one loss is calculated based on a first input comprising at least one training heat map image and a second input comprising at least one heat map image with a ground truth annotation.

10. The system according to claim 9, wherein: The one or more processing units are further configured to: generating at least one training mask based at least on the at least one ground truth annotated heatmap image; as well as The generator model is trained based at least on road marking features of training data images located within the at least one training mask.

11. The system according to claim 1, wherein: The one or more pixels of the at least one heat map image represent one or more road marking positions of the one or more road marking line characteristics, the one or more road marking line characteristics comprising at least one of the following: Solid lane markings, dashed lane markings, white lane markings or yellow lane markings.

12. The system of claim 1, wherein: The at least one heat map image includes an indication of a line marking angle of at least one lane marking.

13. The system of claim 1, wherein: The one or more processing units are further configured to: A map of the path surface is generated based at least on the at least one heat map image.

14. The system of claim 1, wherein: The one or more processing units are included in at least one of the following: control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; A system for performing simulation operations; Systems for performing digital twin operations; A system for performing light transport simulations; A system for performing collaborative content creation of three-dimensional assets; A system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; Systems for performing deep learning operations; A system for performing real-time streaming; Systems implemented using edge devices; Systems implemented using robots; Systems for performing conversational AI operations; Systems for generating synthetic data; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; Systems for performing generative AI operations; A system implemented at least in part using a language model; or A system implemented at least in part using cloud computing resources.

15. A processor, comprising: One or more processing units for: generating at least one tile image of a path surface based at least on an orthogonal projection of aggregated path surface data onto a baseline surface representation of the path surface, wherein the aggregated path surface data comprises at least one of LIDAR intensity point cloud data, LIDAR depth point cloud data, or path surface color data; applying the at least one tile image to a generator model to perform semantic segmentation on the at least one tile image to classify pixels of the at least one tile image based at least on road marking characteristics; and At least one heat map image for at least one road marking characteristic is generated based at least on the classification of pixels of the at least one tile image as representing a road marking.

16. The processor of claim 15, wherein: The one or more processing units are further configured to: downsampling the at least one tile image into a multi-scale image representation comprising a plurality of resolution layers using an encoder model of the generator model; performing one or more multi-scale feature extractions to estimate one or more segmentations of road marking landmarks from the plurality of resolution layers using a multi-scale feature extraction functional layer of the generator model; extracting, using a context extraction function layer of the generator model, one or more context features indicative of at least one contextual association between the one or more segmentations of road marking landmarks; as well as The one or more segmentations of road marking landmarks are upsampled using a decoder model of the generator model to generate the at least one heat map image.

17. The processor of claim 15, wherein: The aggregated path surface data includes at least a composite point cloud of LIDAR data captured from at least one sensor on a mobile platform at a plurality of locations along at least a segment of the path surface.

18. The processor of claim 15, wherein: One or more pixels of the at least one heat map image represent a road marking location of a road marking characteristic, the road marking characteristic comprising at least one of: a solid lane marking, a dashed lane marking, a white lane marking, a yellow lane marking; as well as Therein, the at least one heat map image comprises an indication of a road marking angle of at least one road marking.

19. The processor of claim 15, wherein: The one or more processing units are included in at least one of the following: control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; A system for performing simulation operations; Systems for performing digital twin operations; A system for performing light transport simulations; A system for performing collaborative content creation of three-dimensional assets; A system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; Systems for performing deep learning operations; A system for performing real-time streaming; Systems implemented using edge devices; Systems implemented using robots; Systems for performing conversational AI operations; Systems for generating synthetic data; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; Systems for performing generative AI operations; A system implemented at least in part using a language model; or A system implemented at least in part using cloud computing resources.

20. A method comprising: At least one heat map image for at least one lane marking feature is generated based at least on applying at least one tile image of a path surface to a generator model to perform semantic segmentation to classify pixels of the at least one tile image, the at least one tile image of the path surface being based on an orthogonal projection of aggregated path surface data onto a baseline surface representation of the path surface, wherein the aggregated path surface data includes at least one of LIDAR intensity point cloud data, LIDAR depth point cloud data, or path surface color data.

Images