EFFICIENT PIXEL DENSITY MEASUREMENT OF COMBINED IMAGES

Abstract

A first synthetic image including a first pattern and a second synthetic image including a second pattern are processed to generate a merged image comprising a modified version of the first synthetic image and a modified version of the second synthetic image. One or more features of the first pattern are deformed in the modified version of the first synthetic image, and one or more features of the second pattern are deformed in the modified version of the second synthetic image. Pixel areas are determined for each feature of the modified version of the first synthetic image and for each feature of the modified version of the second synthetic image. Feature densities are determined for one or more regions of the merged image based on the determined pixel areas.

Description

TECHNICAL FIELD

[0001] Embodiments disclosed herein relate generally to image processing and, more particularly, to efficient pixel density measurements of stitched images. BACKGROUND

[0002] As there is an increase in the use of technology for fully automated or semi-automated tasks, which may relate to autonomous driving and robotics, there is a corresponding increase in the desire to provide accurate information relevant to the operation of that technology. This may include, for example, capturing live video of an environment in which that technology is operating and providing an accurate video representation. To provide a view-consistent representation, video data captured from multiple cameras can be combined via a process such as image stitching to create a stitched or composite representation. In some cases, deformation may be introduced during the image stitching process, which may affect pixel density.Traditionally, known solutions relied on real-world captures and extensive manual measurements to determine pixel density measurements. SUMMARY

[0003] The invention is defined by the claims. To illustrate the invention, aspects and embodiments are described herein, which may or may not be within the scope of the claims.

[0004] It is disclosed that a first synthetic image including a first pattern and a second synthetic image including a second pattern are processed to generate a merged image comprising a modified version of the first synthetic image and a modified version of the second synthetic image. One or more features of the first pattern are deformed in the modified version of the first synthetic image, and one or more features of the second pattern are deformed in the modified version of the second synthetic image. Pixel areas are determined for each feature of the modified version of the first synthetic image and for each feature of the modified version of the second synthetic image. Feature densities are determined for one or more regions of the merged image based on the determined pixel areas.

[0005] The disclosure extends to any novel aspects or features described and / or illustrated herein.

[0006] Further features of the disclosure are characterized by the independent and dependent claims.

[0007] Any feature in one aspect of the disclosure may be applied to other aspects of the disclosure in any suitable combination. In particular, method aspects may be applied to device or system aspects, and vice versa.

[0008] Furthermore, features implemented in hardware may also be implemented in software, and vice versa. Any reference to software and hardware features in this description should be construed accordingly.

[0009] Any system or device feature described herein may also be provided as a method feature, and vice versa. Functionally described system and / or device aspects (including means-plus-function features) may alternatively be expressed in terms of their corresponding structure, such as a suitably programmed processor and associated memory.

[0010] It should also be understood that certain combinations of the various features described and defined in any aspects of the disclosure may be implemented and / or provided and / or used independently of one another.

[0011] This disclosure also provides computer programs and computer program products comprising software code adapted, when executed on a computing device, to perform any of the methods described herein and / or to embody any of the device and system features described herein, including any or all of the substeps of a method.

[0012] The disclosure also provides a computer or computing system (including networked or distributed systems) having an operating system that supports a computer program for performing any of the methods described herein and / or embodying any of the device or system features described herein.

[0013] The disclosure also provides a computer-readable medium having stored thereon one or more of the aforementioned computer programs.

[0014] The disclosure also provides a signal carrying one or more of the aforementioned computer programs.

[0015] The disclosure extends to methods and / or devices and / or systems as described herein with reference to the accompanying drawings.

[0016] Aspects and embodiments of the disclosure will now be described, by way of example only, with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 illustrates an exemplary computing environment for determining noise profiles for image sensors based on image data, according to at least one embodiment of the present disclosure. Fig. 2A and Fig. 2B illustrate merged images that may be generated in accordance with at least one embodiment. Fig. 3 is a block diagram illustrating an exemplary pixel density component for determining pixel densities according to at least one embodiment of the present disclosure. Fig. 4A illustrates an example synthetic image provided as input to an image stitching pipeline, according to at least one embodiment of the present disclosure. Fig. 4B illustrates an exemplary synthetic stitched image received as output from an image stitching pipeline, according to at least one embodiment of the present disclosure. Fig. 5A illustrates an exemplary pixel density map associated with a synthetic stitched image produced by an image stitching pipeline, according to at least one embodiment of the present disclosure. Fig. 5B illustrates an example vector map associated with a synthetic stitched image produced by an image stitching pipeline, according to at least one embodiment of the present disclosure. Fig. 6 illustrates a flowchart of an exemplary method for measuring the pixel density of a synthetic stitched image, according to at least one embodiment of the present disclosure. Fig. 7 illustrates an example network configuration 700, according to at least one embodiment of the present disclosure, that may be used to provide, generate, modify, encode, and / or transmit content. Fig. 8A illustrates an example of an autonomous vehicle, according to at least one embodiment; Fig. Figure 8B illustrates an example of camera locations and fields of view for the autonomous vehicle of Fig. 8A, according to at least one embodiment; Fig. Figure 8C is a block diagram illustrating an example system architecture for the autonomous vehicle of Fig. 8A illustrates, according to at least one embodiment; Fig. 8D is a diagram illustrating a system for communication between one or more cloud-based servers and the autonomous vehicle of Fig. 8A illustrates, according to at least one embodiment; Fig. 9 is a block diagram of a computer system, according to at least one embodiment. Fig. 10 is a block diagram of a computer system, according to at least one embodiment. Fig. 11 illustrates at least portions of a graphics processor according to one embodiment; Fig. 12 illustrates at least portions of a graphics processor according to one embodiment. DETAILED DESCRIPTION

[0017] Camera image sensors are used in a wide variety of applications across various industries. For example, smartphones, video surveillance systems, automobiles, medical imaging devices, and the like use image sensors to capture image and video data for various purposes. In some cases, video data captured by multiple cameras can be combined through a process such as image stitching to create a combined or composite representation of an environment. Image stitching can be used by a variety of applications to provide a more comprehensive view of a scene and enhance understanding of the context in which objects exist. For example, image stitching can be used by medical imaging, surveillance, computer vision, and automotive systems for many different purposes.

[0018] In some cases, deformation may be introduced during the stitching process, which can affect pixel density. Some stitching algorithms may deform or stretch images to align the images and create a seamless stitched image, resulting in variations in pixel density on the stitched image. For example, a pixel corresponding to a central region of a sensing device may map to a smaller region after the stitching process, while a pixel corresponding to an edge region of the sensing device may map to a larger region after the stitching process, resulting in uneven pixel densities on the final stitched image. Understanding and managing pixel densities of stitched images is critical to achieving a seamless and visually coherent stitched image.For example, pixel density information can be used during the image alignment phase of a stitching algorithm to ensure precise feature registration and avoid distortion. Furthermore, the quality of a stitched image can be assessed based on pixel density uniformity. Large deviations from a relatively consistent pixel density can indicate regions of a stitched image for additional refinement or adjustment during the stitching process. Accordingly, accurate and efficient measurement of pixel densities on a stitched image can be useful for image stitching.

[0019] In some cases, approaches to measuring pixel densities on a stitched image use a point array image during a capture device calibration process. For example, the point array image is placed over an image sensor of the capture device. The point array image may be a grid of points or a checkerboard pattern, which can provide a reference point for understanding distortions introduced during the image stitching process. Images may be acquired by multiple capture devices with the point array image placed over associated image sensors. After a stitched output image is generated, the points may be deformed (e.g., enlarged, reoriented, etc.) as a result of the stitching algorithm. Distances between centroids of the points before and after image stitching can be calculated manually.The ratio of the point-to-point distance of the stitched image divided by its original value can provide an estimate of the pixel density. However, such an approach is manual (e.g., involves physically placing the point array image over the image sensor during a calibration process) and labor-intensive. Furthermore, the method may produce results that vary and are difficult to reproduce in subsequent measurements. Furthermore, the use of the acquisition device can be hampered by manual scene setup and human error introduced during the manual calibration process.

[0020] In accordance with embodiments described herein, a presented technique includes a method for automatically measuring the pixel density of a stitched image. To measure the pixel density, synthetic images may be generated and provided as input to an image stitching system. The synthetic images may be automatically generated by the system or provided, for example, by a user / administrator associated with the system. The synthetic images may include a fixed pattern of feature shapes, such as a grid of equidistant dots or a grid of equidistant squares. Each feature of a synthetic image may correspond to a region of images captured by the associated capture device. The synthetic image may be duplicated for each capture device in the system and provided as input to an associated image stitching algorithm.The stitching algorithm can produce a stitched output image including regions associated with each of the input synthetic images. The stitched output image can include deformed shapes that are magnified and / or reoriented according to parameters associated with the stitching algorithm. Each of the deformed shapes can correspond to the shapes of the synthetic images provided as input. The system can measure the area of ​​each shape in the input images and corresponding shapes in the stitched output image. Ratios between areas of input and output shapes can indicate pixel densities of a region of output images stitched by the stitching algorithm relative to a pixel density of input images.In an illustrative example, a first feature associated with a first synthetic input image may have a pixel range of 10 pixels. The stitching algorithm may produce a corresponding deformed feature with a pixel range of 16 pixels. Accordingly, the pixel density of the deformed feature may be 1.6 (16 / 10). In some embodiments, the pixel density may be a magnification level (e.g., 1.6x magnification) of the deformed feature.

[0021] In some embodiments, to determine regions of features in the stitched output images, the system may first segment each feature using one or more computer vision techniques. For example, thresholding techniques may be applied to the stitched output image to produce a binary image. Pixels with intensity values ​​above a threshold intensity level may be labeled as foreground pixels, and pixels with intensity values ​​below the threshold intensity level may be labeled as background pixels. The result is a binary image in which one or more shapes in the binary image are labeled as foreground pixels and everything else is labeled as background pixels. The binary image may be provided as input to a connected component algorithm. The connected component algorithm may produce an image in which each shape is labeled as a separate connected component.Shape properties for the connected components can be determined based on one or more shape analysis algorithms. For example, an orientation, area, and / or eccentricity can be determined for each connected component using one or more Hu moments. Pixel density / magnification ratios can be determined based on the calculated areas.

[0022] In some embodiments, a pixel density map may be created by combining the magnification and orientation of each deformed shape. Each point of the pixel density map may be associated with a deformed feature and may indicate a magnification level and an orientation of the deformed feature.

[0023] Some solutions do not use synthetic images (e.g., they do not use synthetic images with fixed patterns). For example, traditional solutions do not use synthetic images to automatically determine pixel densities of output images from an image stitching pipeline. Instead, these solutions rely on real-world image acquisition and extensive manual measurements using a grid array. Such a manual system is labor-intensive and difficult to scale across many image acquisition devices.

[0024] The systems and methods described herein may be used by, among others, non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), guided and unguided robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, flying vehicles, boats, shuttle vehicles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater vehicles, drones, and / or other types of vehicles.Furthermore, the systems and methods described herein may be used for a variety of purposes, including, but not limited to, machine control, machine locomotion, machine propulsion, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actuator simulation and / or digital twinning, data center processing, conversational artificial intelligence (AI), light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, and / or other suitable applications.

[0025] Disclosed embodiments may be included in a number of different systems, such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented using a robot, antenna systems, media systems, boat systems, intelligent area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twinning operations, systems implemented using an edge device, systems incorporating one or more virtual machines (VMs), systems for performing one or more generative AI operations, systems for performing synthetic data generation operations, systems implemented at least partially in a data center,Systems for performing conversational AI operations, systems that implement one or more large language models (LLMs), systems that implement one or more visual language models (VLMs), systems for hosting real-time streaming applications, systems for presenting one or more virtual reality content, augmented reality content, or mixed reality content, systems for performing light transport simulation, systems for performing collaborative content creation for 3D assets, systems that are implemented at least in part using cloud computing resources, and / or other types of systems.

[0026] It will be appreciated that aspects and implementations of the present disclosure are often described with respect to autonomous vehicles by way of example and not limitation. It should be noted that aspects and implementations of the present disclosure may be applied to numerous systems, such as image stitching pipelines implemented within surveillance systems, medical imaging systems, robotics, virtual reality, augmented reality, and the like.

[0027] Fig. 1 illustrates an example system 100 that may be used according to various embodiments. In this example, a set of images 102 is received as input, which may be received from multiple cameras associated with a given device or system or positioned relative to a given environment, among other such options. In at least one embodiment, these images will represent two or more views of an environment that overlap at most partially. These images may represent a full or partial view of a scene of a location or environment. This image data may include "live" data streamed or transmitted shortly after image capture and / or may include image data previously captured and provided in a more offline manner.Such an image may also be in different color spaces, such as Red, Green, Blue (RGB), Commission on Illumination lab (CIELAB), Cyan, Magenta, Yellow, Key (Black) (CMYK), Luminance, Chrominance (YIQ), Luminance, Chrominance Blue, Chrominance Yellow (YCbCr), Luminance Blue, Projection Red Projection (YUV), Hue, Saturation Value (HSV) or Hue, Saturation, Light (HSL), among other such options.

[0028] Images 102 may be provided to an image stitching pipeline 101, which may use these input images 102 to generate an output view or stitched / composited image, or a video stream, for presentation via at least one display 144, such as a monitor, a projector, or a wearable display, among other such options. In this example, images 102 may be provided to a de-skew component 104. De-skew may refer to the process of correcting distortions, such as lens distortion, perspective distortion, and the like. In at least one embodiment, the de-skew component 104 may apply one or more geometric transformations (e.g., affine transformations, homography transformations, polynomial transformations, etc.) to images 102 to correct such distortion.

[0029] Dewarped images may be provided to a stitching component 110, which will attempt to stitch multiple images together to generate a composite representation of the images 102. The stitching component may use any of a number of different stitching algorithms 112 or compositing algorithms, wherein various blending techniques 114 or other image manipulation techniques may be performed.

[0030] In at least one embodiment, the image stitching pipeline 101 may use various intrinsic and extrinsic parameters of the cameras, at least to the extent that values ​​for these parameters are known and available from a camera database or other such location, to correctly assign the image data for stitching. This may, for example, include information such as the relative poses or orientation of these different cameras, so that at least an initial stitching position and orientation can be determined for each image. It should be understood that for image stitching pipelines that receive sequences of images or streams of video frame data, the images or frames to be stitched are those that correspond to, or were acquired at, the same time or approximately the same time, at least to the extent that such acquisition can be synchronized.

[0031] The stitching component 110 may also use values ​​for one or more stitching 112 parameters 116, which may be stored in a parameter database or other such location. Values ​​for these parameters may determine aspects of how component images are stitched together, which may relate to weights or locations for blending 114, and other such aspects. Example stitching parameters include, but are not limited to: a fade-in method (e.g., alpha fade-in, multi-band fade-in, or Poisson fade-in), a fade-in width, a fade-in alpha map shape (e.g., angle-based or perpendicular-based), fade-in weights in different bands for multi-band fade-in, weights in different color channels in different color spaces (e.g., RGB, CIELAB, CMYK, YIQ, YCbCr, YUV, HSV, or HSL), seam type (e.g.,diagonal seam, vertical seam, or horizontal seam) and / or seam location. The stitching component 110 can use values ​​for these various parameters with the input images 102 to generate a composite image, or stitched image, that provides a single representation of an environment, scene, or location at a point in time, such as the "current" point in time, while accounting for some latency in transmission and processing.

[0032] If a scene is not harmonized between cameras, it may be desirable to use a larger fade weight or radius (e.g., 200) to provide a smoother transition between data from images. For example, if it is a highly textured scene with multiple buildings and / or edges, it may be desirable to use a smaller fade weight (e.g., 2) to avoid ghosting and other artifacts due to misalignment between cameras. Single-band blending can also be used when images are blended in only one band (or a subset) of multiple bands. In at least one embodiment, images can be decomposed into different frequency bands or components, or can be converted to any suitable color space (e.g., RGB, CIELAB, CMYK, YIQ, YCbCr, YUV, HSV, HSL, etc.).) and different fade-in weights can be used for each of these bands, components or color channels.

[0033] In at least one embodiment, image stitching pipeline 101 may project a stitched image generated by stitching component 110 onto a cylindrical surface to generate a bowl-view projection 120. A bowl-view projection 120 is a type of mapping from a three-dimensional (3D) scene to a two-dimensional (2D) image. In a bowl-view projection 120, straight lines in the 3D scene are not preserved as straight lines in the resulting image. Instead, they appear curved, creating a wide-angle effect.In at least one embodiment, a view generator 130 may use the bowl-view projection 120 and camera calculation parameters, such as display configurations 140, to map the bowl-view projection into different views corresponding, for example, to a "bowl" view 132 in a projection space, a free view 134 in a projection space, or a top-down view 136 in a reaction space. The image stitching pipeline 101 may generate an output view or a composite / stitched image, or a video stream, for presentation via at least one display device 144, such as a monitor, a projector, or a wearable display, among other such options. Essentially, the image stitching pipeline 101 may generate a composite image or a stitched image as output based on a set of images 102 received at an input, as described below with respect to FIG. Fig. 2A and Fig. 2B illustrates, generate.

[0034] Fig. 2A illustrates a set of four images 200 that may be captured by respective cameras on or associated with an autonomous vehicle or other such system or device, according to various embodiments. In this example, the cameras have different, but partially overlapping, fields of view, so that the images may be stitched together without gap filling or additional image data generation, although it should be understood that there are many other situations where cameras do not produce images with at least partially overlapping views that may benefit from such tasks. In this example, each camera may have various intrinsic or extrinsic values ​​that may affect the appearance of a captured image, where these values ​​may relate to field of view, optical center, focal length, and / or camera pose, among other such options.As mentioned, it may be desirable to generate a single, consistent view of this surrounding environment based at least in part on these captured images 200. This may involve, for example, generating a stitched / composite image 250, as shown in FIG. Fig. 2B. As illustrated, such a composite image can provide a single, consistent representation of the environment, whether a full 360-degree view or at least a portion of the angular range. Such a view can be presented as a single view showing the entire image 250, or portions of the image can be displayed at different times, with the view being controllable by a user. In this example, an angular shape alpha map is used instead of, for example, a perpendicular shape alpha map to at least avoid the alpha discontinuity problem observed with traditional stitching techniques. Fade-in region color continuity can be improved by extending the fade-in region(s) to hide a color discontinuity problem around corners or other such features. As in Fig. 2B, this composite image 250, which is obtained by joining the individual images 200 of Fig. 2A may contain various artifacts or elements that are not realistic to the environment and indicate that the composite image was not captured as a single image. These may include, for example, one or more seams 252 indicating transitions between data from different frames, as well as seam misalignment, ghosting, and / or vanishing, among other such options. This may also include areas or points of discontinuity 254 where features of the data from different images are not aligned, which may result, for example, from misalignment or calibration of the respective cameras or changes or differences in the parameters of those cameras, among other such options.

[0035] Returning to Fig. 1, in some cases, a deformation (e.g., enlargement, reduction, distortion, etc.) may be introduced at one or more stages of the image stitching pipeline 101 such that pixel densities on the stitched image vary. Some stitching algorithms may deform or stretch images to align the images and create a seamless stitched image, resulting in variations in the pixel density on the stitched image. For example, the deskewing component 104, stitching component 110, and / or view generator 130 may apply one or more geometric transformations that may affect the pixel density in different regions of the resulting stitched image. Regions of the stitched image that are greatly stretched or enlarged may have an overall lower pixel density than regions of the stitched image that are reduced in size.

[0036] In at least one embodiment, system 100 may include a pixel density component 150 to measure pixel densities on a synthetic stitched image produced by image stitching pipeline 101. In this example, pixel density component 150 may provide a set of synthetic images 152 as input to image stitching pipeline 101. The set of synthetic images may each include a fixed pattern of features / shapes, such as a grid of equidistant points or a grid of equidistance squares. In at least one embodiment, each synthetic image in the set may illustrate the same pattern of features. Synthetic images 152 may be generated by image stitching component 150 or retrieved from an image database or other such location.

[0037] The pixel density component 150 may provide synthetic images to the image stitching pipeline 101, which may use these input synthetic images to generate a stitched synthetic image. In at least one embodiment, the image stitching pipeline may provide a stitched synthetic image to the pixel density component 150 after any stage of the pixel density analysis pipeline. For example, the image stitching pipeline 101 may provide a stitched output synthetic image to the output dewarping 104, the stitching component 110, the view generator 130, or any other stage of the image stitching pipeline 101.

[0038] The stitched output image may include deformed features that are magnified and / or reoriented according to parameters (e.g., parameters 116, display configurations 140, etc.) associated with the image stitching pipeline 101. Each of the deformed features may correspond to features / regions of the respective synthetic images 152. The pixel density component 150 may be operable to measure the area of ​​each feature in the input images and corresponding features in the stitched output image. Ratios between areas of input and output features may indicate pixel densities of a region of output images stitched by the image stitching pipeline 101 relative to a pixel density of input images 102.For example, a deformation of features of synthetic images 152 may indicate the same deformation of corresponding regions of images 102 provided as input to image stitching pipeline 101. Pixel density component 150 may determine pixel densities (e.g., magnifications, orientations, etc.) according to one or more techniques described below with respect to

[0044] FIG. Fig. 3 described.

[0039] Fig. 3 is a block diagram 300 illustrating an exemplary pixel density component for determining pixel densities according to at least one embodiment of the present disclosure. In at least one embodiment, the pixel density component 310 may include a segmentation component 312, a feature analyzer 314, and a map generator 316. In at least one embodiment, the pixel density component 310 may be the Fig. 1. In at least one embodiment, the pixel density component may be connected (e.g., via a network, via a communication bus, etc.) to a memory 330. In at least one embodiment, the pixel density component 310 may be connected (e.g., via a network, via a communication bus, etc.) to the image stitching pipeline 101.

[0040] The pixel density component 310 may provide a set of synthetic images 334 as input to the image stitching pipeline 101 and provide a synthetic stitched image 334 as output from the image stitching pipeline 101. Synthetic images 334 may each include a fixed pattern of features / shapes, and the synthetic stitched image 334 may include corresponding features / shapes deformed as a result of one or more stages of the image stitching pipeline 101. An exemplary synthetic image 332 is described below with respect to Fig. 4A and an exemplary synthetic merged image 334 is described below with respect to Fig. 4B illustrated.

[0041] Fig. 4A illustrates an example synthetic image 400 provided as input to an image stitching pipeline, such as image stitching pipeline 101. Synthetic image 400 includes a grid or pattern of evenly spaced points (referred to herein as "features"). It should be appreciated that the pattern of points is used herein as exemplary and not limiting, with the understanding that any fixed pattern of features (e.g., a fixed pattern of squares, crosses, stars, etc.) may be used herein. Each point illustrated in synthetic image 400 may have a known equal area, such as a pixel area of ​​10 pixels.

[0042] Fig. 4B illustrates an example synthetic stitched image 450 received as output from an image stitching pipeline. Specifically, the synthetic stitched image 450 may be the product of the same four synthetic images 400 (each associated with a different camera facing a different direction, such as forward, backward, left, and right of an automobile) stitched together using an image stitching pipeline, such as image stitching pipeline 101. As shown in image 450, several points have been deformed (e.g., enlarged, reoriented, etc.) as a result of one or more components of the image stitching pipeline.

[0043] Returning to Fig. 3, the pixel density component 310 may be operated to determine pixel density measurements based on ratios between areas (e.g., area in pixels) of features of the synthetic images 332 and corresponding areas (e.g., area in pixels) of features of synthetic stitched images 334. To determine feature areas of the synthetic stitched image 334, the pixel density component 310 may provide the synthetic stitched image 334 as input to the segmentation component 312.

[0044] To determine regions of features in the synthetic stitched image 332, the segmentation component 312 may first segment each feature using one or more computer vision techniques. For example, thresholding techniques may be applied to the synthetic stitched image 334 to produce a binary image. The segmentation component 310 may iterate through each pixel of the synthetic stitched image (row by row, column by column, etc.) and label each pixel as a foreground pixel (e.g., labeled as "1") or background pixel (e.g., labeled as "0") to produce a binary image. Any pixel at or above a threshold color value may be labeled as a foreground pixel corresponding to a feature, while any pixel below a threshold color value may be labeled as a background pixel not corresponding to a feature.Since there is no noise in the synthetic stitched image 334, the thresholding process can accurately identify features of the synthetic stitched image 334.

[0045] The result of the thresholding process is a binary image in which shapes / features are labeled as foreground pixels and everything else is labeled as background pixels. In at least one embodiment, the segmentation component 312 may further use a connected component algorithm to produce an image in which each feature / shape of the binary image is labeled as a separate connected component. A connected component is a set of pixels that are contiguous and share a common property or characteristic, such as a common binary label. In at least one embodiment, the segmentation component 312 may utilize one or more connected component algorithms to identify deformed features in the synthetic stitched image 334.In one illustrative example, a connected component algorithm may define a connectivity criterion for determining the connectivity between pixels of the binary image. The connectivity criterion may include, for example, 4-way connectivity (e.g., pixels sharing an edge are considered connected), 8-way connectivity (pixels sharing an edge or corner are considered connected), and the like. Starting at a predetermined pixel of the binary image, the algorithm may recursively examine neighboring pixels to identify connected components. Neighboring pixels may be added to a connected component if they meet the connectivity criterion. This process may continue until no neighboring pixels meet the connectivity criterion, at which point the algorithm may proceed to the next unlabeled pixel.Once a connected component has been identified, the algorithm may assign a unique identifier to the component. In at least one embodiment, each connected component may be a deformed feature in the synthetic stitched image 334. The segmentation component 312 may provide the synthetic stitched image 334 with labeled connected components to the feature analyzer 314 for further processing. It will be appreciated that the technique described with respect to the segmentation component 312 is used by way of example and not limitation. For example, the segmentation component 312 may utilize various segmentation techniques (e.g., thresholding, one or more machine learning models, etc.) in conjunction with one or more connected component algorithms with varying criteria (e.g., 8-way connectivity, 4-way connectivity, etc.).) to produce a synthetic merged image 334 with uniquely labeled features.

[0046] The feature analyzer 314 may use one or more shape analysis algorithms to determine statistics / shape properties (e.g., symmetry, skewness, dilation, etc.) for features (e.g., connected components) of the synthetic stitched image 334. For example, features of the synthetic stitched image may be mathematically described by calculating one or more Hu-moment statistics. Hu-moments include a set of seven invariant (translation-, rotation-, and scale-invariant) image moments for shape analysis and recognition. In computer vision, image moments are quantitative descriptors that can be used to characterize the shape of an object in an image. Image moments may capture the area of ​​the object, the centroid (e.g., the center coordinate of the object), the orientation of the object, among other properties and statistics.For example, the moment of the i-th magnitude of an image I can be defined by the following equation: . Mij=∑x∑yxiyjI(x,y)

[0047] Where i and j represent the magnitude of the moment, and I(x, y) denotes the intensity value of the pixel at coordinates (x, y), so that only non-zero pixel values ​​are counted in the summation. For example, if i = 0 (the zeroth magnitude moment), the labeled synthetic merged image 334, calculated according to equation (1), may be a summation of non-zero pixels in the synthetic merged image 334 (e.g., the number of pixels corresponding to a deformed feature of the image).

[0048] In another example, central moments can be calculated and used to quantify the spatial distribution of shapes / features in an image. Central moments are invariant under translation, meaning a central moment is independent of the shape / feature's location within the image. The central moment of the p-th order and the image I can be defined by the following equation: mpq=∑x∑y(x−x¯)p(y−y¯)qI(x,y)

[0049] Where p and q represent the magnitude of the moment and I(x, y) denotes the intensity value of the pixel at coordinates (x, y), x is the average value of x on image I and y is the average value of y on image I.

[0050] In at least one embodiment, the central moment of zero magnitude (m 00) represent the area of ​​a deformed feature / shape of the synthetic stitched image 334. Each deformed feature / shape of the synthetic stitched image 334 may correspond to a feature of the synthetic image 332 with a predetermined area. The feature analyzer 314 may calculate a ratio of the area of ​​features in the synthetic stitched image 334 and the known area of ​​corresponding features in the synthetic images 332. In an illustrative example, the feature analyzer 314 may calculate the area of ​​a deformed feature of the synthetic stitched image 334 to be 16 pixels using equation (2). A predetermined area of ​​the corresponding feature in the synthetic image may be a pixel area of ​​10 pixels. Accordingly, the feature analyzer can calculate the pixel area ratio of the deformed feature to be 1.6 (16 / 10).In some embodiments, the pixel area ratio may indicate a magnification level (e.g., 1.6x magnification) of the deformed feature. In at least one embodiment, feature analyzer 314 may determine pixel area ratios for each feature in synthetic stitched image 334 and store the pixel area ratios (magnifications) as a measurement included in measurements 336.

[0051] In at least one embodiment, feature analyzer 314 may calculate the orientation of deformed features using the Hu moments described above. Orientation describes the direction of a feature's major axis (the axis passes through the centroid and along the widest part of the feature). In an illustrative example, feature analyzer 314 may estimate the orientation of a feature's shape using the following covariance matrix: C=1m00[m20m11m11m02]

[0052] Eigenvectors of this matrix can correspond to major and minor axes of the image density. The orientation can therefore be extracted from the angle of the eigenvector associated with the largest eigenvalue to the axis closest to that eigenvector. This orientation angle, θ, can be calculated using the following equation: tan 2θ=(2m112m20−2m02)

[0053] Where m 11 is the central moment of the first magnitude, which represents a covariance between the x- and y-coordinates, m 20 and m 02 are the second-order central moments representing the variances along the x- and y-axes, respectively, and θ is the orientation angle or tilt of the feature's shape relative to the coordinate axes. In at least one embodiment, the orientation angle θ may be included as a measurement within measurements 336.

[0054] In at least one embodiment, feature analyzer 314 may calculate the eccentricity of deformed features using the covariance matrix (3) above. Eccentricity is a measure of how stretched or stretched a feature's shape is. Feature analyzer 314 may determine the eccentricity of a feature using eigenvalues ​​of the covariance matrix (3). For example, the length of a semi-principal axis is given by 2λ1 given and the length of a semi-major axis 2λ2 In at least one embodiment, eccentricity may be included as a measurement within measurements 336.

[0055] In at least one embodiment, the map generator 316 may be operated to generate a pixel density map of the synthetic stitched image 334 using one or more measurements 336, described below with respect to Fig. 5A and Fig. 5B. For example, map generator 316 may generate a pixel density map by combining magnifications and orientations of each deformed feature.

[0056] Fig. 5A illustrates an exemplary pixel density map 500 associated with a synthetic stitched image produced by an image stitching pipeline, according to at least one embodiment of the present disclosure. Each deformed point in the pixel density map 500 may correspond to a feature of a raw synthetic image, such as the synthetic image 400 of Fig. 4, that has been magnified, stretched, and / or otherwise deformed by the image stitching pipeline. Each deformed point in the pixel density map 500 can be quantitatively described using a magnification level and orientation calculated according to feature / shape analysis techniques described above.

[0057] Fig. 5B illustrates an exemplary vector map 550 associated with a synthetic stitched image produced by an image stitching pipeline, according to at least one embodiment of the present disclosure. Each vector in the vector map 500 may correspond to a feature of a raw synthetic image, such as the synthetic image 400 of Fig. 4, that has been magnified, stretched, and / or otherwise deformed by the image stitching pipeline. Each vector in the pixel density map 500 can be quantitatively described using a magnification level and an orientation calculated according to feature analysis techniques described above. For example, the magnitude of a vector can represent a magnification level of the deformed feature, and the direction of the vector can represent the orientation of the deformed feature.

[0058] Returning to Fig. 3, in at least one embodiment, to generate the pixel density map 338, the map generator 316 may divide the synthetic stitched image 334 into fixed-size pixel blocks (e.g., 32x32 pixel blocks, 32x64 pixel blocks, 64x64 pixel blocks, etc.). The map generator 316 may average the pixel densities (magnification levels) and orientations for all features in each block and generate a pixel density map by combining the average pixel magnification levels and orientations for all blocks. Such a pixel density map 338 may illustrate the magnification levels and orientations for different regions of the synthetic stitched image 334.

[0059] In at least one embodiment, the pixel density component 310 may update one or more parameters (e.g., parameters 116, display configurations 140, etc.) of an image stitching pipeline (e.g., image stitching pipeline 101) and / or an image stitching algorithm based on one or more determined measurements 336. For example, the pixel density component 310 may adjust one or more parameters based on calculated pixel densities (magnification levels) of deformed features. In response to a determination that a pixel density of a deformed feature is below a predetermined threshold pixel density, the pixel density component 310 may adjust one or more parameters of the image stitching pipeline to ensure that the image stitching pipeline limits compression in a corresponding region of stitched images such that the threshold is met. In at least one embodiment, thresholds may be determined (e.g.,by a user, a developer, etc.) to ensure that the image stitching pipeline produces high-quality stitched images with minimal information loss. In at least one embodiment, different pixel density thresholds may be assigned to different areas or regions (and features enclosed within those areas) of the synthetic image 334 (e.g., by a user, developer, etc.). In at least one embodiment, thresholds may be determined for other measurements 336, such as feature orientations, eccentricity, etc.

[0060] Fig. 6 illustrates a flowchart of an exemplary method 600 for measuring the pixel density of a synthetic stitched image, according to at least one embodiment of the present disclosure. For simplicity and clarity, the method 600 is illustrated and described as a series of operations. However, according to the present disclosure, such operations may be performed in other orders and / or concurrently with other operations not presented or described herein. Furthermore, not all of the illustrated operations may be required for implementing the methods of the present disclosure. Those skilled in the art will understand and appreciate that the methods may be represented as a series of related states or events via a state representation.Furthermore, it will be appreciated that the disclosed methods are capable of being stored on an article of manufacture. The term "article of manufacture," as used herein, is intended to include a computer-readable device or storage medium provided with a computer program and / or executable instructions that, when executed, perform one or more operations. In some embodiments, method 600 may be performed by processing logic (e.g., processing circuitry) of a computing device.

[0061] At operation 602 of method 600, the processing logic may process a first synthetic image including a first pattern and a second synthetic image including a second pattern to generate a merged image including a modified version of the first synthetic image and a modified version of the second synthetic image. One or more features of the first pattern are deformed in the modified version of the first synthetic image, and one or more features of the second pattern are deformed in the modified version of the second synthetic image. In at least one embodiment, the first pattern and the second pattern include a plurality of points (or other features) arranged in a grid, such as with respect to Fig. 4A. In at least one embodiment, the first synthetic image and the second synthetic image may be a same image including a same pattern. In at least one embodiment, the processing logic may determine the first synthetic image and the second synthetic image.

[0062] At operation 604 of method 600, the processing logic may determine pixel regions for each feature of the modified version of the first synthetic image and for each feature of the modified version of the second synthetic image. In at least one embodiment, the processing logic may perform operations 606 through 610 to determine the pixel regions.

[0063] At operation 606 of method 600, the processing logic may perform a thresholding process on the merged image to obtain a binary image. The features of the modified version of the first synthetic image and the features of the modified version of the second synthetic image may be classified as foreground objects.

[0064] At operation 608 of method 600, the processing logic may further provide the binary image as input to a connected component algorithm. The connected component algorithm provides, as output, a labeled image including multiple connected components. Each connected component is assigned a unique identifier corresponding to a feature of the modified version of the first synthetic image or a feature of the modified version of the second synthetic image.

[0065] At operation 610 of method 600, the processing logic may further determine pixel regions for each connected component using one or more shape analysis algorithms.

[0066] At operation 612 of method 600, the processing logic may further determine feature densities for one or more regions of the merged image based at least on the determined pixel areas. In at least one embodiment, the processing logic may perform operation 614 to determine feature densities.

[0067] At operation 614 of method 600, the processing logic may determine feature densities based on area ratios between the pixel areas of the features of the modified version of the first synthetic image and the second synthetic image and corresponding pixel areas of the features of the first synthetic image and the second synthetic image. The area ratios indicate the magnification levels of the respective features. In at least one embodiment, feature densities are determined in one or more stages of an image stitching pipeline, such as image stitching pipeline 101.

[0068] At operation 616 of method 600, the processing logic may determine orientations for each connected component using a covariance matrix of associated pixel coordinates. The orientations indicate a direction of deformation associated with the deformed features of the first pattern or the deformed features of the second pattern.

[0069] At operation 618 of method 600, the processing logic may generate a vector map based on the pixel areas of the features of the modified version of the first synthetic image and the second synthetic image and associated orientations of the features of the first synthetic image and the second synthetic image.

[0070] At operation 620 of method 600, the processing logic may adjust a first parameter associated with a stitching algorithm that generates the stitched images in response to a determination that one or more feature densities within a first region of the stitched image are below a first threshold density.

[0071] At operation 622 of method 600, the processing logic may adjust a second parameter associated with the stitching algorithm in response to a determination that one or more feature densities within a first region of the stitched image are below a second threshold density. In at least one embodiment, the first parameter and / or the second parameter may be adjusted according to one or more other criteria. Such criteria may include thresholds associated with computed statistics associated with the computed image calculated according to one or more of the shape analysis techniques described above. The computed statistics may include, for example, Hu-moment statistics, such as eccentricity, orientation, and / or other Hu-moments.

[0072] Values ​​for the first and second parameters can determine aspects of how component images are stitched together, which may relate to blending weights or locations, and other such aspects. Example stitching parameters include, but are not limited to, a blending method (e.g., alpha blending, multi-band blending, or Poisson blending), a blending width, a blending alpha map shape (e.g., angle-based or perpendicular-based), blending weights in different bands for multi-band blending, weights in different color channels in different color spaces (e.g., RGB, CIELAB, CMYK, YIQ, YCbCr, YUV, HSV, or HSL), stitch type (e.g., diagonal stitch, vertical stitch, or horizontal stitch), and / or stitch location.

[0073] Fig. 7 illustrates an example network configuration 700 that may be used to provide, generate, modify, encode, and / or transmit content. In at least one embodiment, a client device 702 may generate or receive content for a session using components of a content application 704 on the client device 702 and data stored locally on that client device. In at least one embodiment, a content application 724 (e.g., an image generation or editing application) executing on a content server 720 (e.g., a cloud server or edge server) may initiate a session associated with at least the client device 702 using a session manager and user data stored in a user database 734, and may cause content 732 to be determined by a content manager 726.An image stitching component 728 may receive image data for a scene or environment and generate a composite representation of that scene or environment. In at least one embodiment, the image stitching component 728 may generate a composite representation of multiple synthetic images. A pixel density component 730 may determine pixel densities of features in the multiple synthetic images according to techniques described herein. At least a portion of this content may then be transmitted to the client device 702 using an appropriate transmission manager 722 for transmission via download, streaming, or other such transmission channel. An encoder may be used to encode and / or compress this data before transmitting it to the client device 702. In at least one embodiment, this content 732 may include video or image data for a scene.In at least one embodiment, the client device 702 receiving this content may provide this content to a corresponding content application 704, which may also or alternatively include a stitching component 712 or pixel density component 714 for stitching image data received at or captured by the client device 702. A decoder may also be used to decode data received over the network(s) 740 for presentation via the client device 702, such as image or video content through a display 706 and audio, such as sounds and music, through at least one audio playback device 708, such as speakers or headphones.In at least one embodiment, at least a portion of this content may already be stored, rendered, or accessible on the client device 702, such that transfer over the network 740 is not required for at least that portion of the content, for example, where that content may have been previously downloaded or stored locally on a hard drive or optical disk. In at least one embodiment, a transfer mechanism, such as data streaming, may be used to transfer this content from the server 720, or the content database 734, to the client device 702. In at least one embodiment, a portion of this content may be obtained or streamed from another source, such as a third-party content service 760, which may also include a content application 762 for generating or providing content.In at least one embodiment, portions of this functionality may be performed using multiple computing devices, or multiple processors within one or more computing devices, which may include, for example, a combination of CPUs and GPUs.

[0074] In this example, these client devices may include any suitable computing device, which may include, for example, a desktop computer, a notebook computer, a set-top box, a streaming device, a game console, a smartphone, a tablet computer, a VR headset, AR glasses, a wearable computer, or a smart TV. Each client device may send a request over at least one wired or wireless network, which may include, for example, the Internet, an Ethernet, a local area network (LAN), or a cellular network, among other such options. In this example, these requests may be sent to an address associated with a cloud provider, which may operate or control one or more electronic resources in a cloud provider's environment, which may include, for example, a data center or a server farm.In at least one embodiment, the request may be received or processed by at least one edge server located at a network edge and outside of at least one security layer associated with the cloud provider environment. In this way, latency may be reduced by allowing client devices to interact with servers located in close proximity while simultaneously enhancing the security of resources in the cloud provider environment.

[0075] In at least one embodiment, such a system may be used to perform graphical rendering operations. In other embodiments, such a system may be used for other purposes, such as providing image or video content for testing or validating autonomous machine applications or for performing deep learning operations. In at least one embodiment, such a system may be implemented using an edge device or may include one or more virtual machines (VMs). In at least one embodiment, such a system may be implemented at least partially in a data center or at least partially using cloud computing resources.

[0076] Fig. 8A illustrates an example of an autonomous vehicle 800, according to at least one embodiment. In at least one embodiment, the autonomous vehicle 800 (alternatively referred to herein as "vehicle 800") may be, among other things, a passenger vehicle, such as a car, a truck, a bus, and / or another type of vehicle that accommodates one or more passengers. In at least one embodiment, the vehicle 800 may be a semi-trailer truck used for hauling cargo. In at least one embodiment, the vehicle 800 may be an aircraft, a robotic vehicle, or another type of vehicle.

[0077] Autonomous vehicles may be described in terms of automation levels defined by the National Highway Traffic Safety Administration ("NHTSA"), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers ("SAE") "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (e.g., Standard No. J3016-201806, published June 15, 2018, Standard No. J3016-201609, published September 30, 2016, and prior and future versions of that standard). In at least one embodiment, the vehicle 800 may be capable of functionality consistent with one or more of Level 1 through Level 5 of the autonomous driving levels. For example, in at least one embodiment, the vehicle 800 may be capable of Conditional Automation (Level 3), High Automation (Level 4), and / or Full Automation (Level 5), depending on the embodiment.

[0078] In at least one embodiment, vehicle 800 may include, among other things, components such as a chassis, a vehicle body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of a vehicle. In at least one embodiment, vehicle 800 may include, among other things, a propulsion system 850, such as an internal combustion engine, a hybrid-electric powerplant, an all-electric motor, and / or another type of propulsion system. In at least one embodiment, propulsion system 850 may be connected to a drivetrain of vehicle 800, including, among other things, a transmission to enable propulsion of vehicle 800. In at least one embodiment, propulsion system 850 may be controlled in response to receiving signals from throttle(s) / accelerator(s) 852.

[0079] In at least one embodiment, a steering system 854, which may include, among other things, a steering wheel, is used to steer the vehicle 800 (e.g., along a desired path or route) when the propulsion system 850 is operating (e.g., when the vehicle 800 is in motion). In at least one embodiment, the steering system 854 may receive signals from one or more steering actuators 856. In at least one embodiment, a steering wheel may be optional for full automation functionality (Level 5). In at least one embodiment, a brake sensor system 846 may be used to operate the vehicle brakes in response to receiving signals from one or more brake actuators 848 and / or brake sensors.

[0080] In at least one embodiment, one or more controllers 836, including, but not limited to, one or more system-on-chips (“SoCs”) (in Fig. 8A not shown) and / or one or more graphics processing unit(s) ("GPU(s)"), provide signals (e.g., representative of commands) to one or more components and / or one or more systems of the vehicle 800. For example, in at least one embodiment, one or more controllers 836 may send signals to operate vehicle brakes via brake actuator(s) 848, to operate the steering system 854 via steering actuator(s) 856, to operate the propulsion system 850 via throttle(s) / accelerator(s) 852. In at least one embodiment, one or more controllers 836 may include one or more (e.g., integrated) onboard computing devices that process sensor signals and issue operational commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving the vehicle 800.In at least one embodiment, one or more controllers 836 may include a first controller for autonomous driving functions, a second controller for functional safety functions, a third controller for artificial intelligence functionality (e.g., computer vision), a fourth controller for infotainment functionality, a fifth controller for redundancy in emergency situations, and / or other controllers. In at least one embodiment, a single controller may handle two or more of the above functionalities, two or more controllers may handle a single functionality, and / or any combination thereof.

[0081] In at least one embodiment, one or more controllers 836 provide signals to control one or more components and / or one or more systems of the vehicle 800 in response to sensor data received from one or more sensors (e.g., sensor inputs). For example, in at least one embodiment, sensor data may be received from, among others: global navigation satellite system (“GNSS”) sensors 858 (e.g., global positioning system sensors), RADAR sensors 860, ultrasonic sensors 862, LIDAR sensors 864, inertial measurement unit (“IMU”) sensors 866 (e.g., accelerometers, gyroscopes, a magnetic compass or magnetic compasses, magnetometers, etc.), microphones 896, stereo cameras 868, wide-angle cameras 870 (e.g., fisheye cameras), infrared cameras 872, environmental cameras 874 (e.g., 360-degree cameras), remote cameras (in Fig. 8A not shown), medium-range cameras (in Fig. 8A not shown), speed sensors 844 (e.g., for measuring the speed of the vehicle 800), vibration sensors 842, steering sensors 840, brake sensors (e.g., as part of a brake sensor system 846), and / or other sensor types.

[0082] In at least one embodiment, one or more controllers 836 may receive inputs (e.g., represented by input data) from an instrument cluster 832 of the vehicle 800 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (“HMI”) display 834, an audible annunciator, a speaker, and / or via other components of the vehicle 800. In at least one embodiment, outputs may include information such as vehicle speed, pace, time, map data (e.g., a high-definition map (in Fig. 8A not shown), location data (e.g., location of vehicle 800, as on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and status of objects as perceived by controller(s) 836, etc. For example, in at least one embodiment, HMI display 834 may display information about the presence of one or more objects (e.g., a road sign, caution sign, traffic signal timing, etc.) and / or information about maneuvers the vehicle has made, is making, or will make (e.g., changing lanes now, exit 34B in two miles, etc.).

[0083] In at least one embodiment, the vehicle 800 further includes a network interface 824 that may utilize one or more wireless antennas 826 and / or one or more modems to communicate over one or more networks. For example, in at least one embodiment, the network interface 824 may be capable of communicating over Long Term Evolution ("LTE"), Wideband Code Division Multiple Access ("WCDMA"), Universal Mobile Telecommunications System ("UMTS"), Global System for Mobile Communications ("GSM"), IMT-CDMA Multi-Carrier ("CDMA2000") networks, etc. In at least one embodiment, a wireless antenna(s) 826 may also facilitate communication between objects in an environment (e.g., vehicles, mobile devices, etc.) using local area network protocols such as Bluetooth, Bluetooth Low Energy ("LE"), Z-Wave, ZigBee, etc., and / or for a Low-Power Wide-Area Network(s) (“LPWANs”), such as LoRaWAN, SigFox, etc.

[0084] The pixel density component 150 may be used to perform image processing operations, including pixel density calculations, associated with one or more embodiments. Details regarding the pixel density component 150 are described herein in connection with Fig. 1. In at least one embodiment, the pixel density component in the autonomous vehicle 800 may be Fig. 8A to perform image processing operations, including pixel density calculations.

[0085] Fig. 8B illustrates an example of camera locations and fields of view for the autonomous vehicle 800 of Fig. 8A, according to at least one embodiment. In at least one embodiment, cameras and respective fields of view are an example embodiment and are not intended to be limiting. For example, in at least one embodiment, additional and / or alternative cameras may be included and / or cameras may be located at different locations on vehicle 800.

[0086] In at least one embodiment, camera types may include, but are not limited to, digital cameras that may be adapted for use with components and / or systems of the vehicle 800. In at least one embodiment, camera(s) may be operated at Automotive Safety Integrity Level (“ASIL”) B and / or another ASIL. In at least one embodiment, camera types may be capable of any image capture rate, such as 60 frames per second (fps), 1220 fps, 240 fps, etc., depending on the embodiment. In at least one embodiment, cameras may be capable of using rolling shutters, global shutters, another type of shutter, or a combination thereof.In at least one embodiment, a color filter array may include a red clear clear clear (“RCCC”) color filter array, a red clear clear blue (“RCCB”) color filter array, a red blue green clear (“RBGC”) color filter array, a Foveon X3 color filter array, a Bayer sensor (“RGGB”) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In at least one embodiment, clear-pixel cameras, such as cameras with an RCCC, an RCCB, and / or an RBGC color filter array, may be used in an effort to increase light sensitivity.

[0087] In at least one embodiment, one or more cameras may be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe system). For example, in at least one embodiment, a multi-function mono camera may be installed to provide functions such as lane departure warning, traffic sign assistance, and intelligent headlight control. In at least one embodiment, one or more cameras (e.g., all cameras) may simultaneously record and provide image data (e.g., video).

[0088] In at least one embodiment, one or more cameras may be mounted in a mounting assembly, such as a custom (three-dimensional ("3D") printed) assembly, to eliminate stray light and reflections from within the vehicle 800 (e.g., dashboard reflections in windshield mirrors) that may impair the camera's image data collection capabilities. With respect to the side mirror mounting assembly, in at least one embodiment, side mirror assemblies may be custom 3D printed so that a camera mounting plate conforms to a shape of a side mirror. In at least one embodiment, camera(s) may be integrated into side mirrors. In at least one embodiment, for side view cameras, camera(s) may also be integrated within four pillars at each corner of a cab.

[0089] In at least one embodiment, cameras with a field of view that includes portions of an environment in front of the vehicle 800 (e.g., forward-facing cameras) may be used for environmental viewing to identify forward paths and obstacles, and, with the assistance of one or more controllers 836 and / or control SoCs, to assist in providing information critical for generating an occupancy grid and / or determining preferred vehicle paths. In at least one embodiment, forward-facing cameras may be used to perform many similar ADAS functions, such as LIDAR, including, but not limited to, emergency braking, pedestrian detection, and collision avoidance.In at least one embodiment, forward-facing cameras may also be used for ADAS features and systems, including, but not limited to, Lane Departure Warnings (“LDW”), Autonomous Cruise Control (“ACC”), and / or other features such as traffic sign recognition.

[0090] In at least one embodiment, various cameras may be used in a forward-facing configuration, for example, including a monocular camera platform incorporating a CMOS (complementary metal oxide semiconductor) color imager. In at least one embodiment, a wide-angle camera 870 may be used to detect objects entering the field of view from a periphery (e.g., pedestrians, crossing traffic, or bicycles). Although in Fig. 8B illustrates only one wide-angle camera 870, in other embodiments, there may be any number (including zero) of wide-angle cameras on the vehicle 800. In at least one embodiment, any number of remote cameras 898 (e.g., a long-view stereo camera pair) may be used for depth-based object detection, especially for objects for which a neural network has not yet been trained. In at least one embodiment, remote camera(s) 898 may also be used for object detection and classification, as well as simple object tracking.

[0091] In at least one embodiment, any number of stereo cameras 868 may also be included in a forward-facing configuration. In at least one embodiment, one or more stereo cameras 868 may include an integrated control unit comprising a scalable processing unit that may provide a programmable logic ("FPGA") and a multi-core microprocessor with an integrated controller area network ("CAN") or Ethernet interface on a single chip. In at least one embodiment, such a unit may be used to generate a 3D map of an environment of the vehicle 800, including distance estimation for all points in an image.In at least one embodiment, one or more stereo cameras 868 may include, among other things, a compact stereo vision sensor(s), which may include, among other things, two camera lenses (one each on the left and right) and an image processing chip, with which the distance from the vehicle 800 to a target object can be measured and generated information (e.g., metadata) can be used to activate autonomous emergency braking and lane departure warning functions. In at least one embodiment, other types of stereo cameras 868 may be used in addition to or alternatively to those described herein.

[0092] In at least one embodiment, cameras with a field of view that includes portions of an environment to the side of the vehicle 800 (e.g., side view cameras) may be used for environmental viewing, thus providing information used to generate and update an occupancy grid and generate side impact collision warnings. For example, in at least one embodiment, environmental cameras 874 (e.g., four environmental cameras as in Fig. 8B) may be positioned on the vehicle 800. In at least one embodiment, the surround view cameras 874 may include, among other things, any number and combination of wide-angle cameras, fisheye cameras, 360-degree cameras, and / or similar cameras. For example, in at least one embodiment, four fisheye cameras may be positioned on a front, a rear, and sides of the vehicle 800. In at least one embodiment, the vehicle 800 may utilize three surround view cameras 874 (e.g., left, right, and rear) and may utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround view camera.

[0093] In at least one embodiment, cameras with a field of view that includes portions of an environment behind the vehicle 800 (e.g., rear view cameras) may be used for parking assistance, surround view, rear collision warnings, and generating and updating an occupancy grid. In at least one embodiment, a wide range of cameras may be used, including, but not limited to, cameras that are also suitable as forward-facing cameras (e.g., long-range cameras 898 and / or mid-range cameras 876, stereo cameras 868, infrared cameras 872, etc.), as described herein.

[0094] The pixel density component 150 may be used to perform image processing operations, including pixel density calculations, associated with one or more embodiments. Details regarding the pixel density component 150 are described herein in connection with Fig. 1. In at least one embodiment, the pixel density component in the autonomous vehicle 800 may be Fig. 8B can be used to perform image processing operations, including pixel density calculations.

[0095] Fig. 8C is a block diagram illustrating an example system architecture for the autonomous vehicle 800 of Fig. 8A, according to at least one embodiment. In at least one embodiment, the components, features, and systems of the vehicle 800 are Fig. 8C as being connected via a bus 802. In at least one embodiment, bus 802 may include, among other things, a CAN data interface (alternatively referred to herein as a "CAN bus"). In at least one embodiment, a CAN may be a network within vehicle 800 used to support the control of various features and functionality of vehicle 800, such as brake application, acceleration, braking, steering, windshield wipers, etc. In at least one embodiment, bus 802 may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). In at least one embodiment, bus 802 may be read to find steering wheel angle, ground speed, engine revolutions per minute ("RPMs"), button positions, and / or other vehicle status indicators.In at least one embodiment, bus 802 may be a CAN bus that is ASIL-B compliant.

[0096] In at least one embodiment, in addition to or alternatively to CAN, FlexRay, and / or Ethernet protocols may be used. In at least one embodiment, there may be any number of buses forming bus 802, which may include, but is not limited to, zero or more CAN buses, zero or more FlexRay buses, zero or more Ethernet buses, and / or zero or more other types of buses using different protocols. In at least one embodiment, two or more buses may be used to perform different functions and / or used for redundancy. For example, a first bus may be used for collision avoidance functionality, and a second bus may be used for actuation control.In at least one embodiment, each bus of bus 802 may communicate with any components of vehicle 800, and two or more buses of bus 802 may communicate with corresponding components. In at least one embodiment, any number of systems-on-chips (SoC(s)) 804 (such as SoC 804(A) and SoC 804(B)), each of one or more controller(s) 836, and / or each computer within a vehicle may each have access to the same input data (e.g., inputs from sensors of vehicle 800) and be connected to a common bus, such as a CAN bus.

[0097] In at least one embodiment, the vehicle 800 may include one or more controllers 836, such as those described herein with respect to Fig. 8A. In at least one embodiment, the controller(s) 836 may be used for a variety of functions. In at least one embodiment, the controller(s) 836 may be coupled to any of various other components and systems of the vehicle 800 and used for control of the vehicle 800, artificial intelligence of the vehicle 800, infotainment for the vehicle 800, and / or other functions.

[0098] In at least one embodiment, the vehicle 800 may include any number of SoCs 804. In at least one embodiment, each SoC 804 may include, among other things, central processing units ("CPU(s)") 806, graphics processing units ("GPU(s)") 808, processors 810, caches 812, accelerators 814, data storage 816, and / or other unillustrated components and features. In at least one embodiment, SoC(s) 804 may be used to control the vehicle 800 in a variety of platforms and systems. For example, in at least one embodiment, SoC(s) 804 in a system (e.g., system of the vehicle 800) may be combined with a high-definition ("HD") map 822, whereby map refreshes and / or updates may be received via the network interface 824 from one or more servers (in Fig. 8C not shown).

[0099] In at least one embodiment, CPU(s) 806 may include a CPU cluster or a CPU complex (alternatively referred to herein as a "CCPLEX"). In at least one embodiment, CPU(s) 806 may include multiple cores and / or level-two ("L2") caches. For example, in at least one embodiment, CPU(s) 806 may include eight cores in a coherent multiprocessor configuration. In at least one embodiment, CPU(s) 806 may include four dual-core clusters, each cluster having a dedicated L2 cache (e.g., a 2-megabyte (MB) L2 cache). In at least one embodiment, CPU(s) 806 (e.g., CCPLEX) may be configured to support concurrent cluster operations, allowing any combination of clusters of CPU(s) 806 to be active at any given time.

[0100] In at least one embodiment, one or more CPU(s) 806 may implement power management capabilities including, but not limited to, one or more of the following features: individual hardware blocks may be automatically clock-gated when idle to conserve dynamic power; each core clock may be decoupled when such core is not actively executing instructions due to the execution of Wait-For-Interrupt ("WFI") / Wait-For-Event ("WFE") instructions; each core may be independently power gated; each core cluster may be independently clock-gated if all cores are clock-gated or power-gated; and / or each core cluster may be independently power-gated if all cores are power-gated.In at least one embodiment, CPU(s) 806 may further implement an enhanced performance state management algorithm, where the allowable performance states and expected wake-up times are specified, and hardware / microcode for the core, cluster, and CCPLEX determines the best performance state to enter. In at least one embodiment, processing cores may support simplified performance state entry sequences in software, offloading the work to microcode.

[0101] In at least one embodiment, GPU(s) 808 may include an integrated GPU (alternatively referred to herein as an "iGPU"). In at least one embodiment, GPU(s) 808 may be programmable and efficient for parallel workloads. In at least one embodiment, GPU(s) 808 may utilize an enhanced tensor instruction set. In at least one embodiment, GPU(s) 808 may include one or more streaming microprocessors, where each streaming microprocessor may include a level-one ("L1") cache (e.g., an L1 cache with at least 96 KB of storage capacity) and two or more streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512 KB of storage capacity). In at least one embodiment, GPU(s) 808 may include at least eight streaming microprocessors.In at least one embodiment, a GPU(s) 808 may utilize compute application programming interface(s) (API(s)). In at least one embodiment, a GPU(s) 808 may utilize one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA model).

[0102] In at least one embodiment, one or more of the GPU(s) 808 may be power-optimized to achieve the best performance in automotive and embedded use cases. In at least one embodiment, one or more of the GPU(s) 808 may be fabricated, for example, on fin field-effect transistor ("FinFET") circuitry. In at least one embodiment, each streaming microprocessor may include a number of mixed-precision processing cores divided into multiple blocks. For example, 64 PF32 cores and 32 PF64 cores could be divided into four processing blocks, among others. In at least one embodiment, each processing block may be assigned 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two NVIDIA Tensor cores for mixed-precision deep learning matrix arithmetic, a level zero ("L0") instruction cache, a warp scheduler, a dispatch unit, and / or a 64 KB register file.In at least one embodiment, streaming microprocessors may include independent parallel integer and floating-point datapaths to enable efficient execution of workloads with a mix of arithmetic and addressing calculations. In at least one embodiment, streaming microprocessors may include independent thread scheduling capabilities to enable finer-grained synchronization and collaboration between parallel threads. In at least one embodiment, streaming microprocessors may include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

[0103] In at least one embodiment, one or more GPU(s) 808 may include high-bandwidth memory ("HBM") and / or a 16 GB HBM2 memory subsystem to provide, in some examples, a peak memory bandwidth of approximately 900 GB / second. In at least one embodiment, synchronous graphics random access memory ("SGRAM") may be used in addition to, or as an alternative to, the HBM memory, e.g., synchronous graphics random access memory (“GDDR5”), such as double data rate synchronous graphics random access memory (“GDDR5”).

[0104] In at least one embodiment, GPU(s) 808 may include unified memory technology. In at least one embodiment, Address Translation Services ("ATS") support may be used to enable GPU(s) 808 to directly access the page tables of CPU(s) 806. In at least one embodiment, an address translation request may be communicated to CPU(s) 806 when a GPU of the memory management unit (MMU) of GPU(s) 808 experiences a miss. In response, in at least one embodiment, two CPUs of CPU(s) 806 may look up a virtual-to-physical mapping for an address in their page tables and transmit the translation back to GPU(s) 808.In at least one embodiment, unified memory technology may enable a single unified virtual address space for the memory of both the CPU(s) 806 and the GPU(s) 808, thereby simplifying programming of the GPU(s) 808 and porting applications to the GPU(s) 808.

[0105] In at least one embodiment, a GPU(s) 808 may include any number of access counters that can track the frequency of GPU(s) 808 accessing the memory of other processors. In at least one embodiment, access counters can help ensure that memory pages are moved to the physical memory of a processor that accesses pages most frequently, thereby improving the efficiency of memory regions shared between processors.

[0106] In at least one embodiment, one or more of the SoC(s) 804 may include any number of cache(s) 812, including those described herein. For example, in at least one embodiment, cache(s) 812 may include a level-three ("L3") cache available to both CPU(s) 806 and GPU(s) 808 (e.g., connected to CPU(s) 806 and GPU(s) 808). In at least one embodiment, cache(s) 812 may include a write-back cache that may track states of lines, e.g., using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). In at least one embodiment, an L3 cache may include 4 MB of memory or more, depending on the embodiment, although smaller cache sizes may be used.

[0107] In at least one embodiment, one or more of the SoC(s) 804 may include one or more accelerators 814 (e.g., hardware accelerators, software accelerators, or a combination thereof). In at least one embodiment, the SoC(s) 804 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. In at least one embodiment, large on-chip memory (e.g., 4 MB of SRAM) may enable a hardware acceleration cluster to accelerate neural networks and other computations. In at least one embodiment, a hardware acceleration cluster may be used to supplement the GPU(s) 808 and offload some tasks from the GPU(s) 808 (e.g., to free up more cycles of the GPU(s) 808 to perform other tasks).In at least one embodiment, the accelerator(s) 814 may be used for targeted workloads (e.g., perception, convolutional neural networks ("CNNs"), recurrent neural networks ("RNNs"), etc.) that are robust enough to be amenable to acceleration. In at least one embodiment, a CNN may include region-based or regional convolutional neural networks ("RCNNs") and fast RCNNs (e.g., as used for object detection), or other types of CNNs.

[0108] In at least one embodiment, the accelerator(s) 814 (e.g., hardware acceleration clusters) may include one or more deep learning accelerators (“DLAs”). In at least one embodiment, DLA(s) may include, among other things, one or more tensor processing units (“TPUs”) that may be configured to provide an additional ten trillion operations per second for deep learning applications and inferencing. In at least one embodiment, TPUs may be accelerators configured and optimized to perform image processing functions (e.g., for CNNs, RCNNs, etc.). In at least one embodiment, DLA(s) may be further optimized for a specific set of neural network types and floating-point operations, as well as for inferencing.In at least one embodiment, the design of DLA(s) can provide more performance per millimeter than a typical general-purpose GPU, typically far exceeding the performance of a CPU. In at least one embodiment, a TPU(s) can perform multiple functions, including a single-instance convolution function, supporting, for example, INT8, INT16, and FP16 data types for features and weights, as well as post-processing functions.In at least one embodiment, DLA(s) can quickly and efficiently execute neural networks, particularly CNNs, on processed or unprocessed data for any variety of functions, including, for example, but not limited to: a CNN for object identification and detection using data from camera sensors; a CNN for estimating distances using data from camera sensors; a CNN for emergency vehicle detection and identification and detection using data from microphones; a CNN for facial recognition and vehicle owner identification using data from camera sensors; and / or a CNN for safety-related events.

[0109] In at least one embodiment, DLA(s) may perform any function of GPU(s) 808, and using an inference accelerator, for example, a designer may use either DLA(s) or GPU(s) 808 for any function. For example, in at least one embodiment, a designer may focus the processing of CNNs and floating-point operations on DLA(s) and leave other functions to GPU(s) 808 and / or other accelerator(s) 814.

[0110] In at least one embodiment, the accelerator(s) 814 may include a programmable vision accelerator (“PVA”), which may alternatively be referred to herein as a computer vision accelerator. In at least one embodiment, the PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (“ADAS”) 838, autonomous driving, augmented reality (“AR”) applications, and / or virtual reality (“VR”) applications. In at least one embodiment, a PVA may provide a balance between performance and flexibility. For example, in at least one embodiment, each PVA may include, among other things, any number of reduced instruction set computing (“RISC”) cores, direct memory access (“DMA”) cores, and / or any number of vector processors.

[0111] In at least one embodiment, RISC cores may interact with image sensors (e.g., image sensors of any of the cameras described herein), image signal processor(s), etc. In at least one embodiment, each RISC core may include any memory location. In at least one embodiment, RISC cores may use one of several protocols, depending on the embodiment. In at least one embodiment, RISC cores may execute a real-time operating system ("RTOS"). In at least one embodiment, RISC cores may be implemented using one or more integrated circuit devices, application-specific integrated circuits ("ASICs"), and / or memory devices. For example, in at least one embodiment, RISC cores may include an instruction cache and / or tightly coupled RAM.

[0112] In at least one embodiment, DMA may enable components of the PVA to access system memory independently of CPU(s) 806. In at least one embodiment, DMA may support any number of features provided to optimize PVA, including, but not limited to, support for multi-dimensional addressing and / or circular addressing. In at least one embodiment, DMA may support up to six or more addressing dimensions, which may include, but are not limited to, block width, block height, block depth, horizontal block gradation, vertical block gradation, and / or depth gradation.

[0113] In at least one embodiment, vector processors may be programmable processors that can be designed to efficiently and flexibly execute the programming of computer vision algorithms and provide signal processing capabilities. In at least one embodiment, a PVA may include a PVA core and two vector processing subsystem partitions. In at least one embodiment, a PVA core may include a processor subsystem, a DMA engine(s) (e.g., two DMA engines), and / or other peripherals. In at least one embodiment, a vector processing subsystem may operate as the primary processing engine of a PVA and may include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., a VMEM). In at least one embodiment, the VPU core may include a digital signal processor, such as aa single-instruction-multiple-data ("SIMD"), very-long-instruction-word ("VLIW") digital signal processor. In at least one embodiment, a combination of SIMD and VLIW can improve throughput and speed.

[0114] In at least one embodiment, each of the vector processors may include an instruction cache and be coupled to dedicated memory. As a result, in at least one embodiment, each vector processor may be configured to operate independently of other vector processors. In at least one embodiment, vector processors included in a particular PVA may be configured to use data parallelism. For example, in at least one embodiment, multiple vector processors included in a single PVA may execute a common computer vision algorithm, but in different regions of an image.In at least one embodiment, vector processors included in a particular PVA may concurrently execute different computer vision algorithms on an image, or even execute different algorithms on sequential images or portions of an image. In at least one embodiment, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each PVA. In at least one embodiment, a PVA may include additional error correction code (ECC) memory to improve overall system security.

[0115] In at least one embodiment, the accelerator(s) 814 may include a computer vision network on a chip and static random-access memory (“SRAM”) to provide high-bandwidth, low-latency SRAM for the accelerator(s) 814. In at least one embodiment, the on-chip memory may include at least 4 MB of SRAM, including, for example, among other things, eight field-configurable memory blocks accessible by both a PVA and a DLA. In at least one embodiment, each pair of memory blocks may include an advanced peripheral bus (“APB”) interface, configuration circuitry, a controller, and a multiplexer. In at least one embodiment, any type of memory may be used.In at least one embodiment, a PVA and a DLA may access memory via a backbone that provides high-speed access to memory to a PVA and a DLA. In at least one embodiment, a backbone may include an on-chip computer vision network that connects a PVA and a DLA to memory (e.g., using APB).

[0116] In at least one embodiment, a computer vision network-on-chip may include an interface that determines that both a PVA and a DLA are providing ready and valid signals prior to transmitting control signals / addresses / data. In at least one embodiment, an interface may provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst communications for continuous data transfer. In at least one embodiment, an interface may conform to International Organization for Standardization ("ISO") 26262 or International Electrotechnical Commission ("IEC") 61508 standards, although other standards and protocols may be used.

[0117] In at least one embodiment, one or more of the SoC(s) 804 may include a hardware accelerator for real-time ray tracing. In at least one embodiment, a hardware accelerator may be used to quickly and efficiently determine positions and extents of objects (e.g., within a world model), generate real-time visualization simulations, interpret radar signals, synthesize and / or analyze sound propagation, simulate sonar systems, perform general wave propagation simulation, compare with lidar data for localization and / or other functions, and / or for other purposes.

[0118] In at least one embodiment, the accelerator(s) 814 may have a broad field of application for autonomous driving. In at least one embodiment, a PVA may be used for critical processing stages in ADAS and autonomous vehicles. In at least one embodiment, the capabilities of a PVA are well suited for algorithmic domains that require predictable processing with low power and low latency. In other words, a PVA is well-suited for semi-dense or dense periodic computations, even on small datasets that may require predictable runtimes with low latency and low power. In at least one embodiment, such as in the vehicle 800, PVAs may be designed to execute classical computer vision algorithms because they can be efficient at object detection and operate on integer mathematics.

[0119] For example, according to at least one embodiment of the technology, a PVA is used to perform computer stereo vision. In at least one embodiment, a semi-global matching-based algorithm may be used in some examples, although this is not intended to be limiting. In at least one embodiment, Level 3-5 autonomous driving applications utilize on-the-fly motion estimation / stereo matching (e.g., structure from motion, pedestrian detection, lane detection, etc.). In at least one embodiment, a PVA may perform computer stereo vision functions given inputs from two monocular cameras.

[0120] In at least one embodiment, a PVA may be used to perform dense optical flow. For example, in at least one embodiment, a PVA may process raw radar data (e.g., using a 6D Fast Fourier Transform) to provide processed radar data. In at least one embodiment, a PVA is used for deep time-of-flight processing, for example, by processing the raw time-of-flight data to provide processed time-of-flight data.

[0121] In at least one embodiment, a DLA may be used to drive any type of network to improve control and driving safety, for example, among other things, a neural network that outputs a confidence measure for each object detection. In at least one embodiment, confidence may be represented or interpreted as a probability, or as providing a relative "weight" of each detection compared to other detections. In at least one embodiment, a confidence measure allows a system to make further decisions about which detections should be considered true positives rather than false positives. In at least one embodiment, a system may set a confidence threshold and consider only those detections that exceed the threshold to be true positives.In an embodiment using an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically perform emergency braking, which is clearly undesirable. In at least one embodiment, high-confidence detections may be considered as triggers for AEB. In at least one embodiment, a DLA may run a neural network to regress the confidence value. In at least one embodiment, the neural network may take as input at least a subset of parameters, such as the bounding box dimensions, the ground plane estimate obtained (e.g., from another subsystem), the output of IMU sensor(s) 866 correlated with the orientation of the vehicle 800, the distance, 3D location estimates of the object obtained by the neural network and / or other sensors (e.g.,LIDAR sensor(s) 864 or RADAR sensor(s) 860), and others.

[0122] In at least one embodiment, one or more of the SoC(s) 804 may include the data storage(s) 816 (e.g., memory). In at least one embodiment, the data storage(s) 816 may be on-chip memory of the SoC(s) 804 that may store neural networks to be executed on the GPU(s) 808 and / or a DLA. In at least one embodiment, the data storage(s) 816 may be large enough to store multiple instances of neural networks for redundancy and security. In at least one embodiment, the data storage(s) 816 may include L2 or L3 cache(s).

[0123] In at least one embodiment, one or more of the SoC(s) 804 may include any number of processor(s) 810 (e.g., embedded processors). In at least one embodiment, the processor(s) 810 may include a boot and power management processor, which may be a dedicated processor and subsystem that handles boot power and management functions and associated security enforcement. In at least one embodiment, a boot and power management processor may be part of a boot sequence of SoC(s) 804 and may provide runtime power management services.In at least one embodiment, a boot power and management processor may provide clock and voltage programming, support for low-power system state transitions, management of thermal and temperature sensors of SoC(s) 804, and / or management of power supply states of SoC(s) 804. In at least one embodiment, each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and SoC(s) 804 may use ring oscillators to detect temperatures of CPU(s) 806, GPU(s) 808, and / or accelerator(s) 814.In at least one embodiment, if temperatures are determined to exceed a threshold, a boot and power management processor may enter a temperature fault routine and place the SoC(s) 804 into a lower power consumption state and / or place the vehicle 800 into a chauffeur-to-safe-stop mode (e.g., bring the vehicle 800 to a safe stop).

[0124] In at least one embodiment, the processor(s) 810 may further include a set of embedded processors that may serve as an audio processing engine, which may be an audio subsystem enabling full hardware support for multi-channel audio across multiple interfaces and a wide and flexible range of audio I / O interfaces. In at least one embodiment, an audio processing engine is a dedicated processor core with a digital signal processor with dedicated RAM.

[0125] In at least one embodiment, the processor(s) 810 may further include an always-on processor engine that may provide the necessary hardware features to support low-power sensor management and wake-up use cases. In at least one embodiment, an always-on processor engine may include, among other things, a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0126] In at least one embodiment, the processor(s) 810 may further include a security cluster engine, including, among other things, a dedicated processor subsystem for handling security management for automotive applications. In at least one embodiment, a security cluster engine may include, among other things, two or more processor cores, tightly coupled RAM memory, supporting peripherals (e.g., timers, an interrupt controller, etc.), and / or routing logic. In a security mode, two or more cores, in at least one embodiment, may operate in a lockstep mode, acting as a single core with comparison logic to detect differences between their operations.In at least one embodiment, the processor(s) 810 may further include a real-time camera engine, which may include, among other things, a dedicated processor subsystem for handling real-time camera management. In at least one embodiment, the processor(s) 810 may further include a high dynamic range signal processor, which may include, among other things, an image signal processor that is a hardware engine that is part of a camera processing pipeline.

[0127] In at least one embodiment, the processor(s) 810 may include a video image compositor, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to produce a final image for a playback window. In at least one embodiment, a video image compositor may perform lens distortion correction on the wide-angle camera(s) 870, the surround camera(s) 874, and / or the in-cabin surveillance camera sensor(s). In at least one embodiment, the in-cabin surveillance camera sensor(s) is / are preferably monitored by a neural network running on another instance of SoC 804 and configured to identify events in the cabin and respond accordingly.In at least one embodiment, an in-cabin system may perform, among other things, lip reading to activate cellular service and place a phone call, dictate emails, change a vehicle's destination, activate or change a vehicle's infotainment system and settings, or provide voice-activated web browsing. In at least one embodiment, certain functions are available to a driver when a vehicle is operating in an autonomous mode and are disabled otherwise.

[0128] In at least one embodiment, a video image compositor may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, in at least one embodiment where motion occurs in a video, the noise reduction may appropriately weight spatial information, thereby reducing weights of information provided by neighboring frames. In at least one embodiment where an image or portion of an image does not include motion, the temporal noise reduction performed by the video image compositor may use information from a previous frame to reduce noise in a current frame.

[0129] In at least one embodiment, a video image compositor may also be configured to perform stereo rectification on the individual images from the input stereo lenses. In at least one embodiment, a video image compositor may further be used for user interface composition when an operating system desktop is in use and the GPU(s) 808 is / are not required to continuously render new surfaces. In at least one embodiment, a video image compositor may be used to offload the GPU(s) 808 when the GPU(s) 808 is / are turned on and actively performing 3D rendering to improve performance and responsiveness.

[0130] In at least one embodiment, one or more of SoC(s) 804 may further include a Mobile Industry Processor Interface ("MIPI") camera serial interface for receiving video and inputs from cameras, a high-speed interface, and / or a video input block that may be used for a camera and associated pixel input functions. In at least one embodiment, one or more SoC(s) 804 may further include one or more input / output controllers that may be controlled by software and used to receive I / O signals that are not dedicated to a specific role.

[0131] In at least one embodiment, one or more SoC(s) 804 may further include a wide range of peripheral interfaces to enable communication with peripherals, audio encoders / decoders ("codecs"), power management, and / or other devices. In at least one embodiment, the SoC(s) 804 may be used to process data from cameras (e.g., connected via Gigabit Multimedia Serial Link and Ethernet channels), sensors (e.g., LIDAR sensor(s) 864, RADAR sensor(s) 860, etc., which may be connected via Ethernet channels), data from bus 802 (e.g., speed of vehicle 800, steering wheel position, etc.), data from GNSS sensor(s) 858 (e.g., connected via an Ethernet bus or a CAN bus), etc.In at least one embodiment, one or more SoC(s) of SoC(s) 804 may further include dedicated high-performance mass storage controllers, which may include their own DMA engines and which may be used to offload routine data management tasks from the CPU(s) 806.

[0132] In at least one embodiment, SoC(s) 804 may be an end-to-end platform with a flexible architecture that includes automation levels 3-5, thus providing a comprehensive functional safety architecture that leverages and efficiently deploys computer vision and ADAS techniques for diversity and redundancy, and provides a platform for a flexible, reliable propulsion software stack, along with deep learning tools. In at least one embodiment, SoC(s) 804 may be faster, more reliable, and even more power and space efficient than conventional systems. For example, in at least one embodiment, accelerator(s) 814, in combination with CPU(s) 806, GPU(s) 808, and data storage(s) 816, may provide a fast, efficient platform for Level 3-5 autonomous vehicles.

[0133] In at least one embodiment, computer vision algorithms may be executed on CPUs, which may be configured using a high-level programming language, such as C, to execute a wide range of processing algorithms on a wide range of visual data. However, in at least one embodiment, CPUs are often unable to meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In at least one embodiment, many CPUs are unable to execute complex, real-time object detection algorithms used in in-vehicle ADAS applications and practical Level 3-5 autonomous vehicles.

[0134] The embodiments described herein allow multiple neural networks to be executed concurrently and / or sequentially and the results to be combined to enable Level 3-5 autonomous driving functionality. For example, in at least one embodiment, a CNN executing on a DLA or a discrete GPU (e.g., GPU(s) 420) may include text and word recognition that allows for reading and understanding traffic signs, including signs for which a neural network has not been specifically trained. In at least one embodiment, a DLA may further include a neural network capable of identifying, interpreting, and providing semantic understanding of a sign and passing this semantic understanding to route planning modules running on a CPU complex.

[0135] In at least one embodiment, multiple neural networks may operate simultaneously, such as when driving at Level 3, 4, or 5. For example, in at least one embodiment, a warning sign reading "Caution: Flashing lights indicate black ice" along with an electric light may be interpreted independently or jointly by multiple neural networks. In at least one embodiment, such a warning sign may itself be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "flashing lights indicate black ice" may be interpreted by a second deployed neural network, which informs the vehicle's route planning software (preferably running on a CPU complex) that black ice is present upon detecting flashing lights.In at least one embodiment, a flashing light may be identified by operating a third deployed neural network over multiple frames, thereby informing vehicle route planning software of the presence (or absence) of flashing lights. In at least one embodiment, all three neural networks may be operated simultaneously, e.g., within a DLA and / or on GPU(s) 808.

[0136] In at least one embodiment, a CNN for facial recognition and vehicle owner identification may use data from camera sensors to identify the presence of an authorized driver and / or vehicle owner of the vehicle 800. In at least one embodiment, an always-on sensor processing engine may be used to unlock a vehicle when an owner approaches a driver's door and turns on the lights, and to disable such a vehicle in a security mode when an owner exits such a vehicle. In this way, the SoC(s) 804 provide security against theft and / or forced car hijacking.

[0137] In at least one embodiment, a CNN for emergency vehicle detection and identification may use data from microphones 896 to detect and identify emergency vehicle sirens. In at least one embodiment, the SoC(s) 804 use a CNN to classify ambient and urban sounds, as well as to classify visual data. In at least one embodiment, a CNN running on a DLA is trained to identify a relative approach speed of an emergency vehicle (e.g., using a Doppler effect). In at least one embodiment, a CNN may also be trained to identify emergency vehicles specific to a local area in which a vehicle is operating, as identified by GNSS sensor(s) 858.In at least one embodiment, a CNN operating in Europe will attempt to detect European sirens, and a CNN operating in North America will attempt to identify only North American sirens. In at least one embodiment, once an emergency vehicle is detected, a control program assisted by one or more ultrasonic sensors 862 may be used to execute a safety routine for the emergency vehicle, decelerate a vehicle, pull over on a road, park a vehicle, and / or idle a vehicle until emergency vehicles have passed.

[0138] In at least one embodiment, vehicle 800 may include CPU(s) 818 (e.g., discrete CPU(s) or dCPU(s)) that may be coupled to SoC(s) 804 via a high-speed interconnect (e.g., PCIe). In at least one embodiment, CPU(s) 818 may include, for example, an x86 processor. CPU(s) 818 may be used, for example, to perform any of a number of functions, including arbitrating potentially inconsistent results between ADAS sensors and SoC(s) 804 and / or monitoring the status and health of controller(s) 836 and / or an infotainment system on a chip ("infotainment SoC") 830.

[0139] In at least one embodiment, vehicle 800 may include GPU(s) 820 (e.g., discrete GP(s) or dGPU(s)) that may be coupled to SoC(s) 804 via a high-speed interconnect (e.g., NVIDIA's NVLINK channel). In at least one embodiment, GPU(s) 820 may provide additional artificial intelligence functionality, e.g., by executing redundant and / or distinct neural networks, and may be used to train and / or update neural networks based at least in part on inputs (e.g., sensor data) from sensors of vehicle 800.

[0140] In at least one embodiment, vehicle 800 may further include a network interface 824, which may include, among other things, wireless antenna(s) 826 (e.g., one or more wireless antennas for various communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). In at least one embodiment, network interface 824 may be used to enable wireless connectivity to internet cloud services (e.g., to servers and / or other network devices), to other vehicles, and / or to computing devices (e.g., passenger client devices). In at least one embodiment, a direct connection between vehicle 40 and another vehicle and / or an indirect connection (e.g., via networks and over the internet) may be established to communicate with other vehicles.In at least one embodiment, direct connections may be provided using a vehicle-to-vehicle communication link. In at least one embodiment, a vehicle-to-vehicle communication link may provide information to the vehicle 800 about vehicles in the vicinity of the vehicle 800 (e.g., vehicles in front of, to the side of, and / or behind the vehicle 800). In at least one embodiment, such aforementioned functionality may be part of a cooperative adaptive cruise control functionality of the vehicle 800.

[0141] In at least one embodiment, the network interface 824 may include an SoC that provides modulation and demodulation functionality and enables the controller(s) 836 to communicate over wireless networks. In at least one embodiment, the network interface 824 may include a radio frequency front-end for upconverting from baseband to radio frequency and for downconverting from radio frequency to baseband. In at least one embodiment, frequency conversions may be performed in any technically suitable manner. For example, frequency conversions may be performed by known processes and / or using super-heterodyne processes. In at least one embodiment, the radio frequency front-end functionality may be provided by a separate chip.In at least one embodiment, network interfaces may include wireless functionality for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0142] In at least one embodiment, the vehicle 800 may further include one or more data stores 828, which may include, but are not limited to, off-chip (e.g., off-SoC(s) 404) storage. In at least one embodiment, the data stores 828 may include, but are not limited to, one or more storage elements, including RAM, SRAM, dynamic random-access memory (“DRAM”), video random-access memory (“VRAM”), flash memory, hard drives, and / or other components and / or devices capable of storing at least one bit of data.

[0143] In at least one embodiment, vehicle 800 may further include GNSS sensor(s) 858 (e.g., GPS and / or assisted GPS sensors) to support mapping, sensing, occupancy grid generation, and / or route planning functions. In at least one embodiment, any number of GNSS sensor(s) 858 may be used, including, for example, but not limited to, a GPS using a USB port with an Ethernet-to-serial (e.g., RS-232) bridge.

[0144] In at least one embodiment, the vehicle 800 may further include RADAR sensor(s) 860. In at least one embodiment, the RADAR sensor(s) 860 may be used by the vehicle 800 for long-range vehicle detection, even in darkness and / or adverse weather conditions. In at least one embodiment, the RADAR functional safety levels may be ASIL B. In at least one embodiment, the RADAR sensor(s) 860 may use a CAN bus and / or bus 802 (e.g., for transmitting data generated by RADAR sensor(s) 860) for control and access to object tracking data, with access to Ethernet channels to access raw data in some examples. In at least one embodiment, a wide range of RADAR sensor types may be used. For example, RADAR sensor(s) 860 may be suitable for front, rear, and side RADAR use, among others.In at least one embodiment, one or more sensors of the RADAR sensor(s) 860 are a pulse Doppler RADAR sensor.

[0145] In at least one embodiment, the RADAR sensor(s) 860 may include various configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In at least one embodiment, long-range RADAR may be used for adaptive cruise control functionality. In at least one embodiment, long-range RADAR systems may provide a wide field of view realized by two or more independent scans, e.g., within a range of 250 m (meters). In at least one embodiment, the RADAR sensor(s) 860 may assist in distinguishing between static and moving objects and may be used by the ADAS system 838 for emergency braking and forward collision warning.In at least one embodiment, sensors 860(s) included in a long-range radar system may include, among other things, monostatic multimodal radar with multiple (e.g., six or more) fixed radar antennas and a high-speed CAN and FlexRay interface. In at least one embodiment with six antennas, four central antennas may create a focused beam pattern designed to record the environment of the vehicle 800 at higher speeds with minimal interference from traffic in adjacent lanes. In at least one embodiment, two other antennas may expand the field of view so that vehicles entering or exiting a lane of the vehicle 800 can be quickly detected.

[0146] In at least one embodiment, mid-range radar systems may include, for example, a range of up to 160 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 150 degrees (rear). In at least one embodiment, short-range radar systems may include, among other things, any number of radar sensors 860 configured for installation at both ends of a rear bumper. When installed at both ends of a rear bumper in at least one embodiment, a radar sensor system may generate two beams that constantly monitor the blind spot in a direction rearward and near a vehicle. In at least one embodiment, short-range radar systems may be used in ADAS system 838 for blind spot detection and / or lane change assistance.

[0147] In at least one embodiment, the vehicle 800 may further include one or more ultrasonic sensors 862. In at least one embodiment, the ultrasonic sensors 862, which may be positioned at a location at the front, rear, and / or side of the vehicle 800, may be used for parking assistance and / or for generating and updating an occupancy grid. In at least one embodiment, a wide array of ultrasonic sensors 862 may be used, and different ultrasonic sensors 862 may be used for different detection ranges (e.g., 2.5 m, 4 m). In at least one embodiment, the ultrasonic sensor(s) 862 may operate to ASIL B functional safety levels.

[0148] In at least one embodiment, the vehicle 800 may include one or more LIDAR sensors 864. In at least one embodiment, the LIDAR sensor(s) 864 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. In at least one embodiment, the LIDAR sensor(s) 864 may operate to ASIL B functional safety level. In at least one embodiment, the vehicle 800 may include multiple LIDAR sensors 864 (e.g., two, four, six, etc.) that may use an Ethernet channel (e.g., to provide data to a Gigabit Ethernet switch).

[0149] In at least one embodiment, the LIDAR sensor(s) 864 may provide a list of objects and their distances for a 360-degree field of view. In at least one embodiment, a commercially available LIDAR sensor(s) 864 may, for example, have a stated range of approximately 100 m with an accuracy of 2 cm to 3 cm and support for a 100 Mbps Ethernet connection. In at least one embodiment, one or more non-protruding LIDAR sensors may be used. In such an embodiment, the LIDAR sensor(s) 864 may include a small device that may be embedded at a location in the front, rear, side, and / or corner of the vehicle 800.In at least one embodiment, the LIDAR sensor(s) 864 in such an embodiment may provide a horizontal field of view of up to 120 degrees and a vertical field of view of up to 35 degrees, with a range of 200 m, even for low-reflectivity objects. In at least one embodiment, one or more front-mounted LIDAR sensors 864 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0150] In at least one embodiment, LIDAR technologies, such as 3D flash LIDAR, may also be used. In at least one embodiment, 3D flash LIDAR uses a laser flash as a transmission source to illuminate the area surrounding the vehicle 800 to approximately 200 m. In at least one embodiment, a flash LIDAR unit includes, among other things, a receptor that records the laser pulse time of flight and reflected light on each pixel, which in turn corresponds to the range of the vehicle 800 to objects. In at least one embodiment, flash LIDAR can generate highly accurate and distortion-free images of the area surrounding it with each laser flash. In at least one embodiment, four flash LIDAR sensors can be employed, one on each side of the vehicle 800. In at least one embodiment, 3D flash LIDAR systems include, among other things, a solid-state, rigid-array 3D LIDAR camera with no moving parts, except for a fan (e.g.,a non-scanning LIDAR device). In at least one embodiment, a flash LIDAR device may use a 5-nanosecond Class I (eye-safe) laser pulse per frame and collect reflected laser light as a 3D range point cloud and co-registered intensity data.

[0151] In at least one embodiment, the vehicle 800 may further include one or more IMU sensors 866. In at least one embodiment, the IMU sensors 866 may be located at a center of a rear axle of the vehicle 800. In at least one embodiment, the IMU sensors 866 may include, for example, accelerometers, magnetometers, gyroscopes, a magnetic compass, magnetic compasses, and / or other types of sensors, among others. In at least one embodiment, such as in six-axis applications, the IMU sensors 866 may include, among others, accelerometers and gyroscopes. In at least one embodiment, such as in nine-axis applications, the IMU sensors 866 may include, among others, accelerometers, gyroscopes, and magnetometers.

[0152] In at least one embodiment, the IMU sensor(s) 866 may be implemented as a miniature, high-performance GPS-aided inertial navigation system (“GPS / INS”) that combines micro-electro-mechanical systems (MEMS)-based inertial sensors, a high-sensitivity GPS receiver, and advanced Kalman filter algorithms to provide position, velocity, and attitude estimates. In at least one embodiment, the IMU sensor(s) 866 may enable the vehicle 800 to estimate its heading without requiring input from a magnetic sensor by directly observing and correlating velocity changes from a GPS to the IMU sensor(s) 866. In at least one embodiment, the IMU sensor(s) 866 and the GNSS sensor(s) 858 may be combined into a single integrated unit.

[0153] In at least one embodiment, the vehicle 800 may include one or more microphones 896 mounted in and / or around the vehicle 800. In at least one embodiment, the microphone(s) 896 may be used for, among other things, emergency vehicle detection and identification.

[0154] In at least one embodiment, the vehicle 800 may further include any number of camera types, including stereo camera(s) 868, wide-angle camera(s) 870, infrared camera(s) 872, surround camera(s) 874, long-range camera(s) 898, medium-range camera(s) 876, and / or other camera types. In at least one embodiment, cameras may be used to capture image data around an entire periphery of the vehicle 800. In at least one embodiment, the camera types used depend on the vehicle 800. In at least one embodiment, any combination of camera types may be used to provide necessary coverage around the vehicle 800. In at least one embodiment, the number of cameras employed may vary depending on the embodiment.In at least one embodiment, the vehicle 800 may include, for example, six cameras, seven cameras, ten cameras, twelve cameras, or another number of cameras. In at least one embodiment, the cameras may support, for example, Gigabit Multimedia Serial Link ("GMSL") and / or Gigabit Ethernet communications, among others. In at least one embodiment, each camera may be configured as described hereinbefore with respect to [ ]. Fig. 8A and Fig. 8B will be described in more detail.

[0155] In at least one embodiment, vehicle 800 may further include vibration sensors 842. In at least one embodiment, vibration sensors 842 may measure vibrations of components of vehicle 800, such as axles. For example, in at least one embodiment, changes in vibrations may indicate a change in road surfaces. In at least one embodiment, when two or more vibration sensors 842 are used, differences between vibrations may be used to determine friction or slipperiness of the road surface (e.g., when there is a difference in vibration between a driven axle and a freely rotating axle).

[0156] In at least one embodiment, the vehicle 800 may include an ADAS system 838. In at least one embodiment, the ADAS system 838 may include, among other things, an SoC in some examples. In at least one embodiment, the ADAS system 838 may include, among other things, any number and combination of an autonomous / adaptive / automatic cruise control system ("ACC system"), a cooperative adaptive cruise control system ("CACC" system), a forward collision warning system ("FCW" system), an automatic emergency braking system ("AEB" system), a lane departure warning system ("LDW" system), a lane keeping assistance system ("LKA" system), a blind spot warning system ("BSW" system), a rear cross traffic alert system ("RCTW" system), a forward collision warning system ("CW" system), a lane centering system ("LC system"), and / or other systems, features, and / or functions.

[0157] In at least one embodiment, the ACC system may use RADAR sensors 860, LIDAR sensors 864, and / or any number of cameras. In at least one embodiment, the ACC system may include a longitudinal ACC system and / or a lateral ACC system. In at least one embodiment, the longitudinal ACC system monitors and controls the distance to another vehicle immediately in front of the vehicle 800 and automatically adjusts the speed of the vehicle 800 to maintain a safe distance from vehicles ahead. In at least one embodiment, the lateral ACC system maintains the distance and advises the vehicle 800 to change lanes if necessary. In at least one embodiment, the lateral ACC communicates with other ADAS applications, such as LC and CW.

[0158] In at least one embodiment, a CACC system utilizes information from other vehicles, which may be received via the network interface 824 and / or the wireless antenna(s) 826 from other vehicles over a wireless connection or indirectly via a network connection (e.g., over the Internet). In at least one embodiment, direct connections may be provided through a vehicle-to-vehicle ("V2V") communication link, while indirect connections may be provided through an infrastructure-to-vehicle ("I2V") communication link. Generally, V2V communication provides information about immediately ahead vehicles (e.g., vehicles immediately ahead of and in the same lane as vehicle 800), while I2V communication provides information about traffic further away.In at least one embodiment, a CACC system may include one or both of the I2V and V2V information sources. In at least one embodiment, a CACC system may be more reliable thanks to information about vehicles ahead of the vehicle 800 and has the potential to improve traffic flow and reduce congestion on the road.

[0159] In at least one embodiment, an FCW system is designed to alert a driver to a hazard so that such a driver can take corrective action. In at least one embodiment, an FCW system uses a forward-facing camera and / or radar sensors 860 coupled with a dedicated processor, FPGA, and / or ASIC electrically coupled to provide driver feedback, such as a display, speaker, and / or vibrating component. In at least one embodiment, an FCW system can provide a warning, e.g., in the form of a sound, a visual warning, a vibration, and / or a rapid braking pulse.

[0160] In at least one embodiment, an AEB system detects an impending forward collision with another vehicle or other object and may automatically apply the brakes if a driver does not take corrective action within a specified time or distance parameter. In at least one embodiment, the AEB system may utilize one or more forward-facing cameras and / or one or more radar sensors 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. In at least one embodiment, when an AEB system detects a hazard, it will typically first alert a driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system will automatically apply the brakes in an effort to avoid or at least mitigate the impact of a predicted collision.In at least one embodiment, the AEB system may include techniques such as dynamic brake support and / or imminent impact braking.

[0161] In at least one embodiment, an LDW system is equipped with visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle 800 crosses lane markings. In at least one embodiment, an LDW system is not activated when a driver indicates an intended lane departure, such as by activating a turn signal. In at least one embodiment, an LDW system may utilize forward-facing cameras coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to provide driver feedback, such as a display, speaker, and / or vibration component. In at least one embodiment, an LKA system is a variant of an LDW system.In at least one embodiment, an LKA system provides steering input or braking to correct the vehicle 800 when the vehicle 800 begins to depart from the lane.

[0162] In at least one embodiment, a BSW system detects and warns a driver of vehicles in the blind spot of an automobile. In at least one embodiment, a BSW system may provide a visual, audible, and / or tactile alert to indicate that merging or changing lanes is unsafe. In at least one embodiment, a BSW system may provide an additional warning when a driver uses a turn signal. In at least one embodiment, a BSW system may utilize rear-facing camera(s) and / or RADAR sensor(s) 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0163] In at least one embodiment, an RCTW system may provide visual, audible, and / or tactile notification when an object outside the range of the rearview camera is detected while reversing the vehicle 800. In at least one embodiment, an RCTW system includes an AEB system to ensure that the vehicle brakes are applied to avoid a collision. In at least one embodiment, an RCTW system may utilize one or more rear-facing RADAR sensors 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to provide driver feedback, such as a display, speaker, and / or vibrating component.

[0164] In at least one embodiment, conventional ADAS systems may be prone to false positives, which may be annoying and distracting for a driver, but are typically not catastrophic because conventional ADAS systems warn a driver and allow that driver to decide whether a safety condition actually exists and act accordingly. In at least one embodiment, the vehicle 800 itself decides, in the case of conflicting results, whether to consider a result from a primary computer or a secondary computer (e.g., a first controller or a second controller of controllers 436). For example, in at least one embodiment, the ADAS system 838 may be a backup and / or secondary computer that provides perception information to a rationality module of the backup computer.In at least one embodiment, redundant, diverse software on hardware components may run on a rationality monitor of the backup computer to detect perceptual errors and dynamic driving tasks. In at least one embodiment, outputs from the ADAS system 838 may be provided to a higher-level MCU. If outputs from a primary computer and outputs from a secondary computer conflict, in at least one embodiment, a higher-level MCU determines how to resolve the conflict to ensure safe operation.

[0165] In at least one embodiment, a primary computer may be configured to provide a parent MCU with a confidence value indicating that primary computer's confidence in a selected outcome. In at least one embodiment, if that confidence value exceeds a threshold, that parent MCU may follow that primary computer's guidance regardless of whether that secondary computer provides a conflicting or inconsistent outcome. In at least one embodiment, where a confidence value does not meet a threshold and the primary and secondary computers indicate different outcomes (e.g., a conflict), a parent MCU may arbitrate between the computers to determine an appropriate outcome.

[0166] In at least one embodiment, a parent MCU may be configured to operate a neural network(s) trained and configured to determine, based at least in part on outputs from the primary computer and outputs from the secondary computer, conditions under which that secondary computer will provide false alarms. In at least one embodiment, the neural network(s) in a parent MCU may learn when the output of a secondary computer can and cannot be trusted. For example, if that secondary computer is a radar-based FCW system in at least one embodiment, the neural network(s) in that parent MCU may learn when an FCW system identifies metallic objects that are not actually hazards, such asa drainage grate or a manhole cover that triggers an alarm. In at least one embodiment, when a secondary computing device is a camera-based LDW system, a neural network in a higher-level MCU can learn to override the LDW when cyclists or pedestrians are present and lane departure is actually the safest maneuver. In at least one embodiment, a higher-level MCU can include at least one DLA or GPU suitable for operating one or more neural networks with associated memory. In at least one embodiment, a higher-level MCU can include and / or be included as a component of SoC(s) 804.

[0167] In at least one embodiment, the ADAS system 838 may include a secondary processor that performs ADAS functionality using traditional computer vision rules. In at least one embodiment, this secondary processor may use classic computer vision rules (if-then), and the presence of one or more neural networks in a higher-level MCU may improve reliability, safety, and performance. For example, in at least one embodiment, diverse implementation and intentional non-identity make an overall system more fault-tolerant, particularly against errors caused by software (or software-hardware interface) functionality.For example, in at least one embodiment, if there is a software bug or error in software running on a primary computer and non-identical software code running on a secondary computer provides a consistent overall result, a higher-level MCU may have greater confidence that an overall result is correct and a bug in the software or hardware on that primary computer does not cause a significant error.

[0168] In at least one embodiment, an output of the ADAS system 838 may be fed into the perception block of a primary computer and / or the dynamic driving task block of the primary computer. For example, in at least one embodiment, if the ADAS system 838 displays a forward collision warning due to an object immediately ahead, a perception block may use this information in object identification. In at least one embodiment, a secondary computer may have its own neural network trained, thus reducing the risk of false positives, as described herein.

[0169] In at least one embodiment, the vehicle 800 may further include an infotainment SoC 830 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, in at least one embodiment, the infotainment system 830 may not be an SoC and may include, among other things, two or more discrete components. In at least one embodiment, the infotainment SoC 830 may include, among other things, a combination of hardware and software that may be used to provide the vehicle 800 with audio (e.g., music, a personal digital assistant, navigation directions, news, radio, etc.), video (e.g., television, movies, streaming, etc.), phone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g.,Navigation systems, reverse parking assistance, a radio data system, vehicle-related information such as fuel level, total distance traveled, brake fluid level, oil level, door open / closed, air filter information, etc.). The infotainment SoC 830 may include, for example, radios, record players, navigation systems, video players, USB and Bluetooth connectivity, car computers, in-car entertainment, WiFi, steering wheel audio controls, hands-free voice control, a heads-up display ("HUD"), HMI display 834, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, functions, and / or systems), and / or other components. In at least one embodiment, the infotainment SoC 830 may be further used to provide information (e.g.,visual and / or audible) to the user(s) of the vehicle 800, such as information from the ADAS system 838, autonomous driving information such as planned vehicle maneuvers, trajectories, information about the environment (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0170] In at least one embodiment, the infotainment SoC 830 may include any amount and type of GPU functionality. In at least one embodiment, the infotainment SoC 830 may communicate with other devices, systems, and / or components of the vehicle 800 via the bus 802. In at least one embodiment, the infotainment SoC 830 may be coupled to a higher-level MCU so that a GPU of an infotainment system may perform some self-controlling functions in the event that the primary controller(s) 836 (e.g., primary and / or backup computers of the vehicle 800) fail. In at least one embodiment, the infotainment system SoC 830 may place the vehicle 800 into a chauffeur-to-safe stop mode, as described herein.

[0171] In at least one embodiment, the vehicle 800 may further include an instrument cluster 832 (e.g., a digital instrument panel, an electronic instrument cluster, a digital instrument panel, etc.). In at least one embodiment, the instrument cluster 832 may include, among other things, a controller and / or supercomputer (e.g., a discrete controller or supercomputer). In at least one embodiment, the instrument cluster 832 may include, among other things, any number and combination of a set of instruments such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn signal indicator, gear shift indicator, seat belt warning light(s), parking brake warning light(s), engine malfunction light(s), supplemental restraint system information (e.g., airbag), lighting controls, safety system controls, navigation information, etc.In some examples, information may be displayed and / or shared between infotainment SoC 830 and instrument cluster 832. In at least one embodiment, instrument cluster 832 may be included as part of infotainment SoC 830, or vice versa.

[0172] The pixel density component 150 may be used to perform image processing operations, including pixel density calculations, associated with one or more embodiments. Details regarding the pixel density component 150 are described herein in connection with Fig. 1. In at least one embodiment, the pixel density component 150 may be incorporated in the system architecture of Fig. 8C to perform image processing operations, including pixel density calculations.

[0173] Fig. 8D is an illustration of a system 877 for communication between one or more cloud-based servers and the autonomous vehicle 800 of Fig. 8A, according to at least one embodiment. In at least one embodiment, system 877 may include, among other things, one or more servers 878, one or more networks 890, and any number and types of vehicles, including vehicle 800. In at least one embodiment, the server(s) 878 may include, among other things, a plurality of GPUs 484(A)-484(H) (collectively referred to herein as GPUs 484), PCIe switches 482(A)-482(D) (collectively referred to herein as PCIe switches 482), and / or CPUs 480(A)-480(B) (collectively referred to herein as CPUs 480). In at least one embodiment, the GPUs 884, CPUs 880, and PCIe switches 882 may be interconnected using high-speed interconnects, such as, but not limited to, NVLink interfaces 888 and / or PCIe interconnects 886 developed by NVIDIA.In at least one embodiment, the GPUs 884 are connected via an NVLink and / or NVSwitch SoC, and the GPUs 884 and PCIe switches 882 are connected via PCIe connections. Although eight GPUs 884, two CPUs 880, and four PCIe switches 882 are illustrated, this is not intended to be limiting. In at least one embodiment, each of the servers 878 may include, among other things, any number of GPUs 884, CPUs 880, and / or PCIe switches 882 in any combination. For example, in at least one embodiment, the server(s) 878 may each include eight, sixteen, thirty-two, and / or more GPUs 884.

[0174] In at least one embodiment, the server(s) 878 may receive, via the network(s) 890 and from vehicles, image data representing images depicting unexpected or changed road conditions, such as recently commenced road construction. In at least one embodiment, the server(s) 878 may transmit, via the network(s) 890 and to the vehicles, neural networks 892, updated or otherwise, and / or map information 894, including, but not limited to, information about traffic and road conditions. In at least one embodiment, updates to the map information 894 may include, but are not limited to, updates to an HD map 822, e.g., information about construction, potholes, detours, flooding, and / or other obstacles.In at least one embodiment, neural networks 892 and / or map information 894 may result from new training and / or new experience represented in data received from any number of vehicles in an environment and / or may be based at least in part on training performed in a data center (e.g., using one or more servers 878 and / or other servers).

[0175] In at least one embodiment, the server(s) 878 may be used to train machine learning models (e.g., neural networks) based at least in part on training data. In at least one embodiment, training data may be generated by vehicles and / or in a simulation (e.g., using a game engine). In at least one embodiment, any amount of training data is tagged (e.g., if the associated neural network benefits from supervised learning) and / or undergoes other preprocessing. In at least one embodiment, any amount of training data is untagged and / or preprocessed (e.g., if the associated neural network does not require supervised learning). In at least one embodiment, after machine learning models are trained, machine learning models of vehicles may be used (e.g.,transmitted to vehicles via network(s) 890) and / or machine learning models may be used by one or more servers 878 for remote vehicle monitoring.

[0176] In at least one embodiment, the server(s) 878 may receive data from vehicles and apply data to real-time neural networks for intelligent real-time inference. In at least one embodiment, the server(s) 878 may include deep learning supercomputers and / or dedicated AI machines powered by GPU(s) 884, such as the DGX and DGX Station machines developed by NVIDIA. However, in at least one embodiment, the server(s) 878 may include a deep learning infrastructure using CPU-powered data centers.

[0177] In at least one embodiment, the deep learning infrastructure of server(s) 878 may be capable of fast, real-time inferencing and may use this capability to evaluate and verify the state of processors, software, and / or associated hardware in vehicle 800. For example, in at least one embodiment, the deep learning infrastructure may receive periodic updates from vehicle 800, such as a sequence of images and / or objects that vehicle 800 has located in that sequence of images (e.g., through computer vision and / or other machine learning object classification techniques).In at least one embodiment, the deep learning infrastructure may run its own neural network to identify objects and compare them to the objects identified by the vehicle 800, and if the results do not match and the deep learning infrastructure concludes that the AI ​​in the vehicle 800 is faulty, the server(s) 878 may send a signal to the vehicle 800 and instruct a fail-safe computer of the vehicle 800 to take over control, notify the passengers, and perform a safe parking maneuver.

[0178] In at least one embodiment, the server(s) 878 may include GPU(s) 884 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT 3 devices). In at least one embodiment, a combination of GPU-powered servers and inference acceleration may enable real-time responsiveness. In at least one embodiment, such as when performance is less critical, servers powered by CPUs, FPGAs, and other processors may be used for inference purposes. In at least one embodiment, a hardware structure(s) implementing a pixel density component 150 is used to perform one or more embodiments. Details regarding the pixel density component 150 are described herein in connection with Fig. 1 provided.

[0179] Fig. 9 is a block diagram illustrating an example computing system, which may be a system with interconnected devices and components, a system-on-a-chip (SOC), or other combination thereof, formed with a processor, which may include execution units for executing an instruction, according to at least one embodiment. In at least one embodiment, a computing system 900 may include, among other things, a component, such as a processor 902, for employing execution units, including logic for performing algorithms on process data, in accordance with the present disclosure, such as in the embodiment described herein.In at least one embodiment, computing system 900 may include processors such as the PENTIUM® processor family, Xeon™, Itanium®, XScale™ and / or StrongARM™, Intel® Core™, or Intel® Nervana™ microprocessors available from Intel Corporation, Santa Clara, California, although other systems (including PCs with other microprocessors, engineering workstations, set-top boxes, and the like) may also be used. In at least one embodiment, computing system 900 may run a version of a WINDOWS operating system available from Microsoft Corporation, Redmond, Wash., although other operating systems (e.g., UNIX and Linux), embedded software, and / or graphical user interfaces may also be used.

[0180] Embodiments may be used in other devices, such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol devices, digital cameras, personal digital assistants ("PDAs"), and handheld PCs. In at least one embodiment, embedded applications may include a microcontroller, a digital signal processor ("DSP"), a system-on-a-chip, network computers ("NetPCs"), set-top boxes, network hubs, wide area network ("WAN") switches, or any other system capable of performing one or more instructions according to at least one embodiment.

[0181] In at least one embodiment, computing system 900 may include, among other things, processor 902, which may include, among other things, one or more execution units 908 for performing image processing and pixel density calculations according to techniques described herein. In at least one embodiment, computing system 900 is a single-processor desktop or server system, but in another embodiment, computing system 900 may be a multiprocessor system. In at least one embodiment, processor 902 may include, among other things, a complex instruction set computer ("CISC") microprocessor, a reduced instruction set computing ("RISC") microprocessor, a very long instruction word ("VLIW") microprocessor, a processor implementing a combination of instruction sets, or another processor device, such as a digital signal processor.In at least one embodiment, the processor 902 may be coupled to a processor bus 910 that transmits data signals between the processor 902 and other components in the computer system 900.

[0182] In at least one embodiment, processor 902 may include, among other things, an internal Level 1 ("L1") cache ("Cache") 904. In at least one embodiment, processor 902 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache may be external to processor 902. Other embodiments may also include a combination of both internal and external caches, depending on the particular implementation and needs. In at least one embodiment, a register file 906 may store different types of data in various registers, including, but not limited to, integer registers, floating-point registers, status registers, and an instruction pointer register.

[0183] In at least one embodiment, execution unit 908, including, among other things, logic for performing integer and floating-point operations, is also present in processor 902. In at least one embodiment, processor 902 may also include microcode ("ucode") read-only memory ("ROM") that stores microcode for certain macroinstructions. In at least one embodiment, execution unit 908 may include logic for handling a packaged instruction set 909. In one embodiment, by including a packaged instruction set 909 in an instruction set of a general-purpose processor, along with associated circuitry for executing instructions, operations used by many multimedia applications may be performed using packaged data in a general-purpose processor 902.In at least one embodiment, many multimedia applications may be accelerated and executed more efficiently by using a full width of a processor data bus to perform operations on packed data, thereby eliminating a need to transfer smaller units of data across that processor data bus to perform one or more operations on one data item at a time.

[0184] In at least one embodiment, execution unit 908 may also be used in microcontrollers, embedded processors, graphics devices, DSPs, and other types of logic circuits. In at least one embodiment, computing system 900 may include, among other things, a memory 920. In at least one embodiment, memory 920 may be a dynamic random access memory ("DRAM") device, a static random access memory ("SRAM") device, a flash memory device, or another storage device. In at least one embodiment, memory 920 may store instruction(s) 919 and / or data 921, represented by data signals, that may be executed by processor 902.

[0185] In at least one embodiment, a system logic chip may be coupled to the processor bus 910 and the memory 920. In at least one embodiment, a system logic chip may include, among other things, a memory controller hub (“MCH”) 916, and the processor 902 may communicate with the MCH 916 via the processor bus 910. In at least one embodiment, the MCH 916 may provide a high-bandwidth memory path 918 to the memory 920 for instruction and data storage and for storage of graphics commands, data, and textures. In at least one embodiment, the MCH 916 may route data signals between the processor 902, memory 920, and other components in the computing system 900, and for bridging data signals between the processor bus 910, memory 920, and a system I / O interface 922. In at least one embodiment, a system logic chip may provide a graphics port for coupling to a graphics controller.In at least one embodiment, the MCH 916 may be coupled to the memory 920 via a high-bandwidth memory path 918, and a graphics / video card 912 may be coupled to the MCH 916 via an Accelerated Graphics Port ("AGP") interconnect 914.

[0186] In at least one embodiment, computing system 900 may use system I / O interface 922 as a proprietary hub interface bus to couple MCH 916 to an I / O controller hub ("ICH") 930. In at least one embodiment, ICH 930 may provide direct connections to some I / O devices via a local I / O bus. In at least one embodiment, a local I / O bus may include, among other things, a high-speed I / O bus for connecting peripherals to memory 920, a chipset, and processor 902. Examples may include, but are not limited to, an audio controller 929, a firmware hub (“Flash BIOS”) 928, a wireless transceiver 926, data storage 924, a legacy I / O controller 923 containing user input and keyboard interfaces 925, a serial expansion port 927, such as a Universal Serial Bus (“USB”) port, and a network controller 934.In at least one embodiment, data storage 924 may include a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device.

[0187] In at least one embodiment, Fig. 9 a system including interconnected hardware devices or “chips”, while in other embodiments Fig. 9 may illustrate an exemplary SoC. In at least one embodiment, Fig. 9 may be interconnected using proprietary interconnects, standardized interconnects (e.g., PCIe), or a combination thereof. In at least one embodiment, one or more components of computing system 900 are interconnected using Compute Express Link (CXL) interconnects.

[0188] The pixel density component 150 may be used to perform image processing operations, including pixel density calculations, associated with one or more embodiments. Details regarding the pixel density component 150 are described herein in connection with Fig. 1. In at least one embodiment, the pixel density component 150 may be incorporated in the system architecture of Fig. 9 to perform image processing operations, including pixel density calculations.

[0189] Fig. 10 is a block diagram illustrating an electronic device 1000 for utilizing a processor 1010 according to at least one embodiment. In at least one embodiment, the electronic device 1000 may be, for example, among others, a notebook, a tower server, a rack server, a blade server, a laptop, a desktop, a tablet, a mobile device, a phone, an embedded computer, or any other suitable electronic device.

[0190] In at least one embodiment, electronic device 1000 may include, among other things, processor 1010 communicatively coupled to any suitable number or type of components, peripherals, modules, or devices. In at least one embodiment, processor 1010 is connected to the processor using a bus or interface, such as an I 2C-Bus, a System Management Bus (SMBus), a Low Pin Count (LPC) bus, a Serial Peripheral Interface (SPI), a High Definition Audio (HDA) bus, a Serial Advance Technology Attachment (SATA) bus, a Universal Serial Bus (USB) (versions 1, 2, 3, etc.), or a Universal Asynchronous Receiver / Transmitter (UART) bus. In at least one embodiment, Fig. 10 a system including interconnected hardware devices or “chips”, while in other embodiments Fig. 10 may illustrate an exemplary SoC. In at least one embodiment, Fig. 10 illustrated devices may be interconnected with proprietary interconnects, standardized interconnects (e.g., PCIe), or a combination thereof. In at least one embodiment, one or more components of Fig. 10 interconnected using Compute Express Link (CXL) interconnects.

[0191] In at least one embodiment, Fig. 8 a display 1024, a touchscreen 1025, a touchpad 1030, a near-field communication unit (“NFC”) 1045, a sensor hub 1040, a thermal sensor 1046, an express chipset (“EC”) 1035, a trusted platform module (“TPM”) 1038, a BIOS / firmware / flash memory (“BIOS, FW Flash”) 1022, a DSP 1060, a drive 1020 such as a solid-state drive (“SSD”) or a hard disk drive (“HDD”), a wireless local area network unit (“WLAN”) 1050, a Bluetooth unit 1052, a wireless wide-area network unit (“WWAN”) 1056, a global positioning system unit (GPS unit) 1055, a camera (“USB 3.0 camera”) 1054 such as a A USB 3.0 camera and / or a low-power double data rate ("LPDDR") memory device ("LPDDR3") 1015, for example, implemented in an LPDDR3 standard. These components may each be implemented in any suitable manner.

[0192] In at least one embodiment, other components may be communicatively coupled to processor 1010 via components described herein. In at least one embodiment, an accelerometer 1041, an ambient light sensor (ALS) 1042, a compass 1043, and a gyroscope 1044 may be communicatively coupled to sensor hub 1040. In at least one embodiment, a thermal sensor 1039, a fan 1037, a keyboard 1036, and a touchpad 1030 may be communicatively coupled to EC 1035. In at least one embodiment, a speaker 1063, headphones 1064, and a microphone ("Mic") 1065 may be communicatively coupled to an audio unit ("Audio Codec and Class-D Amplifier") 1062, which in turn may be communicatively coupled to DSP 1060. For example, in at least one embodiment, the audio unit 1062 may include, among other things, an audio encoder / decoder (“codec”) and a Class-D amplifier.In at least one embodiment, a SIM card ("SIM") 1057 may be communicatively coupled to the WWAN unit 1056. In at least one embodiment, components such as the WLAN unit 1050 and Bluetooth unit 1052, as well as the WWAN unit 1056, may be implemented in a next-generation form factor ("NGFF").

[0193] The pixel density component 150 may be used to perform image processing operations, including pixel density calculations, associated with one or more embodiments. Details regarding the pixel density component 150 are described herein in connection with Fig. 1. In at least one embodiment, the pixel density component 150 may be incorporated in the system architecture of Fig. 10 can be used to perform image processing operations, including pixel density calculations.

[0194] Fig. 11 is a block diagram of a processing system according to at least one embodiment. In at least one embodiment, system 1100 includes one or more processors 1102 and one or more graphics processors 1108, and may be a single-processor desktop system, a multiprocessor workstation system, or a server system with a large number of processors 1102 or processor cores 1107. In at least one embodiment, system 1100 is a processing platform embodied in a system-on-a-chip (SoC) integrated circuit for use in mobile, handheld, or embedded devices.

[0195] In at least one embodiment, system 1100 may include or be included in a server-based gaming platform, a gaming console, including a game and media console, a mobile gaming console, a handheld gaming console, or an online gaming console. In at least one embodiment, system 1100 is a mobile phone, a smartphone, a tablet computing device, or a mobile internet device. In at least one embodiment, processing system 1100 may also include, be coupled to, or integrated with a wearable device, such as a wearable smartwatch device, a smartglass device, an augmented reality device, or a virtual reality device.In at least one embodiment, the processing system 1100 is a television or set-top box device having one or more processors 1102 and a graphical interface generated by one or more graphics processors 1108.

[0196] In at least one embodiment, the one or more processors 1102 each include one or more processor cores 1107 to process instructions that, when executed, perform operations for system and user software. In at least one embodiment, the one and more processor cores 1107 are each configured to process a specific instruction sequence 1109. In at least one embodiment, the instruction sequence 1109 may facilitate complex instruction set computing (CISC), reduced instruction set computing (RISC), or via very long instruction word (VLIW) computing. In at least one embodiment, the processor cores 1107 may each process a different instruction sequence 1109, which may include instructions to facilitate emulation of other instruction sequences.In at least one embodiment, the processor cores 1107 may also include other processing devices, such as a digital signal processor (DSP).

[0197] In at least one embodiment, processor 1102 may include a cache memory 1104. In at least one embodiment, processor 1102 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache memory is shared by various components of processor 1102. In at least one embodiment, processor 1102 also uses an external cache (e.g., a Level 3 (L3) cache or Last Level Cache (LLC)) (not shown), which may be shared by processor cores 1107 using known cache coherence techniques. In at least one embodiment, a register file 1106 is additionally included in processor 1102, which may include different types of registers for storing different types of data (e.g., integer registers, floating-point registers, status registers, and an instruction pointer register).In at least one embodiment, register file 1106 may include general-purpose registers or other registers.

[0198] In at least one embodiment, one or more processors 1102 are coupled to one or more interface buses 1110 for transmitting communication signals, such as address, data, or control signals, between processor 1102 and other components in system 1100. In at least one embodiment, interface bus 1110 may be a processor bus, such as a version of a Direct Media Interface (DMI) bus. In at least one embodiment, interface bus 1110 is not limited to a DMI bus and may include one or more Peripheral Component Interconnect buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. In at least one embodiment, processor(s) 1102 may include an integrated memory controller 1116 and a platform controller hub 1130.In at least one embodiment, the memory controller 1116 facilitates communication between a memory device and other components of the system 1100, while the platform controller hub (PCH) 1130 provides connections to I / O devices via a local I / O bus.

[0199] In at least one embodiment, a memory device 1120 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase-change memory device, or other memory device having suitable performance to serve as process memory. In at least one embodiment, the memory device 1120 may operate as system memory for the system 1100 to store data 1122 and instructions 1121 for use when one or more processors 1102 are executing an application or process. In at least one embodiment, the memory controller 1116 is also coupled to an optional external graphics processor 1112 that can communicate with one or more graphics processors 1108 in the processors 1102 to perform graphics and media operations.In at least one embodiment, a display device 1111 may be connected to the processor(s) 1102. In at least one embodiment, the display device 1111 may include one or more of an internal display device, such as in a mobile electronic device or a laptop device, or an external display device connected via a display interface (e.g., DisplayPort, etc.). In at least one embodiment, the display device 1111 may include a head-mounted display (HMD), such as a stereoscopic display device for use in virtual reality (VR) applications or augmented reality (AR) applications.

[0200] In at least one embodiment, the platform controller hub 1130 may enable peripherals to connect to the storage device 1120 and the processor 1102 via a high-speed I / O bus. In at least one embodiment, I / O peripherals include, among others, an audio controller 1146, a network controller 1134, a firmware interface 1128, a wireless transceiver 1126, touch sensors 1125, and a data storage device 1124 (e.g., hard disk drive, flash memory, etc.). In at least one embodiment, the data storage device 1124 may be connected via a storage interface (e.g., SATA) or via a peripheral bus, such as a Peripheral Component Interconnect bus (e.g., PCI, PCI Express). In at least one embodiment, the touch sensors 1125 may include touchscreen sensors, pressure sensors, or fingerprint sensors.In at least one embodiment, the wireless transceiver 1126 may be a WiFi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, or Long Term Evolution (LTE) transceiver. In at least one embodiment, the firmware interface 1128 enables communication with the system firmware and may be, for example, a Unified Extensible Firmware Interface (UEFI). In at least one embodiment, the network controller 1134 enables network connection to a wired network. In at least one embodiment, a high-performance network controller (not shown) is coupled to the interface bus 1110. In at least one embodiment, the audio controller 1146 is a multi-channel high-definition audio controller. In at least one embodiment, the system 1100 includes an optional legacy I / O controller 1140 for coupling legacy devices (e.g.,Personal System 2 (PS / 2) devices) to the system 1100. In at least one embodiment, the platform controller hub 1130 may also be connected to one or more Universal Serial Bus (USB) controller 1142 interconnect input devices, such as keyboard and mouse combinations 1143, a camera 1144, or other USB input devices.

[0201] In at least one embodiment, an instance of the memory controller 1116 and platform controller hub 1130 may be integrated into a discrete external graphics processor, such as the external graphics processor 1112. In at least one embodiment, the platform controller hub 1130 and / or the memory controller 1116 may be external to one or more processors 1102. For example, in at least one embodiment, the system 1100 may include an external memory controller 1116 and platform controller hub 1130, which may be configured as a memory controller hub and a peripheral controller hub within a system chipset in communication with the processor(s) 1102.

[0202] The pixel density component 150 may be used to perform image processing operations, including pixel density calculations, associated with one or more embodiments. Details regarding the pixel density component 150 are described herein in connection with Fig. 1. In at least one embodiment, the pixel density component 150 may be incorporated in the system architecture of Fig. 11 to perform image processing operations, including pixel density calculations.

[0203] Fig. 12 is a block diagram of a processor 1200 including one or more processor cores 1202A-1202N, an integrated memory controller 1214, and an integrated graphics processor 1208, according to at least one embodiment. In at least one embodiment, the processor 1200 may include additional cores up to and including additional core 1202N, represented by dashed boxes. In at least one embodiment, the processor cores 1202A-1202N each include one or more internal cache units 1204A-1204N. In at least one embodiment, each processor core also has access to one or more shared cache units 1206.

[0204] In at least one embodiment, internal cache units 1204A-1204N and shared cache units 1206 represent a cache hierarchy within processor 1200. In at least one embodiment, cache units 1204A-1204N may include at least one level of instruction and data cache within each processor core and one or more levels of shared mid-level cache, such as Level 2 (L2), Level 3 (L3), Level 4 (L4), or other levels of cache, with a highest level of cache prior to external memory classified as LLC. In at least one embodiment, cache coherence logic maintains coherence between various cache units 1206 and 1204A-1204N.

[0205] In at least one embodiment, the processor 1200 may also include a set of one or more bus controller units 1216 and a system agent core 1210. In at least one embodiment, one or more bus controller units 1216 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, the system agent core 1210 provides management functionality for various processor components. In at least one embodiment, the system agent core 1210 includes one or more integrated memory controllers 1214 for managing access to various external memory devices (not shown).

[0206] In at least one embodiment, one or more processor cores 1202A-1202N include support for simultaneous multithreading. In at least one embodiment, system agent core 1210 includes components for coordinating and operating cores 1202A-1202N during multithreaded processing. In at least one embodiment, system agent core 1210 may additionally include a power control unit (PCU) that includes logic and components for regulating one or more power states of processor cores 1202A-1202N and graphics processor 1208.

[0207] In at least one embodiment, processor 1200 additionally includes graphics processor 1208 to perform graphics processing operations. In at least one embodiment, graphics processor 1208 is coupled to shared cache units 1206 and system agent core 1210, including one or more integrated memory controllers 1214. In at least one embodiment, system agent core 1210 also includes a display controller 1211 to drive the graphics processor output to one or more coupled displays. In at least one embodiment, display controller 1211 may also be a separate module coupled to graphics processor 1208 via at least one interconnect or may be integrated with graphics processor 1208.

[0208] In at least one embodiment, a ring-based interconnect 1212 is used to couple internal components of processor 1200. In at least one embodiment, an alternative interconnect may be used, such as a point-to-point interconnect, a switched interconnect, or other techniques. In at least one embodiment, graphics processor 1208 is coupled to ring interconnect 1212 via an I / O link 2113.

[0209] In at least one embodiment, I / O link 1213 represents at least one of several types of I / O interconnects, including an on-packet I / O interconnect that facilitates communication between various processor components and an embedded high-performance memory module 1218, such as an eDRAM module. In at least one embodiment, processor cores 1202A-1202N and graphics processor 1208 each use embedded memory module 1218 as a shared last-level cache.

[0210] In at least one embodiment, processor cores 1202A-1202N are homogeneous cores executing a common instruction set architecture. In at least one embodiment, processor cores 1202A-1202N are heterogeneous with respect to instruction set architecture (ISA), where one or more of processor cores 1202A-1202N execute a common instruction set, while one or more other cores of processor cores 1202A-1202N execute a subset of a common instruction set or a different instruction set. In at least one embodiment, processor cores 1202A-1202N are microarchitecturally heterogeneous, where one or more cores having relatively higher power consumption are coupled with one or more performance cores having lower power consumption.In at least one embodiment, the processor 1200 may be implemented on one or more chips or as an SoC integrated circuit.

[0211] The pixel density component 150 may be used to perform image processing operations, including pixel density calculations, associated with one or more embodiments. Details regarding the pixel density component 150 are described herein in connection with Fig. 1. In at least one embodiment, the pixel density component 150 may be included in the graphics processor 1208. For example, in at least one embodiment, image processing and / or pixel density calculations described herein may include one or more in a 3D pipeline, graphics processor core(s) 1202, shared functional logic, or other logic in Fig. 12 embodied ALUs. Furthermore, in at least one embodiment, image processing and / or pixel density calculations described herein may be performed using logic different from that described in Fig. 1. In at least one embodiment, parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown) that configure ALUs of a processor 1200 to perform one or more image processing and / or pixel density techniques described herein.

[0212] The disclosure of this application also includes the following numbered clauses: Clause 1. A procedure which includes: Processing a first synthetic image comprising a first pattern and a second synthetic image comprising a second pattern to generate a merged image comprising a first modified version of the first synthetic image and a modified version of the second synthetic image, wherein one or a plurality of features of the first pattern are deformed in the modified version of the first synthetic image and one or more features of the second pattern are deformed in the modified version of the second synthetic image; Determining the pixel areas for one or more features of the modified version of the first synthetic image and for the one or more features of the modified version of the second synthetic image; and Determining one or more feature densities for one or more regions of the merged image based at least on the determined pixel areas. Clause 2. The method of Clause 1, wherein determining the one or more feature densities for the one or more regions of the merged image comprises: Determining the one or more feature densities based at least on area ratios between the pixel areas of the one or more features of the modified version of the first synthetic image and the second synthetic image and corresponding pixel areas of the one or more features of the first synthetic image and the second synthetic image, wherein the area ratios indicate magnification levels of respective features. Clause 3. The method of Clause 1 or Clause 2, wherein the one or more feature densities are determined in one or more stages of an image stitching pipeline. Clause 4. Procedure under any preceding clause, further comprising: responsive to determining that one or more feature densities within a first region of the merged image are below a first threshold density, Adjusting a first parameter associated with a stitching algorithm that generates the stitched images; and responsive to determining that one or more feature densities within a first region of the stitched image are below a second threshold density, adjusting a second parameter associated with the stitching algorithm. Clause 5. The method of any preceding clause, wherein determining the pixel areas for the one or more features of the modified version of the first synthetic image and for the one or more features of the modified version of the second synthetic image comprises: performing a thresholding process on the merged image to obtain a binary image, wherein the one or more features of the modified version of the first synthetic image and the one or more features of the modified version of the second synthetic image are classified as foreground objects; Providing the binary image as input to a connected component algorithm, wherein the connected component algorithm provides, as output, a labeled image comprising a plurality of connected components, wherein at least one connected component is assigned a unique identifier corresponding to a feature of the modified version of the first synthetic image or a feature of the modified version of the second synthetic image; and Determining pixel areas for each connected component using one or more shape analysis algorithms. Clause 6. Procedure under any preceding clause, further comprising: Determining orientations for each connected component using a covariance matrix of associated pixel coordinates, wherein the orientations indicate a direction of deformation associated with the deformed features of the first pattern or the deformed features of the second pattern. Clause 7. Procedure under any preceding clause, further comprising: Generating a vector map based at least on the pixel areas of the one or more features of the modified version of the first synthetic image and the second synthetic image and associated orientations of the one or more features of the first synthetic image and the second synthetic image. Clause 8. The method of any preceding clause, wherein the first pattern and the second pattern comprise a plurality of shapes arranged in a grid. Clause 9. The method of any preceding clause, wherein the first synthetic image and the second synthetic image are a same image comprising a same pattern. Clause 10. Procedure under any preceding clause, further comprising: Generating the first synthetic image and the second synthetic image. Clause 11. A system comprising: a variety of image capture devices. an image stitching pipeline coupled to the plurality of image capture devices, the image stitching pipeline to: Processing a first synthetic image comprising a first pattern and a second synthetic image comprising a second pattern to generate a merged image comprising a first modified version of the first synthetic image and a modified version of the second synthetic image, wherein one or more features of the first pattern are included in the modified version of the first synthetic image are deformed and one or more features of the second pattern are deformed in the modified version of the second synthetic image; Determining the pixel areas for one or more features of the modified version of the first synthetic image and for the one or more features of the modified version of the second synthetic image; and determining one or more feature densities for one or more regions of the merged image based at least on the determined pixel areas. Clause 12. The system of Clause 11, wherein to determine the one or more feature densities for one or more regions of the stitched image based at least on the determined pixel areas, the image stitching pipeline is to: Determining the one or more feature densities based at least on area ratios between the pixel areas of the features of the modified version of the first synthetic image and the second synthetic image, and corresponding pixel areas of the one or more features of the first synthetic image and the second synthetic image, wherein the area ratios indicate magnification levels of respective features. Clause 13. The system of Clause 11 or Clause 12, wherein the one or more feature densities are determined in one or more stages of an image stitching pipeline. Clause 14. The system of any of Clauses 11-13, wherein the image stitching pipeline further serves to: responsive to a determination that one or more feature densities within a first region of the merged image are below a first threshold density, Adjusting a first parameter associated with a stitching algorithm that generates the stitched images; and responsive to a determination that one or more feature densities within a first region of the stitched image are below a second threshold density, adjusting a second parameter associated with the stitching algorithm. Clause 15. The system of any of clauses 11-14, wherein for determining the pixel regions for the one or more features of the modified version of the first synthetic image and for the one or more features of the modified version of the second synthetic image, the image stitching pipeline is to: performing a thresholding process on the merged image to obtain a binary image, wherein the one or more features of the modified version of the first synthetic image and the one or more features of the modified version of the second synthetic image are classified as foreground objects; Providing the binary image as input to a connected component algorithm, wherein the connected component algorithm provides, as output, a labeled image comprising a plurality of connected components, each connected component being assigned a unique identifier corresponding to a feature of the modified version of the first synthetic image or a feature of the modified version of the second synthetic image; and determining pixel regions for each connected component using one or more shape analysis algorithms. Clause 16. The system of any of Clauses 11-15, wherein the image stitching pipeline further serves to: Determining orientations for each connected component using a covariance matrix of associated pixel coordinates, wherein the orientations indicate a direction of deformation associated with the deformed features of the first pattern or the deformed features of the second pattern. Clause 17. The system of any of Clauses 11-16, wherein the image stitching pipeline further serves to: Generating a vector map based on the pixel areas of the features of the modified version of the first synthetic image and the second synthetic image and associated orientations of the features of the first synthetic image and the second synthetic image. Clause 18. A system according to any one of clauses 11-17, wherein the first pattern and the second pattern comprise a plurality of shapes arranged in a grid. Clause 19. A system under any of Clauses 11-18, where the system consists of at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for carrying out one or more digital twinning operations; a system for performing a light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system for presenting at least one of augmented reality content, virtual reality content, or mixed reality content; a system for hosting one or more real-time streaming applications; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more conversational AI operations; a system that implements one or more language models; a system that implements one or more large language models (LLMs); a system that implements one or more visual language models (VLMs); a system for performing one or more generative AI operations; a system for generating synthetic data; a system that includes one or more virtual machines (VMs); a system that is implemented at least partially in a data center; or a system implemented at least in part using cloud computing resources. Clause 20. One or more processors comprising processing circuitry for: Processing a first synthetic image comprising a first pattern and a second synthetic image comprising a second pattern to generate a merged image comprising a first modified version of the first synthetic image and a modified version of the second synthetic image, wherein one or a plurality of features of the first pattern are deformed in the modified version of the first synthetic image and one or more features of the second pattern are deformed in the modified version of the second synthetic image; Determining the pixel areas for one or more features of the modified version of the first synthetic image and for the one or more features of the modified version of the second synthetic image; and Determining feature densities for one or more regions of the merged image based at least on the determined pixel areas.

[0213] Other variations are within the spirit of the present disclosure. Therefore, while the disclosed techniques are susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are illustrated in the drawings and have been described in detail above. It should be understood, however, that there is no intention to limit the disclosure to any specific form or forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents that are within the spirit and scope of the disclosure as defined in the appended claims.

[0214] The use of the terms "a" and "an" and "the" and reference designations in the context of describing the disclosed embodiments (particularly in the context of the claims below) are to be interpreted to cover both singular and plural, unless otherwise stated herein or clearly contradicted by the context, and not as a definition of any term. The terms "comprising," "having," "including," and "containing" are to be interpreted as open-ended terms (meaning "including, but not limited to") unless otherwise noted. "Connected," when unmodified and referring to physical connections, is to be interpreted as partially or wholly contained within, attached to, or joined with, even if something in between.The repetition of ranges of values ​​herein is intended as a quick way to individually refer to each separate value falling within the range, unless otherwise noted herein, and each separate value is incorporated into the specification as if individually recited herein. In at least one embodiment, the use of the term "set" (e.g., "a set of things") or "subset," unless otherwise noted or contradicted by context, is to be construed as a non-empty collection comprising one or more items. Furthermore, unless otherwise noted or contradicted by context, the term "subset" of a corresponding set does not necessarily denote a proper subset of the corresponding set; however, the subset and corresponding set may be the same.

[0215] Conjunctive expression, such as expressions of the form "at least one of A, B, and C" or "at least one of A, B, and C," unless specifically indicated otherwise or otherwise clearly contradicted by the context, is otherwise to be understood with the context as it is generally used to present that a thing, concept, etc., can be either A or B or C, or a non-empty subclause of the set of A and B and C. For example, in an illustrative example of a set having three elements, the conjunctive expressions "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Such conjunctive language is therefore generally not intended to imply that certain embodiments require that at least one of A, at least one of B, and at least one of C be present.Furthermore, unless otherwise noted or contradicted by context, the term "plurality" indicates that a state is plural (e.g., "a plurality of things" indicates multiple things). In at least one embodiment, the number of things in a plurality is at least two, but may be more if indicated either explicitly or by context. Furthermore, unless otherwise noted or clear from context, the phrase "based on" means "at least partially based on" and not "solely based on."

[0216] Operations of processes described herein may be performed in any suitable order, unless otherwise specified herein or otherwise clearly contradicted by context. In at least one embodiment, a process such as those described herein (or variations and / or combinations thereof) is performed under the control of one or more computer systems configured with executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executed collectively on one or more processors, by hardware, or combinations thereof. In at least one embodiment, code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors.In at least one embodiment, a computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., propagating transient electrical or electromagnetic transmission) but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues) within transceivers of transient signals. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media having executable instructions stored thereon (or on other memory to store executable instructions) that, when executed by one or more processors of a computer system (e.g., as a result of being executed), cause a computer system to perform the operations described herein.In at least one embodiment, a set of non-transitory computer-readable storage media comprises a plurality of non-transitory computer-readable storage media, and one or more individual non-transitory storage media of a plurality of non-transitory computer-readable storage media lack all code, while a plurality of non-transitory computer-readable storage media together store all code. In at least one embodiment, executable instructions are executed such that different instructions are executed by different processors—for example, a non-transitory computer-readable storage medium stores instructions, and a main central processing unit ("CPU") executes some instructions, while a graphics processing unit ("GPU") executes other instructions.In at least one embodiment, different components of a computer system have separate processors, and different processors execute different subsets of instructions.

[0217] Accordingly, in at least one embodiment, computing systems are configured to implement one or more services that alone or collectively perform operations of processes described herein, and such computing systems are configured with applicable hardware and / or software that enable operations to be performed. Furthermore, a computing system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment, is a distributed computing system that includes multiple devices that operate differently such that distributed computing systems perform operations described herein and such that a single device does not perform all operations.

[0218] The use of any example or exemplary language (e.g., "such as") provided herein is intended only to clarify embodiments of the disclosure and is not intended to limit the scope of the disclosure unless otherwise claimed. No language in the specification should be construed to imply that any unclaimed element is essential to the practice of the disclosure.

[0219] All references, including publications, patent applications, and patents cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically incorporated by reference and set forth in its entirety herein.

[0220] Throughout the description and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It should be understood that these terms may not be intended as synonyms. Rather, in specific examples, "connected" or "coupled" may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" may also mean that two or more elements are not in direct contact with each other, but nevertheless cooperate or interact with each other.

[0221] Unless specifically stated otherwise, it may be recognized throughout the specification that terms such as "processing," "computing," "calculating," "determining," or the like, refer to operations and / or processes of a computer or computing system, or similar electronic computing device, that manipulate data represented as physical, such as electronic, quantities within the registers and / or memories of the computing system and / or transform data into other data similarly represented as physical quantities within the memories, registers, or other such information storage, transmission, or display devices of the computing system.

[0222] Similarly, the term "processor" may refer to a device or part of a device that processes electronic data from registers and / or memory and transforms that electronic data into other electronic data that can be stored in registers and / or memory. As non-limiting examples, the "processor" may be a CPU or a GPU. A "computing platform" may include one or more processors. As used herein, "software" processes may include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Furthermore, any process may refer to multiple processes for executing instructions sequentially or in parallel, continuously or intermittently.In at least one embodiment, the terms "system" and "method" are used interchangeably herein in that the system may include one or more methods and methods may be considered a system.

[0223] In this document, references may be made to obtaining, acquiring, receiving, or inputting analog or digital data to a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data may be implemented in various ways, such as by receiving data as a parameter of a function call or a call to an application programming interface. In at least one embodiment, the processes of obtaining, acquiring, receiving, or inputting analog and digital data may be implemented by transferring data over a serial or parallel interface.In at least one embodiment, the processes of obtaining, capturing, receiving, or inputting analog or digital data may be implemented by transferring data across a computer network from a providing entity to a capturing entity. In at least one embodiment, references may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, processes for providing, outputting, transmitting, sending, or presenting analog or digital data may be implemented by transferring data as an input or output parameter of a function call, a parameter of an application programming interface, or an inter-process communication mechanism.

[0224] Although descriptions herein set forth exemplary implementations of described techniques, other architectures may be used to implement the described functionality and are intended to be within the scope of this disclosure. Furthermore, although specific distributions of responsibilities may be defined above for the purpose of description, various functions and responsibilities may be distributed and divided in different ways depending on the circumstances.

[0225] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or acts described. Rather, specific features and acts are disclosed as exemplary forms for implementing the claims.

[0226] It is to be understood that aspects and embodiments described above are merely exemplary and that modifications in detail may be made within the scope of the claims.

[0227] Each device, method, and feature disclosed in the description and (if applicable) in the claims and drawings may be provided independently or in any suitable combination.

[0228] Reference signs appearing in the claims are for illustrative purposes only and are not intended to have a limiting effect on the scope of the claims. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] US Department of Transportation, and Society of Automotive Engineers (“SAE”) “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (e.g., Standard No. J3016-201806, published June 15, 2018

[0077]

Claims

[1] A process comprising: Processing a first synthetic image comprising a first pattern and a second synthetic image comprising a second pattern to generate a merged image comprising a first modified version of the first synthetic image and a modified version of the second synthetic image, wherein one or more features of the first pattern are deformed in the modified version of the first synthetic image and one or more features of the second pattern are deformed in the modified version of the second synthetic image; Determining the pixel areas for one or more features of the modified version of the first synthetic image and for the one or more features of the modified version of the second synthetic image; and Determining one or more feature densities for one or more regions of the merged image based at least on the determined pixel areas. [2] The method of claim 1, wherein determining the one or more feature densities for the one or more regions of the merged image comprises: Determining the one or more feature densities based at least on area ratios between the pixel areas of the one or more features of the modified version of the first synthetic image and the second synthetic image and corresponding pixel areas of the one or more features of the first synthetic image and the second synthetic image, wherein the area ratios indicate magnification levels of respective features. [3] The method of claim 1 or claim 2, wherein the one or more feature densities are determined in one or more stages of an image stitching pipeline. [4] A method according to any preceding claim, further comprising: responsive to determining that one or more feature densities within a first region of the stitched image are below a first threshold density, adjusting a first parameter associated with a stitching algorithm that generates the stitched images; and responsive to determining that one or more feature densities within a first region of the stitched image are below a second threshold density, adjusting a second parameter associated with the stitching algorithm. [5] A method according to any preceding claim, wherein determining the pixel areas for the one or more features of the modified version of the first synthetic image and for the one or more features of the modified version of the second synthetic image comprises: performing a thresholding process on the merged image to obtain a binary image, wherein the one or more features of the modified version of the first synthetic image and the one or more features of the modified version of the second synthetic image are classified as foreground objects; Providing the binary image as input to a connected component algorithm, wherein the connected component algorithm provides, as output, a labeled image comprising a plurality of connected components, wherein at least one connected component is assigned a unique identifier corresponding to a feature of the modified version of the first synthetic image or a feature of the modified version of the second synthetic image; and Determining pixel areas for each connected component using one or more shape analysis algorithms. [6] The method of claim 5, further comprising: Determining orientations for each connected component using a covariance matrix of associated pixel coordinates, wherein the orientations indicate a direction of deformation associated with the deformed features of the first pattern or the deformed features of the second pattern. [7] A method according to any preceding claim, further comprising: Generating a vector map based at least on the pixel areas of the one or more features of the modified version of the first synthetic image and the second synthetic image and associated orientations of the one or more features of the first synthetic image and the second synthetic image. [8] A method according to any preceding claim, wherein the first pattern and the second pattern comprise a plurality of shapes arranged in a grid. [9] A method according to any preceding claim, wherein the first synthetic image and the second synthetic image are a same image comprising a same pattern. [10] A method according to any preceding claim, further comprising: Generating the first synthetic image and the second synthetic image. [11] A system comprising: a variety of image capture devices. an image stitching pipeline coupled to the plurality of image capture devices, the image stitching pipeline to: Processing a first synthetic image comprising a first pattern and a second synthetic image comprising a second pattern to generate a merged image comprising a first modified version of the first synthetic image and a modified version of the second synthetic image, wherein one or more features of the first pattern are deformed in the modified version of the first synthetic image and one or more features of the second pattern are deformed in the modified version of the second synthetic image; Determining the pixel areas for one or more features of the modified version of the first synthetic image and for the one or more features of the modified version of the second synthetic image; and Determining one or more feature densities for one or more regions of the merged image based at least on the determined pixel areas. [12] The system of claim 11, wherein to determine the one or more feature densities for one or more regions of the stitched image based at least on the determined pixel areas, the image stitching pipeline is to: Determining the one or more feature densities based at least on area ratios between the pixel areas of the features of the modified version of the first synthetic image and the second synthetic image and corresponding pixel areas of the one or more features of the first synthetic image and the second synthetic image, wherein the area ratios indicate magnification levels of respective features. [13] The system of claim 11 or claim 12, wherein the one or more feature densities are determined in one or more stages of an image stitching pipeline. [14] The system of any of claims 11-13, wherein the image stitching pipeline further serves to: responsive to a determination that one or more feature densities within a first region of the stitched image are below a first threshold density, adjusting a first parameter associated with a stitching algorithm that generates the stitched images; and responsive to a determination that one or more feature densities within a first region of the stitched image are below a second threshold density, adjusting a second parameter associated with the stitching algorithm. [15] The system of any of claims 11-14, wherein for determining the pixel areas for the one or more features of the modified version of the first synthetic image and for the one or more features of the modified version of the second synthetic image, the image stitching pipeline is to: performing a thresholding process on the merged image to obtain a binary image, wherein the one or more features of the modified version of the first synthetic image and the one or more features of the modified version of the second synthetic image are classified as foreground objects; Providing the binary image as input to a connected component algorithm, wherein the connected component algorithm provides, as output, a labeled image comprising a plurality of connected components, each connected component being assigned a unique identifier corresponding to a feature of the modified version of the first synthetic image or a feature of the modified version of the second synthetic image; and Determining pixel areas for each connected component using one or more shape analysis algorithms. [16] The system of claim 15, wherein the image stitching pipeline further serves to: Determining orientations for each connected component using a covariance matrix of associated pixel coordinates, wherein the orientations indicate a direction of deformation associated with the deformed features of the first pattern or the deformed features of the second pattern. [17] The system of any of claims 11-16, wherein the image stitching pipeline further serves to: Generating a vector map based on the pixel areas of the features of the modified version of the first synthetic image and the second synthetic image and associated orientations of the features of the first synthetic image and the second synthetic image. [18] The system of any of claims 11-17, wherein the first pattern and the second pattern comprise a plurality of shapes arranged in a grid. [19] A system according to any one of claims 11-18, wherein the system consists of at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for carrying out one or more digital twinning operations; a system for performing a light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system for presenting at least one of augmented reality content, virtual reality content, or mixed reality content; a system for running one or more real-time streaming applications; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more conversational AI operations; a system that implements one or more language models; a system that implements one or more large language models (LLMs); a system that implements one or more visual language models (VLMs); a system for performing one or more generative AI operations; a system for generating synthetic data; a system that includes one or more virtual machines (VMs); a system that is implemented at least partially in a data center; or a system implemented at least in part using cloud computing resources. [20] One or more processors comprising processing circuitry for: Processing a first synthetic image comprising a first pattern and a second synthetic image comprising a second pattern to generate a merged image comprising a first modified version of the first synthetic image and a modified version of the second synthetic image, wherein one or more features of the first pattern are deformed in the modified version of the first synthetic image and one or more features of the second pattern are deformed in the modified version of the second synthetic image; Determining the pixel areas for one or more features of the modified version of the first synthetic image and for the one or more features of the modified version of the second synthetic image; and Determining feature densities for one or more regions of the merged image based at least on the determined pixel areas.

Images