CREATING ALTERNATIVE IMAGE VIEWS FROM STEREO DISPARITY DATA

Abstract

The approaches presented here enable the generation of alternative views from disparity data acquired for one or more objects in a scene. This generation can be performed using an embedded processor with DMA memory access or other hardware with limited capacity. An intermediate representation can be created, depicting a 2D histogram view of the disparity data. This intermediate representation can then be transformed, using the embedded processor, into an alternative view image, such as a bird's-eye view. Morphological or similar filtering can be applied to the one or more objects in the intermediate representation using a filter of equal size, regardless of their distance from the camera plane used to acquire the disparity data.

Description

TECHNICAL AREA

[0001] This disclosure relates to the conversion of image data between different views or representations, in particular in one or more non-restrictive embodiments the generation of an intermediate image representation from a set of disparity data, which enables processing and conversion using resources with limited capacity. BACKGROUND

[0002] Various computational operations require determining the positions of different objects within a scene or geographic area. This can include, for example, analyzing captured image information to support tasks such as navigation, localization, controlled interaction, and collision avoidance for robots and autonomous or semi-autonomous vehicles or machines. Performing operations like image recognition and computer vision can demand significant resources, including access to memory with sufficient capacity to store an entire image. Tasks such as generating a bird's-eye view (BEV) of a scene from captured disparity data can be difficult or even impossible to accomplish with limited resources, such as embedded processors without access to external memory.Furthermore, there are tasks such as morphological filtering and motion analysis that are resource-intensive when performed on bird's-eye view images, where objects at different distances may have different levels of quality or amounts of recorded information. SUMMARY

[0003] The invention is defined by the claims. To illustrate the invention, aspects and embodiments that may or may not fall under the claims are described here.

[0004] The approaches presented here enable the generation of alternative views from disparity data acquired for one or more objects in a scene. This generation can be performed using an embedded processor with DMA memory access or other hardware with limited capacity. An intermediate representation can be created, which is a 2D histogram view of the disparity data. This intermediate representation can then be transformed by the embedded processor into an alternative view image, such as a bird's-eye view. Morphological or similar filtering can be performed on the one or more objects in the intermediate representation using a filter of equal size, regardless of their distance from the camera plane used to acquire the disparity data.

[0005] Further features of the disclosure are characterized by the independent and dependent claims.

[0006] Any feature in one aspect of the disclosure can be applied in any suitable combination to other aspects of the disclosure. In particular, methods can be applied to device or system aspects and vice versa.

[0007] Furthermore, features implemented in hardware can be implemented in software and vice versa. Any reference to software and hardware features in this description should be interpreted accordingly.

[0008] Each system or device feature described herein can also be provided as a procedure, and vice versa. Functionally described system and / or device aspects (including means-plus functional features) can alternatively be expressed in terms of their corresponding structure, for example, as a suitably programmed processor and associated memory.

[0009] It should also be noted that certain combinations of the various features described and defined in any aspect of the revelation can be implemented and / or provided and / or used independently of one another.

[0010] The disclosure also includes computer programs and computer program products containing software code adapted to perform any of the methods and / or embody any of the equipment and system features described herein when executed on a data processing device, including all or part of the individual steps of a method.

[0011] The disclosure also provides a computer or computer system (including networked or distributed systems) with an operating system that supports a computer program to perform any of the procedures described herein and / or to embody any of the device or system features described herein.

[0012] The revelation also provides a computer-readable medium on which one or more of the aforementioned computer programs are stored.

[0013] The disclosure also includes a signal containing one or more of the computer programs mentioned above.

[0014] The disclosure extends to methods and / or devices and / or systems as described herein with reference to the accompanying drawings.

[0015] Aspects and embodiments of the revelation will now be described purely by way of example with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Various embodiments according to the present disclosure are described with reference to the drawings, in which: Fig. 1A, Fig. 1B, Fig. 1C and Fig. 1D image views are shown that can be generated from captured image data according to at least one embodiment; Fig. 2A shows an intermediate image that can be generated using captured image data according to at least one embodiment; Fig. 2B shows views of similar objects both in a bird's-eye view (BEV) or a top-down image as well as in an intermediate histogram image according to at least one embodiment; Fig. Figure 3 shows the corresponding image data blocks in a disparity image and an intermediate histogram image, at least in one version; Fig. 4 shows corresponding image data blocks in an intermediate histogram image and a bird's-eye view image according to at least one embodiment; Fig. Figure 5 shows an example method that can be carried out to generate a bird's-eye view image from disparity image data using an embedded processor according to at least one embodiment; Fig. Figure 6 shows an example system with an embedded processor with direct memory access (DMA) functionality according to at least one embodiment; Fig. Figure 7A shows a comparison view of the level of detail captured for objects at different distances from a camera, according to at least one embodiment; Fig. Figure 7B shows different filter sizes required to process the same amount of detail information for objects at different distances in a bird's-eye view image according to at least one embodiment; Fig. Figure 8 shows a comparison of filter sizes that can be used to process the same amount of detail information for objects at different distances in a bird's-eye view image and an intermediate histogram image according to at least one embodiment; Fig. Figure 9 shows an example method that can be carried out using a single filter size for objects at different distances to perform morphological filtering with respect to an intermediate histogram image according to at least one embodiment; Fig. Figure 10 shows components of a distributed system that can be used to generate, process and provide sensor-based content, according to at least one embodiment; Fig. Figure 11 shows an example of a computer environment in which one or more devices operate to process data using a SoC, according to at least one embodiment; Fig. Figure 12 shows an example of a data center system according to at least one embodiment; Fig. Figure 13 shows a computer system according to at least one embodiment; Fig. Figure 14 shows a computer system according to at least one embodiment; Fig. 15 shows at least parts of a graphics processor according to one or more embodiments; Fig. 16 shows at least parts of a graphics processor according to one or more embodiments; Fig. Figure 17A shows an example of an autonomous vehicle according to at least one embodiment; Fig. Figure 17B shows an example of camera positions and fields of view for the autonomous vehicle. Fig. 17A according to at least one embodiment; Fig. 17C is a block diagram that provides an example of a system architecture for the autonomous vehicle. Fig. 17A according to at least one embodiment; and Fig. 17D is a diagram that represents a system for communication between one or more cloud-based servers and the autonomous vehicle. Fig. 17A according to at least one embodiment. DETAILED DESCRIPTION

[0017] The following description details various embodiments. Specific configurations and details are provided to facilitate a comprehensive understanding of these embodiments. However, it is also apparent to a person skilled in the art that the embodiments can be implemented without these specific details. Furthermore, known features may be omitted or simplified to ensure that nothing about the described embodiment remains obscured.

[0018] The systems and methods described here can be used without restriction on non-autonomous vehicles or machines, semi-autonomous or autonomous vehicles or machines (e.g., in one or more advanced driver assistance systems (ADAS), autonomous vehicles or machines, one or more in-vehicle infotainment systems, one or more emergency vehicle detection systems), manned and unmanned robots or robot platforms, autonomous mobile robots (AMRs), humanoid robots, warehouse vehicles, all-terrain vehicles, vehicles coupled with one or more trailers, aircraft, boats, shuttles, emergency vehicles, motorcycles, electric or motorcycles, airplanes, construction vehicles, trains, underwater vehicles, remotely controlled vehicles such as drones and / or other vehicle types.Furthermore, the systems and methods described here can be used for various purposes, for example, and without limitation, for machine control, machine locomotion, machine management, synthetic data generation, generative AI, model training or updating, perception, augmented reality, virtual reality, mixed reality, robotics, security and monitoring, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracking, etc.), collaborative content creation for 3D assets, generative AI, cloud computing, and / or other suitable applications.

[0019] The described elements can occur in various systems, such as automotive systems (e.g., an infotainment system for autonomous or semi-autonomous machines, a perception system for autonomous or semi-autonomous machines), systems with robots, flight systems, medical systems, boat systems, intelligent monitoring systems, systems for deep learning operations, systems for simulating operations, digital twin systems, systems implemented with an edge device, systems with one or more virtual machines (VMs), synthetic data generation systems, systems that run at least partially in a data center, systems for conversational AI operations, systems with one or more language models—such as large language models (LLMs), visual language models (VLMs), multimodal language models, etc.—and systems for performing generative AI operations (e.g.,(using one or more language models, transformer models, etc.), systems for performing light transport simulations, systems for performing collaborative content creation for 3D assets, systems that are implemented at least partially using cloud computing resources, and / or other types of systems.

[0020] Approaches according to various illustrative embodiments can enable the generation of alternative image views, such as bird's-eye views, of one or more objects, based on image data that contains (or can be used to determine) distance information, such as stereo disparity data. In at least one embodiment, such an approach can generate these or other such alternative views in memory-constrained environments and / or for low-power operation, for example, when direct access to external memory is not possible for an embedded processor with direct memory access (DMA), such as an NVIDIA Corporation programmable image accelerator (PVA). Partly due to DMA-related (and other) limitations, an intermediate representation of a scene can be generated, for example, using stereo disparity data.This intermediate representation can be used to generate a specific view, such as a bird's-eye view (BEV) or a top-down view of the scene, each of which is DMA-friendly and does not require the processor to access external memory, although the DMA can still access it. In at least one embodiment, stereo disparity data can be used to generate an intermediate representation in the form of a (quasi-bird's-eye view) 2D histogram or a histogram-like image that is a function of the camera angle θ and the distance from the camera plane z, and is represented as H(z, θ). This intermediate representation H(z, θ) can then be transformed into a bird's-eye view in Cartesian coordinates B(z, x). Such an approach makes it possible to generate an alternative image, such as...to generate a bird's-eye view image from stereo disparity data, for example by using an embedded processor with DMA capabilities.

[0021] Approaches according to various illustrative embodiments can also involve the processing (e.g., filtering) of data for operations such as those related to computer vision. When multiple objects at different distances from a (physical or virtual) camera are represented in an image, these objects are typically represented by different numbers of pixels and thus with different levels of quality. For example, an object closer to the camera would normally appear larger in the image and be represented with a greater number of pixels, while objects (at least of similar size) farther away from the camera would appear smaller and could be represented with fewer pixels, such as even a single pixel. An object far from the camera could therefore be represented in the image with very little detail.In alternative image views, such as bird's-eye views generated from stereo disparity data, objects at different distances can be treated the same, leading to varying quality results. Other approaches allow objects at different distances to be treated differently, which introduces additional complexity and cost, partly due to the need to account for distance differences. Limitations, such as those imposed by the use of DMA, can prevent efficient implementation of such treatment, if it is even possible at all. In at least one embodiment, an intermediate representation can be generated using stereo disparity data. This intermediate representation can then be used to create an alternative view (e.g.,to generate a bird's-eye view of a scene, where each of these steps is DMA-friendly (or can otherwise be performed using local memory) and does not require access to external memory. Such an intermediate representation can take the form of a (quasi-bird's-eye view) 2D histogram that is a function of the camera angle H(z, θ). This intermediate representation H(z, θ) can be used to perform various types of processing. Because the intermediate representation is a function of the camera angle, nearby objects appear larger (or are represented with a greater number of pixels) in the intermediate image and do not lose any information when they are scaled down in the final bird's-eye view.The same filter size can be used for all objects in this intermediate representation, instead of using smaller filters for objects closer to the camera (or larger filters for objects farther away), thus avoiding additional complexity due to distance from the camera. Using such an intermediate representation also has similar advantages for optical flow operations, where the distance from the camera plane then also does not need to be considered.

[0022] Variations of these and other such functions can also be used within the framework of the various embodiments, as should be clear to a person skilled in the art in view of the teachings and proposals contained herein.

[0023] Many computer processes involve determining the position of objects in a three-dimensional environment. This can include, for example, capturing image and / or sensor data of a physical environment and creating a digital reconstruction of that environment, which can be used for various purposes. Examples include determining the navigation of a robot by a data center or collision avoidance for an autonomous (or semi-autonomous) vehicle, which is partly based on the determined position and / or movement of objects in the general environment. The data to be analyzed can include, among other things, stereoscopic data captured by a pair of aligned cameras at a specific distance, or point cloud data acquired with a LiDAR system.Once acquired, this data can be used to generate one or more views of at least the portion of the environment represented by the data. In some cases, a first type of view can be acquired using one or more sensors, but at least a second type of view must be generated to accurately perform one or more operations. In some cases, it may be necessary to generate data that represents a different view and depicts nearby objects in a specific way that is advantageous or even required for the intended purpose.

[0024] Fig. Figure 1A shows an example image 100 that, according to at least one embodiment, could be captured by a 2D camera on a vehicle. As mentioned, this can be one of two images captured simultaneously by two coordinated cameras of a stereoscopic imaging system. In this example, the vehicle is traveling on a road, and the camera is positioned so that the camera view is in front of the vehicle. The camera can then capture image data representing objects that are at least partially in front of the vehicle and within the camera's field of view. These can include moving objects such as pedestrians or other vehicles 102, 104 that are at least partially in front of the (ego) vehicle, as well as stationary objects such as road signs 106, trees, buildings, sidewalks, and the like.An "ego" vehicle is generally a vehicle equipped with a series of sensors positioned around it that can collect sensor data, enabling the vehicle's control and / or operating system to perceive its surroundings. In at least one embodiment, such a vehicle may have cameras mounted around it that allow the capture of image data at a 360-degree angle, which can be used, for example, to generate a 360-degree top-down or "bird's-eye view" of the ego vehicle's physical environment.

[0025] For tasks such as collision avoidance and route determination, it can be important to accurately determine the relative positions of various objects near a self-driving vehicle (or other controllable system, device, or component) to ensure that the vehicle does not collide with or interact with these objects. As explained earlier, an image pair, such as the one in Fig. Figure 1A shows image 100, which was taken with a pair of matched cameras (e.g., cameras with similar camera and imaging parameters aligned to a common focal length and a small lateral distance) and used to generate a disparity image 130, as shown in Fig. Figure 1B illustrates this. Due to the lateral distance between the two aligned cameras, each object appears in slightly different positions in the images captured by these cameras, based on a slightly different viewing angle used for the shot. Objects closer to the camera have a greater difference in apparent position than objects farther away. Knowing the camera parameters, one can calculate the distance between the positions (or "disparity") of an object in each of the captured images, often referred to as the "left" and "right" images due to the lateral distance. This can be done for individual pixel positions in the left and right images, calculating the distance for each of these pixel positions.This calculated distance to the object represented at each pixel position can then be used to generate a disparity map or disparity image 130, as in . Fig. Figure 1B illustrates this. In this example disparity image, objects closer to the camera appear brighter, approaching white, while objects farther away appear darker. For parts of the image (e.g., the sky) where no discernible objects are present, it can be assumed that objects are either at infinity or beyond a measurable distance, and these parts can be represented with black. As shown, it can be determined that one of the vehicles 102 is closer to the camera than another vehicle 104, based on the closer vehicle 102 having a brighter color and representing a shorter distance from the camera. The disparity data for objects in the disparity image shows only the visible portion of the objects in the image, in this case, the rear and right side of the vehicle.If complete shape information is needed, a different process can be used that attempts to identify the object type and then derives additional shape information based on that object type, such as a specific make, model, and year of manufacture. However, for tasks like collision avoidance, it may be sufficient to know the visible part of the vehicle facing the user, as this is the part most likely to be affected by a collision.

[0026] In at least one embodiment, the acquired image and / or disparity data can be analyzed to attempt to identify specific objects within the data. Object identification may involve identifying pixels that are determined to belong to a single object based on factors such as similarity in position and color, and then possibly identifying the object type. For example, if the disparity image 130 consists of Fig. If, using an approach with connected components, the system were analyzed to identify objects within a certain distance from the camera, the approach could identify three groups of pixels that are likely to correspond to specific objects. In some embodiments, certain types of objects, such as roads and sidewalks, can be excluded from consideration. The identified objects can then be treated as separate objects, as shown in schematic view 160 in Figure 1B. Fig. Figure 1C illustrates this. In this example, the two nearest vehicles, 102 and 104, and a traffic sign are identified as objects that meet the current identification criteria. It should be noted that the number of objects detected in this example is relatively small for illustrative purposes, and that an actual object detection or identification process would detect many other objects in disparity image 130. Fig. 1B can identify. In some embodiments, a "connected components" type approach can be used to identify groups of pixels that are likely to correspond to individual objects, and then an object recognition approach can be used to attempt to identify the object type, for example, to distinguish vehicles 102, 104 from road sign 106. Such a distinction can be important for tasks such as object avoidance, since road sign 106 is fixed in place and will not move in a relevant upcoming period, whereas vehicles may be in motion or may move within that upcoming period, and this movement should be taken into account when determining appropriate navigation routes or other such options. This example image in Fig. Figure 1C shows a schematic view of a typical camera view from a vehicle, where the image has an image coordinate system originating from an upper left origin and expressing pixel coordinates as (i, j), where i corresponds to a vertical axis in the figure and j to a horizontal axis. The depicted objects can be considered to be located in camera space, which may correspond to a coordinate system (φ, θ, z) or a similar coordinate system.

[0027] For at least some applications, it may be desirable to generate a top-down or bird's-eye view (BEV), or at least to compute such view data for the detected and / or identified objects in the scene. When using data from a single camera (or a stereoscopic camera array), the available data is limited to the objects within the camera's field of view, unless data from multiple cameras can be combined or otherwise processed to create a larger view. As an example, [reference to relevant example would be inserted here]. Fig. 1D a bird's-eye view image 180, based on the camera image of Fig. 1A. In this bird's-eye view image, objects 102, 104, and 106 are represented with accurate information about their relative size and position. As shown, only a portion of each object is visible to the camera, for example, one or more sides facing the camera. Therefore, the representation of each object includes only the portion of each object visible to the camera, unless additional processing is performed to identify an object type and then fill the space for that object. As shown, such a bird's-eye view image can lie in a conventional Cartesian coordinate space and be represented as B(x, y). In such a coordinate system, the position of the origin can be important and typically corresponds to the center point of a lens (or sensor, etc.) of a camera that captures the image data for the scene.The coordinate system for a bird's-eye view can be angular with respect to the origin point, such as a coordinate system (φ, θ) that represents the inclination and rotation from the origin point.

[0028] In conventional approaches, a 180° bird's-eye view image can be generated from a disparity image using one or more processors, such as a central processing unit (CPU) or a graphics processing unit (GPU). This process can also be performed using hardware acceleration (e.g., using a programmable image accelerator (PVA), a deep learning accelerator (DLA), an optical flow accelerator (OFA), etc.), which allows for very fast bird's-eye view generation, as may be required for real-time operations such as autonomous or semi-autonomous navigation. However, to perform this processing, the processor must have access to sufficient memory, such as external storage, capable of holding all the image data at once. For high-resolution disparity images, this can be more data than can fit into hardware with limited capacity, such as...Memory that an embedded processor can access via DMA or another mechanism.

[0029] With typical CPU-based systems, it's possible to create a bird's-eye view image from disparity data without much concern about data transfer. However, it's possible that a processing unit might not have direct access to the external storage where all the image data is stored. A system used to process captured image and / or sensor data may be limited in terms of available memory or processing capacity. For example, a system might use an embedded processor equipped with DMA that lacks direct access to external memory. Such a device might not have the necessary memory to convert a stereo disparity image of a scene into an alternative image type, such as a bird's-eye view of the scene.While such a task may be relatively simple on a device with a CPU or GPU, it can become a non-trivial task when it has to be performed on a processing unit, such as an embedded processor, without direct access to external memory.

[0030] In at least one embodiment, a disparity image of a scene can be converted into an alternative view image, such as a bird's-eye view, in a two-step process. Both of these steps can be relatively lightweight, allowing them to be performed on DMA-based hardware. For example, data transfers for each of these steps can be performed using DMA for rectangular areas of the input from a disparity image, as well as for data from the intermediate and generated bird's-eye view images. It is understood that additional steps may also be included for preprocessing, postprocessing, data conversion, and other such functions within at least one embodiment.

[0031] In one example, the value of the disparity at a specific location (i, j) of a disparity image can be represented as D(i, j). The indices i and j are functions of the vertical angle φ (representing the tilt) and the horizontal angle θ (representing the rotation). In the simplest case of corrected images, with φ and θ measured from the camera axis, while i and j are the usual image coordinates with the origin at the top right, as can be given by the following: i=aφ+b j=cθ+d where the parameters a, b, c, and d depend at least partially on the intrinsic properties of the camera. The disparity can therefore be represented as D(φ, θ) instead of D(i, j). A relationship between the stereo disparity value D and the distance of an object from the camera image plane z can be given as follows: z=E / D where E is a parameter that depends on the intrinsic properties of the camera.

[0032] In at least one version, the first step can be to take an intermediate image, such as a two-dimensional (2D) histogram or intermediate image 200, as in Fig. Figure 2A shows that the camera is not positioned at a single point in the center, as in a bird's-eye view image, but rather distributed across the entire lower portion of the histogram. The histogram values ​​can be represented by H(z, θ), and an element H(z, θ) can be incremented for each pixel D(φ, θ) such that z = E / D plus or minus the resolution error. Such a transformation can produce an intermediate image 200 that resembles a top-down or bird's-eye view 180 of the scene as depicted in the input image. However, this intermediate image 200 will likely exhibit at least some inaccuracies in the representation of the objects in the scene.These inaccuracies are partly due to the fact that objects closer to a (physical or virtual) camera tend to be displayed as splayed or stretched laterally, since objects closer to the camera tend to occupy a wider angular range. Fig. Figure 2B shows an example of such a lateral stretching effect based on a distance of 260 from a camera. In the bird's-eye view, 250 from Fig. Figure 2B shows two objects 252 and 254 that have the same size and shape. In particular, both objects in this example have the same width. However, when analyzed in histogram space based on the angle, object 254, which is closer to the camera, occupies a larger angular range 258 than the angular range 252 occupied by the more distant object 252. In the example shown, the angular spread 258 for the closer object 254 is two to three times larger than the angular spread 256 for object 252, which is farther from the camera 260. When generating an intermediate histogram image 280, which is a function of the angle, as in Fig. As shown in Figure 2B, the height z for the two objects is not stretched, but the width of the closer object 254 is stretched in the θ direction by an amount that makes the width of the closer object 254 appear two to three times larger than the width of the object 252, which is farther from the virtual camera. This corresponds to the difference in the angular range that these objects occupy based on their distance, even though objects 252 and 254 actually have the same width. If one considers the bird's-eye view image 180 from Fig. 1D and the histogram intermediate image 200 from Fig. If one looks at 2A again, this stretching can also influence the appearance of the object, since the stretching of an object 102 also leads to it appearing to have a different (e.g. distorted) shape in the intermediate histogram image 200 than in the bird's-eye view image 180.

[0033] A second step in this example process can be performed to at least correct this stretching effect. An example is used that is described in Fig. Figure 3 shows that a disparity image 300 is used to generate an intermediate image 310 containing the same example objects 102, 104, 106 that were previously used as examples to simplify the explanation. An advantage of generating and using an intermediate image 310 in an H(z, θ) coordinate system is that all data in a rectangular region 302 (φ1: φ2, θ1: θ2) of D(φ, θ) can be converted to increments of H(z, θ) in a rectangular region 304 (0:z). max, θ1: θ2), which allows the use of DMA. In the disparity image, the pixels within the rectangular areas have values ​​that depend on the distance to these objects, which can lie anywhere between the camera lens (for a distance of zero) and effectively infinity, or a maximum detectable (or maximum allowable) distance, which here is denoted as z. max is fixed. Therefore, when this rectangle is transformed into the quasi-bird's-eye view intermediate image 310, the portion of the corresponding data extends from the position of the camera lens (represented by the bottom edge of intermediate image 310) to the maximum distance from the camera (represented by the top edge of the image). It should be clear that there may be other rectangles above and / or below the rectangular example area 302 in the disparity image 300, which are also mapped onto the same rectangular area 304 of intermediate image 310.

[0034] An example process according to at least one embodiment can convert an intermediate image 400 H(z, θ) into a bird's-eye view image 410 B(z, x), as shown in Fig. 4 is shown, and processing continues on B. Alternatively, an example process can perform the processing on H, create a list (or set, etc.) of object centers (or other representative locations) and other statistics in H, and then transform this list into a corresponding list for a bird's-eye view image. In both approaches, the coordinate transformation can be specified as follows: x=z tan θ

[0035] Such a task can also be DMA-friendly, since, for example, the information from any point in row x of the intermediate image H(z, θ) only affects row x of the final bird's-eye view image B(z, x). As in Fig. As shown in Figure 4, a rectangular area 402 of the intermediate histogram image 400 has the same height as the corresponding rectangle 404 for the same portions of the depicted objects in the bird's-eye view image. This is because the distances from the camera to the objects are the same in both images, with the stretching occurring laterally (or horizontally from left to right in the figure). However, due to this stretching, the rectangular area 402 in the intermediate image 400 is typically wider than the corresponding rectangle 404 in the bird's-eye view image 410, the extent of which may be based partly on the distance from the camera, since objects closer to the camera are depicted with greater lateral stretching in the intermediate image.

[0036] Due to the discrete nature of digital images and the contraction effect of multiplication by tan(θ), it is possible for two or more elements of H to be mapped to the same element of B. In such cases, one can add the values ​​from H or take their maximum value to solve the problem. This might not be a problem when transforming lists, as long as the approach allows different objects to occupy the same position in the bird's-eye view. As mentioned, such an approach is advantageous because the steps or tasks can be performed with limited resources, such as embedded processors with DMA.

[0037] One advantage of using such a block-based approach is that the image data can be stored within a rectangle, such as the one in Fig. The four rectangles 402 and 404 shown can be selected so that the data within a rectangle can be transferred and stored using a technology with limited capacity, such as DMA. A specific rectangular area 402 from an image (such as the intermediate image 400) can be selected, the data can be transferred via DMA, processed by an embedded processor, and then stored relative to the corresponding rectangular area 404 or block in the generated image, such as a bird's-eye view image 410. The results can be transferred via DMA to the resulting memory location for this output representation. The size of the rectangular areas can be selected based on various factors, such as the size of the images to be processed and the amount of available memory transfer, among other options.For example, the memory can hold approximately 32 KB of data for input and output, allowing an upper limit to be chosen for the rectangular area size. This ensures that the amount of data within this area remains within a portion of the available 32 KB, taking into account factors such as image resolution. The number of rectangular areas used can then be calculated based on the total amount of data and the amount of data that can be contained within the different rectangular areas. In at least one embodiment, the rectangular areas can all be the same size, although in other embodiments, the size or shape of the rectangular areas can vary if this offers a performance advantage, as long as the size and shape remain within the permissible parameters.In at least one embodiment, an intermediate image can be processed using 10 rectangular areas horizontally and 10 rectangular areas vertically, resulting in a total of 100 rectangular areas or tiles. The data blocks can be processed as tiles of an image, with each pixel falling into a specific tile, and the tiles can be processed separately without affecting the quality of the resulting output. Processes such as sharpening or low-pass filtering can be performed on individual tiles, which is partly due to the highly localized nature of this type of processing.The advantages of rectangular correspondences between an intermediate image and a bird's-eye view image, particularly regarding where the data can go and how the data relates between the two images, allow the implementation of a relatively complicated transformation from a camera image to a bird's-eye view image using hardware with limited capacity, such as an embedded processor with DMA or another dedicated processing unit or core with limited memory and / or transmission capacity.

[0038] Fig. Figure 5 shows an example of a computational process 500 that can be performed to generate an alternative image, such as a bird's-eye view image, from a camera image according to at least one embodiment. It is understood that for this and other processes described herein, additional, fewer, or alternative steps can be performed in a similar or alternative sequence, or at least partially in parallel, within the scope of the various embodiments, unless expressly stated otherwise. Although this example is explained in the context of a camera image and a bird's-eye view image, other types of image transformation can also be performed using such a process within the scope of various embodiments. Such a computational process can be performed using any combination of hardware, firmware, and / or software.For example, various functions can be executed using one or more processors that execute instructions stored in one or more memories. Such a process can also be implemented as machine-readable instructions stored on a computer storage medium. This process can be provided by a standalone application, a service, or a hosted service (alone or together with another hosted service), as a microservice via an application programming interface (API), or as a plug-in for another product, to name just a few examples. Furthermore, this process is exemplified in relation to the system of [system name]. Fig. 6 described. However, such a process can additionally or alternatively be performed by any system or any combination of systems, including but not limited to those described here.

[0039] In this example computation process 500, disparity image data 502 is acquired, containing representations of one or more objects in a scene. This may involve, for example, receiving disparity image data from a stereoscopic camera array (or device) positioned so that the one or more objects fall within the camera array's field of view. There may also be other and / or additional types of data acquired by one or more sensors, the additional types of data being visual, shape, motion, or other such data relating to the objects, the data being able to have specific values ​​or values ​​relative to those of the camera unit or the vehicle / system to which the camera unit is attached.In this example, the hardware allocated to process the disparity image data can include limited-capacity hardware, such as an embedded processor with DMA functionality. This embedded processor can be used to generate a two-dimensional histogram containing representations of the one or more objects as a function of the angle from the camera unit used to capture the disparity data. This histogram, or intermediate image, can be a function of both distance and angle from the camera unit and serve as a kind of bird's-eye view. The embedded processor can also be used to generate a list of centers and statistics (or other such location indicators and / or metrics) for the one or more objects in the 2D histogram and can transform the values ​​in this list so that they can be represented in a Cartesian coordinate system.A bird's-eye view image of the scene can be generated, containing top-down representations of one or more objects transformed using a list of coordinates and statistics. Image data can be transferred and analyzed using blocks, tiles, or rectangular pixel areas from the intermediate image. This allows the embedded processor to transfer, process, and store the image without needing to access all image data at any given time, which might be stored in external memory. The bird's-eye view image can then be used to perform at least one task related to the scene and / or the one or more objects. This could include, for example, determining a navigation path or interaction sequence with respect to the object(s) in the scene or its immediate surroundings.In other embodiments, the data can be stored for later use and analysis or made available for use in performing other types of tasks.

[0040] Fig. Figure 6 shows an example system 600 in which an embedded processor 614 can be used to perform tasks such as image transformation according to at least one embodiment. In this example, a computer system 602 has (or can at least communicate with) a stereo camera 604 that can acquire stereoscopic image data of one or more objects 608 within a field of view 606 (or at least an overlapping field of view) of the stereo camera assembly. As mentioned earlier, various other types of sensors or devices can also be used to acquire information about the objects in the various embodiments. In this example, the acquired image data can be stored in local memory 616, external memory, or another storage location. The local memory can be provided by a central processing unit (CPU) 618 or another processor (e.g., a CPU, a microcontroller, or a microcontroller).a GPU or DPU), for example, via a system bus 622, which allows the CPU to process the image data, all of which can be accessed from local memory 616. In this example, however, the computer system 602 can have an image processing module 610 or at least cooperate with one. The image processing module can have an embedded processor 614, which may not have access to the local memory 616 (which is outside the image processing module) and may only be able to access portions of the image data via a DMA controller 612 or some other data transfer mechanism. As explained herein, blocks of image data can be transferred via DMA for processing by the embedded processor 614.For image k, an intermediate image can be generated by the embedded processor 614 receiving and transforming blocks of stereoscopic image data (or disparity image data) from the stereoscopic camera 604. The embedded processor can then transform the intermediate image into an image with an alternative view, for example, a bird's-eye view, using a block-based approach that processes portions of the image data separately. Tasks such as the analysis of connected components and the calculation of the center of gravity, which are used for the transformation, can be performed by the embedded processor.The bird's-eye view image can then be transmitted directly or via the CPU 618 or the system bus 622 to a control system 620 or another such destination or receiver for use in one or more tasks such as autonomous navigation, collision avoidance, object interaction, and other such options. In this example, the image processing module 610 can be a system-on-chip (SoC) that can be used by or incorporate at least part of the camera circuitry. The embedded processor 614 can be used as a coprocessor or offload processor for the CPU 618 or another processor, such as a digital signal processor (DSP). The use of DMA in systems with DSPs allows for tight control over data movement while avoiding significant data caching, memory address space management, and other such tasks.

[0041] As mentioned above, one of the challenges in computer vision and image understanding is that objects located at a greater distance from a (physical or virtual) camera inevitably appear smaller in the images captured or generated by that camera. Consider the captured camera image 700 in Fig. Figure 7A, which can represent a pair of stereo images. In this image, there are two objects, 702 and 704, that are approximately the same size. As shown, object 702, which is farther from a virtual camera, appears smaller and is represented by a relatively small number (e.g., 9) or arrangement of pixels 706. Object 704, which is closer to the camera, has a larger representation in the image, represented here by a larger number (e.g., 56) of pixels. The smaller number of pixels associated with objects farther from the camera can result in lower quality representation of these images, including less information about the shape, appearance, and other properties of these distant objects compared to objects closer to the camera.Such an effect can be observed particularly when the scene is transformed from a camera perspective and processed in a bird's-eye view, such as image 750 in . Fig. 7B, particularly considering an occupancy grid or occupancy map. In this example, there are two objects 752 and 754 of similar size, but located at different distances from the position of camera 758. These objects at different distances could be processed using the same analysis functions, but such an approach may be suboptimal, partly due to the different information content of these objects. An alternative approach would be to use different analysis functions for objects at different distances, but this increases complexity and cost, as different distances must be determined and taken into account, including the different processing of the objects based on their respective distances from the camera.As mentioned earlier, in certain situations it may be desirable to process such images on hardware with limited capacity, and such additional processing and complexity may prove problematic for this hardware, at least when various performance criteria must be met.

[0042] The approaches described above enable a two-stage conversion of disparity images into images with alternative views, such as bird's-eye views of a scene, which can be implemented using one or more resources with limited capacity, such as an embedded processor with DMA functionality. Such an approach can generate an intermediate representation H(z, θ) of the bird's-eye view, which, due to the lateral stretching effect described earlier, inherently has a larger number of occupancy cells covering objects closer to the camera. This contrasts with a generated bird's-eye view image 750, as described in Fig. 7B is shown. In this example image, the objects are shown at the same size. However, the amount of information available for each object is limited, as shown in Fig. 7A illustrates this, which is partly due to the different number of pixels based on the distance from the camera. In an isometric bird's-eye view, an object occupies the same number of grid cells regardless of its position or distance from the camera. In the example of Fig. 7A represents object 704, which is closer to the camera, but has approximately six times more information available because the captured image of these objects contained a larger number of pixels. From a bird's-eye view of Fig. 7B means that representing objects at the same size is achieved either by stretching the pixels for object 752, which is farther from the camera, or by compressing the pixels for object 754, which is closer to the camera. Pixels from the disparity view are accumulated in the bird's-eye view, and various types of image processing can be performed to combine these pixels into a single object of the appropriate size.

[0043] In such a bird's-eye view image 750, using filters of the same size results in different amounts of information being processed for different objects at different distances, which, as explained earlier, can affect the quality. As mentioned, one way to ensure that similar amounts of information are used for each filter (or algorithm, etc.) is to use filters of different sizes for objects at different distances. One approach would be to use filters that result in each filter capturing the same number of pixels or data from the captured image. As shown, this can result in a filter 756 of a first size for the object at a greater distance from the camera 758.This filter can capture approximately 9 pixels of information for this distant object (which may include some pixels in an area of ​​the object that do not correspond to the object itself). If a similar filter is to be used for the closer object 754, capturing approximately 9 pixels of information for that object, this filter would need to be smaller. With such an approach, filters of different sizes would be required for each different distance, or at least for distance ranges, which may be partly based on the resolution of the originally captured image data. As mentioned earlier, this need to determine and use multiple filter sizes (or different algorithms, etc.) can lead to additional processing and memory requirements that are difficult to meet, for example, with hardware of limited capacity.

[0044] To avoid using different filter sizes or algorithms for objects at varying distances from a camera, approaches according to various embodiments can employ an intermediate image, as previously described herein, which allows the use of filters of the same size for different objects regardless of their distance from the camera. As mentioned earlier, such an intermediate image avoids the problem of objects closer to the camera being covered by a different number of occupancy cells. In such an intermediate representation of the occupancy grid, nearby objects appear larger than objects farther from the camera due to the lateral stretching effect. The degree of stretching is inversely proportional to the distance from the camera (although in other embodiments there might instead be compression directly proportional to the distance).Since the degree of stretching is inversely proportional to the distance, a filter of the same size can be used for all objects in an intermediate image, regardless of their distance from the camera. In intermediate image 850 of . Fig. Figure 8 shows, for example, two filters 852 and 854, intended for use with the respective objects 702 and 704. Partly due to the stretching of the closer object 704 in the intermediate image 850, each filter 852 and 854 processes an area that corresponds to approximately the same number of pixels as the originally captured image. Such an approach can also help prevent the loss of information for objects closer to the camera, which might otherwise occur if these objects were reduced in size or compressed in the final raster layout.

[0045] In at least one version, using a single filter of a given size for all objects in an intermediate image is the same as using smaller filter sizes (or finer filters) for objects closer to the camera and / or larger filter sizes for more distant objects in the final occupancy grid. Using a single-size filter can also be achieved without the added complexity of accounting for distance from the camera. Further advantages arise when an optical flow map is used to estimate object motion. If this motion is calculated by averaging the optical flow over the object in the camera image, it can be used directly in (z, θ) space without needing to consider the distance from the camera plane (z).The processing of the occupancy grid information can therefore be carried out in such a (z, θ)-space representation of the occupancy grid and not in the occupancy grid area itself.

[0046] In one example of the type of analysis that can be performed with such filters or algorithms, pixel data is analyzed to identify objects in the image and determine which pixels correspond to specific objects. While such a process might be relatively simple for a person viewing the image, it can be quite complex and / or time-consuming and resource-intensive to perform such determinations in software. For example, an input image might need to be analyzed using a connected component algorithm (or a similar approach) to connect related pixels belonging to a single object. Individual pixels can then be flagged as belonging to specific objects or otherwise marked.Using unnecessarily large filters can make it difficult to distinguish between adjacent objects and lead to these objects being incorrectly identified as a single, larger object. Likewise, using filters that are too small can lead to an object being incorrectly identified as two or more smaller objects.

[0047] To enable accurate pixel grouping, some additional preprocessing may be necessary, such as dilation and / or erosion operations to denoise the data (since input image data in various systems can contain an unacceptable amount of noise in many cases) by partially altering the size or shape of one or more objects in an image. Erosion typically removes pixels from the boundaries of objects to decrease the overall size of object representations in the image data and to remove edge pixels whose value can be significantly affected by areas unrelated to the object (e.g., the background). Dilation can be used to add pixels near the boundaries of objects, thereby increasing the size of object representations.This can also help to connect broken or separated parts of an object in an image, which can be helpful for performing connected component analysis and other types of analysis. Erosion can be used to remove noise, but it results in smaller object representations, so dilation can be used to restore the lost object area.

[0048] Pixels can undergo some morphological filtering before being processed using a connected component (or similar) approach. Image data for objects that appear smaller due to their distance from the camera tends to be noisier than image data for closer objects because fewer pixels represent the appearance (and other such properties) of the more distant objects. This can lead to a lack of fine detail as well as inaccuracies in pixel values, since an attempt must be made to select a pixel value for a specific location based partly on potentially many different colors around that location. The final pixel value for that location in the image may differ significantly from the actual color at that location. In such a case, it may be better to use a larger filter on the noisier objects.Using a larger filter for closer and less noisy images can lead to the loss of accurate or high-quality image data that was captured for these larger and less noisy object representations.

[0049] As mentioned earlier, it can be advantageous to perform such morphological (and other types of) filtering using an intermediate representation image, since a single filter size and type can be used for all areas of the image, regardless of the distance of any given object from the camera. Using a filter of the same size everywhere also allows for the use of simpler algorithms and reduces processing and data transmission overhead. Fig. Figure 8 again shows a bird's-eye view (800) of a pair of objects in a scene, where filters of different sizes would have to be used to process the same amount of actually captured image data (or similar data). In comparison, in an intermediate image (850), for the same set of objects, the objects closer to the camera are stretched by an amount inversely proportional to their distance from the camera, so that filters of the same size can be used for each area, containing information for the same number of pixels in the originally captured image data. Comparing the portions of each object represented by filters in the bird's-eye view (800) with those in the intermediate image, one can see that the filter represents essentially the same portion of each object, e.g.,A very similar portion of the closer object 704 is represented in the smaller filter 708 in the bird's-eye view 800 and in the equally sized filter 854 in the intermediate view 850. In this way, the same amount of pixel data is processed for each filter, eliminating the need for different filters for objects at varying distances from the camera. Similar advantages can be achieved in motion analysis or estimation. The size and type of filter can vary depending on the application, and in some cases, different filters can be used to determine the preferred visual quality, which may be subjective and vary depending on the application or intended use.For example, in navigation applications it may be more important to display the shape correctly than to display the objects as accurately as possible, whereas in presentation-based applications a high quality of certain visual aspects may be more important, while the exact shape or position of a particular object may not be crucial.

[0050] Fig. Figure 9 shows an example of a computational process 900 that can be performed to carry out consistent and efficient filtering during image transformation, according to at least one embodiment. In this example, disparity image data 902 are obtained, containing representations of one or more images in a scene, as described in the example process of Fig. As explained in section 5, in this example, the hardware allocated to process the disparity image data may include limited-capacity hardware, such as an embedded processor with DMA functionality. This embedded processor can be used to generate a two-dimensional histogram containing representations of one or more objects as a function of the angle from the camera array used to capture the disparity data. This histogram, or intermediate image, may be a function of both distance and angle from the camera unit and can serve as a kind of bird's-eye view. The intention in this example is to perform morphological filtering to attempt to reduce noise and otherwise improve the quality of the image data to be transformed.A filter of a specific size and shape can be selected 906 for use in morphological filtering and / or other such processing or preprocessing. Morphological filtering can then be performed 908, applying the same specific filter size and shape to all locations in the input image, including each of the one or more objects, regardless of the position or distance of those objects from the camera that captured the disparity image data. The filtering may include, for example, erosion to remove noise and dilation to attempt to recover data lost during erosion. In this example, an analysis of the associated components of the filtered image data can be performed to identify pixels associated with individual or specific objects of the one or more objects.Such an approach effectively identifies which pixels in the image data are (at least largely) associated with each object. This approach can be advantageous when performing an image transformation based on aspects such as a list of object centers, where determining the exact size and shape of the objects may be crucial for accurate center determination. An alternative image, such as a bird's-eye view, can be generated for the scene, containing top-down representations of one or more objects based on a transformation from the two-dimensional histogram. The bird's-eye view image can then be used to perform at least one task related to the scene and / or the one or more objects.This can include, for example, determining a navigation path or interaction sequence with respect to the object or objects in the scene or its immediate surroundings. In other embodiments, the data can be stored for later use and analysis or made available for use in performing other types of tasks.

[0051] Aspects of various approaches presented here can be so lightweight that they can be executed in real time at different locations, for example, on a device such as a client device comprising a personal computer or a game console. Such processing can be performed on or for content generated on or received by that device, or received from an external source, such as streaming data or other content received over at least one network from a cloud server 1020 or a third-party service 1060, among other such options, as described in Fig. 10. In some cases, at least part of the processing, generation, composition and / or determination of this content may be carried out by one of these other devices, systems or units and then made available to the client device (or another such recipient) for display or other such use.

[0052] As an example, Fig. 10 An example network configuration 1000 that can be used to provide, generate, modify, encode, process, and / or transmit data, requests, or other such content. In at least one embodiment, a client device 1002 can generate or receive data for a session using components of a content application 1004 on the client device 1002 and data stored locally on that client device. In at least one embodiment, a content application 1024 running on a server 1020 (e.g., a cloud server or edge server) can initiate a session connected to at least one client device 1002, using a session manager and user data stored in a user database 1036, and capturing or retrieving content such as one or more images or image data, e.g., from an asset repository 1034, as specified by a content manager 1026.A content manager 1026 can work with one or more transformation modules 1028 to transform between image views, for example, from a disparity image to a bird's-eye view image. A content application 1026 can also work with a sensor control module 1030, which can initiate the acquisition and / or preprocessing of sensor data, and with a control module 1032, which can perform various operations based on the sensor data transformed by the transformation module 1028. Transformed image data can also be provided for processing or display via the client device 1002. In this example, the content application 1024 can receive disparity data acquired by the client device 1002 and return an alternative view image transformed by the transformation module 1028.In at least one embodiment, the content application 1024 can cooperate with one or more encoders, transcoders, and / or compressors that can perform tasks such as encoding, decoding, compressing, and / or decompressing an instance of content, such as image data, before or after transformation. Different compression or encoding methods may be advantageous for different operations, such as storage versus processing. At least a portion of the generated, captured, transformed, and / or compressed content can be transmitted to the client device 1002 using a suitable transmission manager 1022 for delivery via download, streaming, or another such transmission channel. An encoder can be used to encode and / or compress at least a portion of this data before transmission to the client device 1002.In at least one embodiment, the client device 1002, which receives such content, can make this content available to a corresponding content application 1004, which may also or alternatively include a graphical user interface 1010, an image control module 1012, and a transformation module 1014. This serves to provide, synthesize, reproduce, compose, modify, transform, or use image- or sensor-based content for display (or other purposes) on or by the client device 1002. A decoder can also be used to decode data received via the network(s) 1040 for display on the client device 1002, e.g., image or video content via a display 1006 and audio content, such as sounds and music, via at least one audio playback device 1008, such as loudspeakers or headphones.In at least one embodiment, at least some of this content can already be stored on, played back on, or accessible to the client device 1002, so that no transmission over the network 1040 is required for at least this part of the content, for example, if this content has been previously downloaded or stored locally on a hard disk or optical disc. In at least one embodiment, a transmission mechanism such as data streaming can be used to transfer this content from the server 1020 or the user database 1036 to the client device 1002. In at least one embodiment, at least some of this content can be obtained, enhanced, and / or streamed from another source, such as a third-party service 1060 or another client device 1050, which may also include a content application 1062 for generating, enhancing, or providing content.In at least one embodiment, parts of this functionality can be implemented using multiple devices or multiple processors within one or more devices, which may, for example, include a combination of CPUs and GPUs.

[0053] Fig. Figure 11 shows components of an example system or operating environment in which image transformation can be performed according to at least one embodiment. Such an environment 1100 may include a processor 1102, a memory 1104, an instruction switch 1106, a memory 1108 (sometimes referred to as dynamic random-access memory or DRAM), and functional blocks 1110a, 1110b (individually referred to as functional block 1110 and together as functional blocks 1110 unless otherwise specified). In some embodiments, the processor 1102, the memory 1104, the instruction switch 1106, the memory 1108, and the functional blocks 1110 may be interconnected via wired and / or wireless links (e.g., to establish a communication link and / or the like). In some embodiments, the components of the environment 1100 may be contained in a system-on-a-chip (SoC).For example, the components of the Environment 1100 can be contained in one or more SoCs, which form integrated circuits by combining some or all of the components of the Environment 1100.

[0054] The Processor 1102 can have one or more processors, such as one or more central processing units (CPUs), graphics processing units (GPUs), microprocessors, microcontrollers, and / or similar devices. The Processor 1102 can be connected to an instruction cache (not explicitly shown) that stores instructions for the Processor 1102 to execute. In some embodiments, the Processor 1102 can be configured to output data related to the configuration and / or control of one or more of the devices from Fig. 11 are connected. For example, the processor 1102 can be configured to output data connected to the configuration of a Direct Memory Access (DMA) Hardware Sequencer 1114a and / or a DMA Hardware Sequencer 1114b to control DMA transfers to and / or from a Vector Memory Electronic Memory (VMEM) 1112a and / or VMEM 1112b of Function Block 1110a or Function Block 1110b, respectively.

[0055] Memory 1104 (sometimes referred to as the L2 buffer or L2 cache) may include a device connected to the DMA hardware sequencer 1114a and / or the DMA hardware sequencer 1114b of functional blocks 1110. In some embodiments, memory 1104 may be configured to receive and store data from the DMA hardware sequencer 1114a and / or the DMA hardware sequencer 1114b of functional blocks 1110, as described herein. In some embodiments, memory 1104 may have one or more (e.g., 2) banks that allow simultaneous read or write requests. For example, memory 1104 can have a first bank assigned to DMA hardware sequencer 1114a and a second bank assigned to DMA hardware sequencer 1114b.

[0056] Command switch 1106 can include one or more processors configured to scan memory 1108, receive data from memory 1108, cause data stored in memory 1108 and / or in the local memory of command switch 1106 to be loaded into VMEM 1112, and / or the like. For example, command switch 1106 can be coupled to memory 1108 and / or include internal memory containing instructions associated with the operation of one or more of the devices of the corresponding functional blocks 1110. In one example, command switch 1106 can be configured to receive and provide data associated with instructions for performing one or more DMA transfers, as described herein.In another example, the command switch 1106 can be configured to retrieve and provide data associated with commands for performing one or more operations specific to one or more devices of the function blocks 1110. In an illustrative example, the command switch 1106 can be configured to receive and provide data associated with commands for performing one or more filter operations, and the command switch 1106 can transfer the data to caches 1120 of corresponding function blocks 1110. In this example, the corresponding cache 1120 can be configured to send (e.g., load) the data associated with the instructions to the VPU 1116 or the PPE 1118 so that the respective device can perform the one or more filter operations.

[0057] Memory 1108 can include a device connected to the DMA hardware sequencer 1114a and / or the DMA hardware sequencer 1114b of the functional blocks 1110. In some embodiments, memory 1108 can receive and store sensor data generated by one or more sensors of a robot. For example, during robot operation, memory 1108 can be configured to receive data, at least partially, based on a direct connection to the one or more sensors or an indirect connection to the one or more sensors (e.g., via communication over a CAN bus and / or the like).In these examples, the sensor data can include image data associated with one or more images generated by one or more cameras, LiDAR data associated with one or more point clouds generated by one or more LiDAR sensors, radar data associated with one or more radar images generated by one or more radar sensors, and / or the like. In some embodiments, the memory 1108 can be configured to provide (e.g., transfer) the sensor data stored therein to one or more components of the functional blocks 1110.For example, during the processing of one or more images generated by the robot's one or more cameras, the DMA hardware sequencer 1114a and / or the DMA hardware sequencer 1114b can retrieve the image data from memory 1108 and cause the image data to be stored in VMEM 1112a and / or VMEM 1112b, respectively. In some embodiments, memory 1108 can receive and store data from the DMA hardware sequencer 1114a and / or the DMA hardware sequencer 1114b of the functional blocks 1110. For example, the DMA hardware sequencer 1114a and / or the DMA hardware sequencer 1114b can deliver image data, which has been updated at least partially based on the processing of the image data, to memory 1108, and memory 1108 can store the updated image data in memory 1108.

[0058] The 1110 functional blocks can include VMEMs 1112a, 1112b, DMA hardware sequencers 1114a, 1114b, vector processing units (VPUs) 1116a, 1116b, pixel processing engines (PPE) 1118a, 1118b, caches 1120a, 1120b, 1120c, 1120d and decoupled lookup tables (DLUTs) 1122a, 1122b. For clarity, these are individually referred to as VMEM 1112, DMA Hardware Sequencer 1114, VPU 1116, PPE 1118, Cache 1120, and DLUT 1122, and collectively as VMEMs 1112, DMA Hardware Sequencer 1114, VPUs 1116, PPEs 1118, Caches 1120, and DLUTs 1122, unless otherwise specified. Although certain connections are shown, it is understood that the connections shown are for simplicity and that one or more of the devices of Functional Blocks 1110 may be connected to one or more other devices of Functional Blocks 1110, unless expressly stated otherwise.

[0059] The VMEMs 1112 can include a device connected to the processor 1102 and the respective DMA hardware sequencers 1114, VPUs 1116, PPEs 1118, and caches 1120 of the functional blocks 1110. In some embodiments, the VMEMs 1112 can receive and store sensor data obtained from memory 1108. For example, the VMEMs 1112 can receive and store sensor data obtained from memory 1108 by the DMA hardware sequencers 1114. Additionally or alternatively, the VMEMs 1112 can receive and store sensor data obtained from memory 1108 via the command switch 1106. In some embodiments, the VMEMs 1112 can be connected to the PPEs 1118 via decoupled load / store units (DLSUs) 1124. As described here, the DLSU 1124 can be configured to buffer data between the VMEM 1112 and the PPE 1118 to reduce latency in communication between the VMEM 1112 and the PPE 1118.

[0060] The DMA hardware sequencers 1114 can comprise one or more processors that control the execution of one or more instructions. For example, the DMA hardware sequencers 1114 can receive instructions from the processor 1102, the respective VPUs 1116 or PPEs 1118, and / or a device (e.g., a device connected to the DMA hardware sequencers 1114, such as internal or external memory, not explicitly shown), and the DMA hardware sequencers 1114 can coordinate with the respective VPUs 1116 and / or PPEs 1118 to perform one or more operations during instruction execution. In one example, the DMA hardware sequencers 1114 can receive instructions that cause them to retrieve data (e.g., sensor data) from memory 1108 and store it in the respective VMEMs 1112.In some embodiments, the DMA hardware sequencers 1114 can perform one or more operations that are at least partially based on the data retrieved from memory 1108. For example, the DMA hardware sequencers 1114 can pad frames (e.g., image frames), manipulate addresses, manage overlapping data, manage different pass orders, accommodate different frame sizes, and / or the like. In some embodiments, the DMA hardware sequencers 1114 can receive signals (e.g., from the VPUs 1116 or PPEs 1118) indicating that one or more operations have been performed on the data stored in the VMEMs 1112, update one or more descriptors at least partially based on the data updates, and perform further operations on the data.

[0061] The VPUs 1116 can comprise one or more processors that execute one or more instructions. For example, the VPUs 1116 can receive instructions from processor 1102, and the respective VPUs 1116 can coordinate with the DMA hardware sequencers 1114 and / or PPEs 1118 to perform one or more operations during instruction execution. In one example, the VPUs 1116 can receive instructions from processor 1102 that cause the VPUs 1116 to trigger the respective DMA hardware sequencers 1114 to retrieve sensor data from memory 1108 and store the sensor data in the respective VMEMs 1112. In other examples, the VPUs 1116 can process the data stored in the respective VMEMs 1112 and write data back to the VMEMs 1112.In these examples, the data written by the VPUs 1116 to the respective VMEMs 1112 may include updated sensor data and / or data generated at least partially based on an analysis of the sensor data performed by the VPUs 1116, including object or feature positions within a frame, a classification indicating the type of an object or agent, and / or the like. In some embodiments, the VPUs 1116 may provide a signal to the respective DMA hardware sequencers 1114 (e.g., send, transmit, forward, etc.) to cause the DMA hardware sequencers 1114 to update one or more descriptors (described herein).For example, the VPUs 1116 can send a signal “ “ to the respective DMA hardware sequencers 1114 to cause the DMA hardware sequencers 1114 to update one or more descriptors, at least partially, based on the data written by the VPUs 1116 to the respective VMEMs 1112.

[0062] The PPEs 1118 can comprise one or more processors that execute one or more instructions. For example, the PPEs 1118 can receive instructions from processor 1102, and the respective PPEs 1118 can coordinate with the DMA hardware sequencers 1114 and / or the VPUs 1116 to perform the one or more operations during instruction execution. In an illustrative example, the PPEs 1118 can receive instructions from processor 1102 that cause the PPEs 1118 to trigger the respective DMA hardware sequencers 1114 to retrieve sensor data from memory 1108 (e.g., receive, acquire, capture, etc.) and store the sensor data in the respective VMEMs 1112. In examples, the PPEs 1118 can process the data stored in the respective VMEMs 1112 and write data back to the VMEMs 1112.In these examples, the data written by the PPEs 1118 to the respective VMEMs 1112 may include updated sensor data and / or data generated at least partially based on an analysis of the sensor data performed by the PPEs 1118, including object or feature positions within a frame, a classification indicating the type of an object or agent, and / or the like. In some embodiments, the PPEs 1118 may send a signal to the respective DMA hardware sequencers 1114 to cause the DMA hardware sequencers 1114 to update one or more descriptors (described herein). For example, the PPEs 1118 can send a signal to the respective DMA hardware sequencers 1114 to cause the DMA hardware sequencers 1114 to update one or more descriptors, at least partially, based on the data written by the PPEs 1118 to the respective VMEMs 1112.

[0063] The caches 1120 can include a device connected to the VMEMs 1112 and / or the command switch 1106. As mentioned above, the caches 1120 can receive data associated with commands from the command switches 1106 and load the commands into one or more devices of the functional blocks 1110 to cause the one or more devices to operate according to the commands. The DLUTs 1122 can include a processor and / or memory configured to store one or more lookup tables. In some embodiments, the DLUTs 1122 can be configured to enable communication between the processor 1102 and one or more components of the functional blocks 1110. For example, the DLUTs 1122 can be configured to communicate with the processor 1102 and / or one or more memory devices. Fig. 11 (e.g., memory 1108 and / or memory 1104). The DLUT 1122 can then manage the data storage and retrieval process between processor 1102 and the one or more devices from Fig. 11 manage. The DLSUs 1124 can include a device connected to the VMEMs 1112 and PPEs 1118 of a specific functional block 1110. For example, the DLSUs 1124 can receive and store the sensor data obtained from memory 1108 by the VMEMs 1112. Additionally or alternatively, the DLSUs 1124 can receive and store the data provided as output by the PPEs 1118.

[0064] In some cases, the systems and procedures described here can be used in a simulation environment (such as NVIDIA DriveSIM) with simulated data (such as simulated sensor data from simulated sensors of a virtual or simulated machine). For example, simulated sensor data and / or map data can be used to identify areas of interest (such as parking spaces) and sub-areas (such as parts of a parking lot with curbs, wheel stops, etc.) within the simulation environment, and this information can be used to perform operations (such as parking) with the virtual machine in that environment. These simulated operations can be used to test the performance of the underlying algorithms, systems, and / or processes before deployment in the real world. In some cases, the simulation can be used to generate synthetic training data—for example,Training data containing areas and / or sub-areas of interest from the simulation. The synthetic training data (in addition to or as an alternative to real-world data) can then be processed to determine the geometry and / or other information relating to regions of interest, such as parking spaces or pallet delivery locations within a warehouse. In any case, for example, when a simulation environment is used for testing, validation, training, etc., the simulation environment and / or the associated training data can be rendered or otherwise generated using one or more light transport algorithms—such as ray tracing and / or path tracing algorithms. In some embodiments, the simulation environment and / or one or more objects, features, or components thereof can be embedded within a three-dimensional (3D) platform for collaborative content creation (e.g., a web interface).NVIDIA's OMNIVERSE) is used to create or manage content for industrial digitization, generative physical AI, and / or other use cases, applications, or services. The content collaboration platform or system may include, for example, a system for using or developing Universal Scene Descriptor (USD) data (e.g., OpenUSD) to manage objects, features, scenes, etc., in a simulated environment, a digital environment, etc. The platform may include realistic physics simulation, such as using NVIDIA's PhysX SDK to simulate real-world physics and physical interactions with simulations hosted by the platform. The platform may integrate OpenUSD, along with ray tracing / path tracing / light transport simulation (e.g., NVIDIA's RTX rendering technologies), into software tools and simulation workflows for creating, training, deploying, or testing AI systems—such as…Systems for testing, validating, training (e.g., machine learning models, neural networks, etc.) and / or other tasks related to automotive, robotics, machinery, or other applications.

[0065] In at least one implementation, a small language model optimized for at least one target language can be hosted in a cloud environment and made available for use by various people, entities, systems, processes, and the like. In at least one implementation, such models can also be deployed and used by various entities on their resources, whether local resources or allocated portions of physical or virtual multi-user resources, among other such options.

[0066] In some implementations, a model can be deployed as part of a software container, such as an NVIDIA Corporation NIM, which can contain the code and support required to perform inference operations using the model. Such a container can include a set of user-friendly inference microservices to accelerate the deployment of base models in a cloud deployment or data center and can help manage the security of request and generated response data. The container can be preconfigured to simplify deployment and include one or more optimized inference engines. The container can also include management functions for handling tasks such as identity management, metric generation, health checks, and status monitoring.In some examples, the machine learning model (small language model) can therefore be packaged as a microservice—such as an inference microservice—which can contain a container (e.g., an operating system-level virtualization package) that may include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model engine. The inference microservice might, for example, contain the container itself and the model (e.g., weights and biases). In some cases, for example, if the machine learning model is small enough (e.g., has a sufficiently small number of parameters), the model can be contained within the container itself. In other cases—e.g., if the model is large—the model can be hosted / stored in the cloud (e.g., in a data center) and / or on-premises and / or at the edge (e.g., on a local server or computer device, but outside the container).In such implementations, the model can be accessible via one or more APIs—such as REST APIs. Therefore, the machine learning models described here can, in some implementations, be deployed as an inference microservice to accelerate model deployment in any cloud, data center, or edge computing system while ensuring data security. The inference microservice might include, for example, one or more APIs, a pre-configured container for simplified deployment, and an optimized inference engine (e.g., a REST API).Built with standardized software for deploying and running AI models (such as NVIDIA's Triton Inference Server) and / or one or more APIs for high-performance deep learning inference, which may include an inference runtime environment and model optimizations that provide low latency and high throughput for production applications (such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). The machine learning models described here can be included as part of the microservice along with accelerated infrastructure that can be deployed with a single command and / or orchestrated and automatically scaled using a container orchestration system on accelerated infrastructure (e.g., from a single device to the size of a data center).As such, the inference microservice may include the machine learning model(s) (e.g., optimized for high-performance inference), inference runtime software for executing the machine learning model(s) and providing outputs / responses to inputs (e.g., user requests, prompts, etc.), and enterprise management software for providing health checks, identity verification, and / or other monitoring capabilities. In some embodiments, the inference microservice may include software for performing an in-place replacement and / or update of the machine learning model(s). During the replacement or update, the software performing the replacement / update may retain the user configurations of the inference runtime software and the enterprise management software.

[0067] The systems and methods described here can be used for a wide variety of purposes, including but not limited to machines (e.g., robots, vehicles, construction equipment, warehouse vehicles / machines, autonomous, semi-autonomous, and / or other machine types), machine locomotion, machine control, synthetic data generation, model training (e.g., using real, enhanced, and / or synthetic data, such as synthetic data generated using a simulation platform or system, synthetic data generation techniques such as those described here, etc.), perception, augmented reality (AR), virtual reality (VR), mixed reality (MR), robotics, security, and surveillance (e.g.,in a smart city implementation), autonomous or semi-autonomous machine applications, deep learning, environment simulation, object or actor simulation and / or digital twinning, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), distributed or collaborative content creation for 3D assets (e.g., using Universal Scene Descriptor (USD) data such as OpenUSD and / or other data types), cloud computing, generative artificial intelligence (e.g., using one or more diffusion models, transformer models, etc.), and / or other suitable applications.

[0068] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented with a robot or robotic platform, flight systems, medical systems, boat systems, intelligent area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations (e.g., in a driving or vehicle simulation, in a robotics simulation, in a smart city or surveillance simulation, etc.), systems for performing digital twin operations (e.g.,in conjunction with a collaborative platform or content creation system, such as, without limitation, NVIDIA OMNIVERSE and / or any other platform, system, or service that uses USD or OpenUSD data types; systems implemented with an edge device; systems containing one or more virtual machines (VMs); systems performing synthetic data generation operations (e.g., using one or more neural rendering fields (NERFs), Gaussian splat techniques, diffusion models, transformer models, etc.); systems implemented at least partially in a data center; systems performing conversational AI operations; systems implementing one or more language models—such as one or more large language models (LLMs), one or more image-language models (VLMs), one or more language and image-language models (VSLMs), etc., etc.), systems that are at least partially implemented in a data center, systems for performing dialogue-oriented AI operations, systems that implement one or more language models - such as one or more large language models (LLMs), one or more image processing language models (VLMs), one or more multimodal language models, etc., systems for performing light transport simulations, systems for performing collaborative content creation for 3D assets (e.g., using Universal Scene Descriptor (USD) data such as OpenUSD, CAD data, 2D and / or 3D graphics or design data and / or other data types), systems that are at least partially implemented using cloud computing resources, and / or other types of systems. DATA CENTER

[0069] Fig. Figure 12 shows an example of a data center or computing center 1200, in which at least one embodiment can be used. In at least one embodiment, the computing center 1200 comprises a computing infrastructure layer 1210, a framework layer 1220, a software layer 1230, and an application layer 1240.

[0070] In at least one version, as in Fig. As shown in Figure 12, the data center infrastructure layer 1210 can have a resource coordinator 1212, clustered compute resources 1214, and node compute resources (“node CRs”) 1216(1)-1216(N), where “N” represents any positive integer. In at least one embodiment, the node CRs 1216(1)-1216(N) can include, among other things, any number of central processing units (“CPUs”) or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processing units, etc.), storage devices (e.g., dynamic read-only memories), storage devices (e.g., solid-state or hard disk drives), network input / output devices (“NW I / O devices”), network switches, virtual machines (“VMs”), power supplies, etc. etc.), storage devices (e.g. dynamic read-only memory), storage devices (e.g.Solid-state or hard disk drives), network input / output devices (“NW I / O”), network switches, virtual machines (“VMs”), power supply modules and cooling modules, etc. In at least one embodiment, one or more node CRs from node CRs 1216(1)-1216(N) can be a server with one or more of the above-mentioned computing resources.

[0071] In at least one embodiment, grouped compute resources 1214 can comprise separate groupings of node CRs housed in one or more racks (not shown), or many racks housed in data centers at different geographic locations (also not shown). Separate groupings of node CRs within the grouped compute resources 1214 can comprise grouped compute, network, storage, or memory resources that can be configured or allocated to support one or more workloads. In at least one embodiment, multiple node CRs, including CPUs or processors, can be grouped in one or more racks to provide compute resources to support one or more workloads.In at least one embodiment, one or more racks can also include any number of power supply modules, cooling modules and network switches in any combination.

[0072] In at least one embodiment, the resource orchestrator 1212 can configure or otherwise control one or more node CRs 1216(1)-1216(N) and / or grouped computing resources 1214. In at least one embodiment, the resource orchestrator 1212 can include a software design infrastructure management unit (“SDI”) for the data center 1200. In at least one embodiment, the resource orchestrator 1212 can include hardware, software, or a combination thereof.

[0073] In at least one embodiment, as in Fig. As shown in Figure 12, the framework layer 1220 comprises a job scheduler 1222, a configuration manager 1224, a resource manager 1226, and a distributed file system 1228. In at least one embodiment, the framework layer 1220 may comprise a framework for supporting the software 1232 of the software layer 1230 and / or one or more applications 1242 of the application layer 1240. In at least one embodiment, the software 1232 or the application(s) 1242 may each comprise web-based service software or applications such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, but not limited to, the framework layer 1220 may be a type of free and open-source software web application framework such as Apache Spark™ (hereinafter “Spark”), which may use a distributed file system 1228 for processing large amounts of data (e.g., “Big Data”).In at least one embodiment, the job scheduler 1222 can include a Spark driver to facilitate the scheduling of workloads supported by different layers of the data center 1200. In at least one embodiment, the configuration manager 1224 can be able to configure different layers, such as the software layer 1230 and the framework layer 1220, including Spark and the distributed file system 1228, to support the processing of large amounts of data. In at least one embodiment, the resource manager 1226 can be able to manage clustered or grouped computing resources that are associated with or allocated to support the distributed file system 1228 and the job scheduler 1222. In at least one embodiment, clustered or grouped computing resources can include grouped computing resources 1214 on the data center infrastructure layer 1210.In at least one version, the resource manager 1226 can work together with the resource orchestrator 1212 to manage these allocated or assigned computing resources.

[0074] In at least one version, the software 1232, which is in software layer 1230, may include software that is used by at least parts of the node CRs 1216(1)-1216(N), the grouped compute resources 1214, and / or the distributed file system 1228 of framework layer 1220. The one or more types of software may include, among others, internet website search software, email virus scanning software, database software, and streaming video content software.

[0075] In at least one embodiment, the applications contained in the application layer 1240 1242 may comprise one or more types of applications used by at least parts of the node CRs 1216(1)-1216(N), grouped compute resources 1214, and / or the distributed file system 1228 of the framework layer 1220. One or more types of applications may include, among others, any number of genomics applications, cognitive computing applications, and machine learning applications, including training or inference software, machine learning software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0076] In at least one embodiment, each of the configuration manager 1224, resource manager 1226, and resource orchestrator 1212 can implement any number and type of self-modifying actions based on any set and type of data acquired in a technically feasible manner. In at least one embodiment, self-modifying actions can relieve a data center operator of data center 1200 of potentially making poor configuration decisions and potentially avoiding underutilized and / or underperforming parts of a data center.

[0077] In at least one embodiment, the Data Center 1200 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or deriving information using one or more machine learning models according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model may be trained by calculating weight parameters according to a neural network architecture using software and computing resources described above in relation to the Data Center 1200.In at least one embodiment, trained machine learning models corresponding to one or more neural networks can be used to derive or predict information using resources described above in relation to the Computing Center 1200, by using weight parameters calculated by one or more training techniques described herein.

[0078] In at least one embodiment, the data center can use CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, or other hardware to perform training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above can be configured as a service to allow users to train or infer information, such as image recognition, speech recognition, or other artificial intelligence services.

[0079] Such components can be used to generate an alternative image, such as a bird's-eye view image, from disparity data with limited capacity hardware, such as an embedded processor without access to external memory. COMPUTER SYSTEMS

[0080] Fig. Figure 13 is a block diagram illustrating an exemplary computer system, which may be a system with interconnected devices and components, a system-on-a-chip (SoC), or a combination thereof, configured with a processor that may include execution units for carrying out an instruction according to at least one embodiment. In at least one embodiment, the computer system 1300 may, without limitation, include a component such as a processor 1302 to utilize execution units, including logic, for carrying out algorithms for processing data according to the present disclosure, as in the embodiment described herein.In at least one embodiment, the Computer System 1300 can use processors such as the PENTIUM® processor family, Xeon™, Itanium®, XScale™ and / or StrongARM™, Intel® Core™ or Intel® Nervana™ microprocessors available from Intel Corporation in Santa Clara, California, although other computer systems (including PCs with other microprocessors, technical workstations, set-top boxes, and the like) can also be used. In at least one embodiment, the Computer System 1300 can run a version of the WINDOWS operating system available from Microsoft Corporation in Redmond, Washington, although other operating systems (e.g., UNIX and Linux), embedded software, and / or graphical user interfaces can also be used.

[0081] Embodiments can be used in other devices such as handheld devices and embedded applications. Some examples of handheld devices are mobile phones, Internet Protocol devices, digital cameras, personal digital assistants (“PDAs”), and handheld PCs. In at least one embodiment, embedded applications can include a microcontroller, a digital signal processor (“DSP”), a system-on-a-chip, network computers (“NetPCs”), set-top boxes, network hubs, wide area network (“WAN”) switches, or any other system capable of executing one or more instructions according to at least one embodiment.

[0082] In at least one version, the computer system 1300 may, for example, have a processor 1302, which may have one or more execution units 1308 for performing the training and / or inference of machine learning models according to the techniques described herein. In at least one version, the computer system 1300 is a desktop or server system with a single processor, but in another version, the computer system 1300 may be a multiprocessor system. In at least one version, the processor 1302 may, without restriction, comprise a complex instruction set (CISC) microprocessor, a reduced instruction set (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor implementing a combination of instruction sets, or any other processing device, such as a digital signal processor.In at least one embodiment, the processor 1302 can be coupled to a processor bus 1310, which can transmit data signals between the processor 1302 and other components in the computer system 1300.

[0083] In at least one embodiment, the processor 1302 can, without limitation, include an internal cache memory (“cache”) 1304 of level 1 (“L1”). In at least one embodiment, the processor 1302 can have a single internal cache or multiple levels of an internal cache. In at least one embodiment, the cache memory can be located outside the processor 1302. Other embodiments can also include a combination of internal and external cache, depending on the specific implementation and requirements. In at least one embodiment, the register file 1306 can store different types of data in different registers, including, but not limited to, integer registers, floating-point registers, status registers, and instruction pointer registers.

[0084] In at least one embodiment, the execution unit 1308, including, without limitation, logic for performing integer and floating-point operations, is also located in the processor 1302. In at least one embodiment, the processor 1302 may also include a read-only microcode memory (“ROM”) that stores microcode for specific macro instructions. In at least one embodiment, the execution unit 1308 may include logic for processing a packed instruction set 1309. In at least one embodiment, by incorporating the packed instruction set 1309 into an instruction set of a general-purpose processor 1302, together with associated instruction execution circuitry, operations used by many multimedia applications can be performed using packed data in a general-purpose processor 1302.In one or more embodiments, many multimedia applications can be accelerated and run more efficiently by utilizing the full width of a processor's data bus for performing operations on packed data, thus eliminating the need to transfer smaller data units across the processor's data bus to perform one or more operations on each data element.

[0085] In at least one embodiment, the execution unit 1308 can also be used in microcontrollers, embedded processors, graphics devices, DSPs, and other types of logic circuits. In at least one embodiment, the computer system 1300 can include a memory 1320 without restriction. In at least one embodiment, the memory 1320 can be implemented as a DRAM (Dynamic Random Access Memory) device, SRAM (Static Random Access Memory) device, flash memory device, or other memory device. In at least one embodiment, the memory 1320 can store instructions 1319 and / or data 1321, which are represented by data signals that can be executed by the processor 1302.

[0086] In at least one version, the system logic chip can be connected to the processor bus 1310 and the memory 1320. In at least one version, the system logic chip can, for example, have a memory controller hub (“MCH”) 1316, and the processor 1302 can communicate with the MCH 1316 via the processor bus 1310. In at least one embodiment, the MCH 1316 can provide a high-bandwidth memory path 1318 to the memory 1320 for storing instructions and data, as well as for storing graphics instructions, data, and textures. In at least one embodiment, the MCH 1316 can route data signals between the processor 1302, the memory 1320, and other components in the computer system 1300, and bridge data signals between the processor bus 1310, the memory 1320, and a system I / O 1322. In at least one embodiment, the system logic chip can provide a graphics port for coupling with a graphics controller.In at least one embodiment, the MCH 1316 can be coupled to the memory 1320 via a high-bandwidth memory path 1318, and the graphics / video card 1312 can be coupled to the MCH 1316 via an Accelerated Graphics Port (“AGP”) connection 1314.

[0087] In at least one embodiment, the computer system 1300 can use a system I / O 1322, which is a proprietary hub interface bus, to connect the MCH 1316 to the I / O controller hub (“ICH”) 1330. In at least one embodiment, the ICH 1330 can provide direct connections to some I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus can, without limitation, include a high-speed I / O bus for connecting peripheral devices to the memory 1320, the chipset, and the processor 1302. Examples include, without limitation, an audio controller 1329, a firmware hub (“Flash BIOS”) 1328, a wireless transceiver 1326, a data storage device 1324, a legacy I / O controller 1323 with user input and keyboard interfaces 1325, a serial expansion port 1327, such as a Universal Serial Bus (“USB”), and a network controller 1334.The data storage device 1324 can include a hard disk drive, a floppy disk drive, a CD-ROM drive, a flash memory, or another mass storage device.

[0088] In at least one embodiment, Fig. 13 a system comprising interconnected hardware devices or “chips”, whereas in other embodiments Fig. 13 may show an exemplary system on a chip (“SoC”). In at least one embodiment, devices may be interconnected by proprietary connections, standardized connections (e.g., PCIe), or a combination thereof. In at least one embodiment, one or more components of the Computer System 900 are interconnected via Compute Express Link (CXL) connections.

[0089] Such components can be used to generate an alternative image, such as a bird's-eye view image, from disparity data with limited capacity hardware, such as an embedded processor without access to external memory.

[0090] Fig. Figure 14 is a block diagram showing a device 1400 for using a processor 1410 according to at least one embodiment. In at least one embodiment, the electronic device 1400 can be, for example, and without limitation, a notebook, a tower server, a rack server, a blade server, a laptop, a desktop computer, a tablet, a mobile device, a telephone, an embedded computer, or any other suitable electronic device.

[0091] In at least one version, the electronic device 1400 can, for example, have a processor 1410 connected to any number or type of components, peripherals, modules, or devices. In at least one version, the processor 1410 is connected via a bus or interface, such as a 1°C bus, a system management bus (“SMBus”), an LPC bus (Low Pin Count), a serial peripheral interface (“SPI”), an HDA bus (High Definition Audio), a SATA bus (Serial Advance Technology Attachment), a USB bus (Universal Serial Bus, versions 1, 2, 3), or a UART bus (Universal Asynchronous Receiver / Transmitter). In at least one embodiment, the Fig. 14 a system comprising interconnected hardware devices or “chips”, whereas in other embodiments Fig. 14 can show an exemplary system-on-a-chip (“SoC”). In at least one embodiment, the Fig. The devices shown in Figure 14 are connected to each other by proprietary connections, standardized connections (e.g., PCIe), or a combination thereof. In at least one embodiment, one or more components of Fig. 14 are interconnected via CXL (Compute Express Link) connections.

[0092] In at least one version Fig. 14 a display 1424, a touchscreen 1425, a touchpad 1430, a near field communication unit (“NFC”) 1445, a sensor hub 1440, a temperature sensor 1446, an Express chipset (“EC”) 1435, a Trusted Platform Module (“TPM”) 1438, BIOS / Firmware / Flash memory (“BIOS, FW Flash”) 1422, a DSP 1460, a drive 1420 such as a solid-state drive (“SSD”) or a hard disk drive (“HDD”), a WLAN unit (“WLAN”) 1450, a Bluetooth unit 1452, a WWAN unit (“WWAN”) 1456, a Global Positioning System (GPS) 1455, a camera (“USB 3.0 camera”) 1454 such as a USB 3.0 camera and / or a Low Power Double Data Rate The LPDDR3 memory unit (LPDDR3) is implemented, for example, in the LPDDR3 standard. These components can each be implemented in any suitable way.

[0093] In at least one embodiment, other components can be connected to the processor 1410 via the components described above. In at least one embodiment, an accelerometer 1441, an ambient light sensor (“ALS”) 1442, a compass 1443, and a gyroscope 1444 can be connected to the sensor hub 1440. In at least one embodiment, a temperature sensor 1439, a fan 1437, a keyboard 1436, and a touchpad 1430 can be connected to the EC 1435. In at least one embodiment, loudspeakers 1463, headphones 1464, and a microphone (“Mic”) 1465 can be connected to an audio unit (“audio codec and Class-D amplifier”) 1462, which in turn can be connected to the DSP 1460. In at least one embodiment, the audio unit 1462 can, for example, and without limitation, comprise an audio encoder / decoder (“codec”) and a Class-D amplifier.In at least one embodiment, the SIM card (“SIM”) 1457 can be connected to the WWAN unit 1456. In at least one embodiment, components for wireless communication (“ “) such as the WLAN unit 1450 and the Bluetooth unit 1452, as well as the WWAN unit 1456, can be implemented in a Next Generation Form Factor (“NGFF”).

[0094] Such components can be used to generate an alternative image, such as a bird's-eye view image, from disparity data with limited capacity, such as an embedded processor without access to external memory.

[0095] Fig. Figure 15 is a block diagram of a processing system according to at least one embodiment. In at least one embodiment, the system 1500 comprises one or more processors 1502 and one or more graphics processors 1508 and can be a single-processor desktop system, a multiprocessor workstation system, or a server system with a large number of processors 1502 or processor cores 1507. In at least one embodiment, the system 1500 is a processing platform integrated into a system-on-a-chip (SoC) integrated circuit for use in mobile, portable, or embedded devices.

[0096] In at least one embodiment, the System 1500 may comprise or be integrated into a server-based gaming platform, a gaming console (including a gaming and media console), a mobile gaming console, a handheld gaming console, or an online gaming console. In at least one embodiment, the System 1500 is a mobile phone, a smartphone, a tablet device, or a mobile internet device. In at least one embodiment, the Processing System 1500 may also comprise, be coupled to, or be integrated into a wearable device, such as a wearable smartwatch device, smart glasses device, augmented reality device, or virtual reality device.In at least one embodiment, the processing system 1500 is a television or set-top box device with one or more processors 1502 and a graphical interface generated by one or more graphics processors 1508.

[0097] In at least one embodiment, one or more processors 1502 each have one or more processor cores 1507 for processing instructions which, when executed, perform operations for system and application software. In at least one embodiment, each of the one or more processor cores 1507 is configured to process a specific instruction set 1509. In at least one embodiment, the instruction set 1509 may enable Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or computation via a Very Long Instruction Word (VLIW). In at least one embodiment, each processor core 1507 may process a different instruction set 1509, which may contain instructions to facilitate the emulation of other instruction sets. In at least one embodiment, the processor core 1507 may also include other processing devices, such as a digital signal processor (DSP).

[0098] In at least one embodiment, the processor(s) 1502 comprises a cache memory 1504. In at least one embodiment, the processor(s) 1502 may have a single internal cache or multiple levels of an internal cache. In at least one embodiment, the cache memory is shared by different components of the processor(s) 1502. In at least one embodiment, the processor(s) 1502 also utilizes an external cache (e.g., a Level 3 cache (L3) or a Last Level Cache (LLC)) (not shown), which can be shared by the processor core(s) 1507 using known cache coherence techniques. In at least one embodiment, the processor(s) 1502 additionally includes a register file 1506, which may contain different types of registers for storing different types of data (e.g., integer registers, floating-point registers, status registers, and an instruction pointer register).In at least one embodiment, the register file may contain 1506 general-purpose registers or other registers.

[0099] In at least one embodiment, one or more processors 1502 are coupled to one or more interface buses 1510 to transmit communication signals, such as address, data, or control signals, between the processors 1502 and other components in the system 1500. In at least one embodiment, the interface bus 1510 can be a processor bus, such as a version of a Direct Media Interface (DMI) bus. In at least one embodiment, the interface bus (or interface buses) 1510 is not limited to a DMI bus and can include one or more Peripheral Component Interconnect buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. In at least one embodiment, the processors 1502 have an integrated memory controller 1516 and a platform controller hub 1530.In at least one version, the 1516 storage controller simplifies communication between a storage device and other parts of the 1500 system, while the 1530 Platform Controller Hub (PCH) establishes connections to I / O devices via a local I / O bus.

[0100] In at least one version, the device 1520 can be dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, phase-change memory, or any other device suitable for use as process memory. In at least one embodiment, the device 1520 can serve as system memory for the system 1500 to store data 1522 and instructions 1521 used when one or more processors 1502 execute an application or process. In at least one embodiment, the memory controller 1516 is also coupled with an optional external graphics processor 1512, which can communicate with one or more graphics processors 1508 within the processors 1502 to perform graphics and media operations. In at least one embodiment, a display device 1511 can be connected to the processor(s) 1502.In at least one embodiment, the display device 1511 can comprise one or more internal display devices, such as in a mobile electronic device or a laptop device, or an external display device connected via a display interface (e.g., DisplayPort, etc.). In at least one embodiment, the display device 1511 can comprise a head-mounted display (HMD), for example, a stereoscopic display device for use in virtual reality (VR) or augmented reality (AR) applications.

[0101] In at least one embodiment, the platform controller hub 1530 enables peripheral devices to be connected to the storage device 1520 and the processor(s) 1502 via a high-speed I / O bus. In at least one embodiment, the I / O peripheral devices include, among others, an audio controller 1546, a network controller 1534, a firmware interface 1528, a wireless transceiver 1526, touch sensors 1525, and a data storage device 1524 (e.g., a hard disk drive, flash memory, etc.). In at least one embodiment, the data storage device 1524 can be connected via a storage interface (e.g., SATA) or via a peripheral bus, such as a peripheral component interconnect bus (e.g., PCI, PCI Express). In at least one embodiment, the touch sensors 1525 can include touchscreen sensors, pressure sensors, or fingerprint sensors.In at least one embodiment, the wireless transceiver 1526 can be a Wi-Fi transceiver, a Bluetooth transceiver, or a cellular transceiver such as a 3G, 4G, or Long Term Evolution (LTE) transceiver. In at least one embodiment, the firmware interface 1528 enables communication with the system firmware and can, for example, be a Unified Extensible Firmware Interface (UEFI). In at least one embodiment, the network controller 1534 can enable a network connection to a wired network. In at least one embodiment, a high-performance network controller (not shown) is coupled to the interface bus(s) 1510. In at least one embodiment, the audio controller 1546 is a multi-channel high-definition audio controller. In at least one embodiment, the system 1500 includes an optional legacy I / O controller 1540 for coupling legacy devices (e.g.,Personal System 2 (PS / 2)) with the system. In at least one embodiment, the platform controller hub 1530 can also be connected to one or more Universal Serial Bus (USB) controllers 1542 to connect input devices such as keyboard and mouse combinations 1543, a camera 1544, or other USB input devices.

[0102] In at least one embodiment, an instance of the memory controller 1516 and the platform controller hub 1530 can be integrated into a discrete external graphics processor, such as the external graphics processor 1512. In at least one embodiment, the platform controller hub 1530, d, and / or the memory controller 1516 can be located outside of one or more processors 1502. For example, in at least one embodiment, the system 1500 can have an external memory controller 1516 and a platform controller hub 1530, which can be configured as a memory controller hub and peripheral controller hub within a system chipset that communicates with the processor(s) 1502.

[0103] Such components can be used to generate an alternative image, such as a bird's-eye view image, from disparity data with limited capacity hardware, such as an embedded processor without access to external memory.

[0104] Fig. Figure 16 is a block diagram of a processor 1600 with one or more processor cores 1602A-1602N, an integrated memory controller 1614, and an integrated graphics processor 1608, as shown in at least one embodiment. In at least one version, the processor 1600 can have additional cores up to and including the additional core 1602N, which are represented by dashed boxes. In at least one embodiment, each of the processor cores 1602A-1602N comprises one or more internal cache units 1604A-1604N. In at least one embodiment, each processor core also has access to one or more shared cache units 1606.

[0105] In at least one embodiment, the internal cache units 1604A-1604N and the shared cache units 1606 constitute a cache memory hierarchy within the processor 1600. In at least one embodiment, the cache units 1604A-1604N can provide at least one level of instruction and data cache within each processor core and one or more levels of shared mid-level cache, such as Level 2 (L2), Level 3 (L3), Level 4 (L4), or other cache levels, with the highest cache level prior to external memory being classified as LLC. In at least one embodiment, the cache coherence logic ensures coherence between different cache units 1606 and 1604A-1604N.

[0106] In at least one embodiment, the processor 1600 can also include a set of one or more bus control units 1616 and a system agent core 1610. In at least one embodiment, one or more bus control units 1616 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, the system agent core 1610 provides management functions for various processor components. In at least one embodiment, the system agent core 1610 includes one or more integrated memory controllers 1614 to manage access to various external storage devices (not shown).

[0107] In at least one embodiment, one or more processor cores 1602A-1602N support simultaneous multithreading. In at least one embodiment, the system agent core 1610 comprises components for coordinating the processor cores 1602A-1602N during multithreading processing. In at least one embodiment, the system agent core 1610 may additionally comprise a power control unit (PCU) comprising logic and components for regulating one or more power states of the processor core(s) 1602A-1602N and the graphics processor 1608.

[0108] In at least one embodiment, the processor 1600 additionally includes a graphics processor 1608 for performing graphics processing operations. In at least one embodiment, the graphics processor 1608 is connected to one or more shared cache units 1606 and the system agent core 1610, which has one or more integrated memory controllers 1614. In at least one embodiment, the system agent core 1610 also includes a display controller 1611 for controlling the output of the graphics processor to one or more coupled displays. In at least one embodiment, the display controller 1611 can also be a separate module that is coupled to the graphics processor 1608 via at least one connection, or it can be integrated into the graphics processor 1608.

[0109] In at least one embodiment, a ring-based interconnect 1612 is used to couple internal components of the processor 1600. In at least one embodiment, an alternative interconnect can be used, such as a point-to-point connection, a switched connection, or other techniques. In at least one embodiment, the graphics processor 1608 is connected to a ring-based interconnect 1612 via an I / O connection 1613.

[0110] In at least one version, the I / O connection 1613 is one of several types of I / O connections, such as an on-package I / O connection, that enables communication between different processor components and an embedded high-performance memory module 1618, such as an eDRAM module. In at least one embodiment, each of the processor cores 1602A-1602N and the graphics processor 1608 utilizes embedded memory modules 1618 as a shared last-level cache.

[0111] In at least one embodiment, the processor cores 1602A-1602N are homogeneous cores executing a common instruction set architecture. In at least one embodiment, the processor cores 1602A-1602N are heterogeneous with respect to the instruction set architecture (ISA), wherein one or more of the processor cores 1602A-1602N execute a common instruction set, while one or more other cores of the processor cores 1602A-1602N execute a subset of a common instruction set or a different instruction set. In at least one embodiment, the processor cores 1602A-1602N are heterogeneous with respect to their microarchitecture, wherein one or more cores with relatively higher power consumption are coupled with one or more performance cores with lower power consumption. In at least one embodiment, the processor 1600 can be implemented on one or more chips or as a system-on-a-chip (SoC) integrated circuit.

[0112] Such components can be used to generate an alternative image, such as a bird's-eye view image, from disparity data with limited capacity, such as an embedded processor without access to external memory. AUTONOMOUS VEHICLE

[0113] Fig. Figure 17A shows an example of an autonomous vehicle 1700 according to at least one embodiment. In at least one embodiment, the autonomous vehicle 1700 (hereinafter referred to alternatively as "vehicle 1700") can be, without restriction, a passenger vehicle, such as a car, a truck, a bus, and / or another type of vehicle capable of carrying one or more passengers. In at least one embodiment, the vehicle 1700 can be a semi-trailer truck used for transporting cargo. In at least one embodiment, the vehicle 1700 can be an aircraft, a robotic vehicle, or another type of vehicle.

[0114] Autonomous vehicles can be described using levels of automation defined by the National Highway Traffic Safety Administration (“NHTSA”), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (“SAE”) in the “Taxonomy and Definitions for Terms Related to Autonomous Driving Systems for On-Road Vehicles” (e.g., Standard No. J3016-201806, published June 15, 2018, Standard No. J3016-201609, published September 30, 2016, and earlier and future versions of this standard). In at least one configuration, the vehicle 1700 can perform functions according to one or more of the five levels of autonomous driving, 1 through 5. For example, in at least one configuration, the vehicle 1700 can perform conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the configuration.

[0115] In at least one embodiment, the vehicle 1700 can comprise, without limitation, components such as a chassis, a vehicle body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. In at least one embodiment, the vehicle 1700 can comprise, without limitation, a drive system 1750, such as an internal combustion engine, a hybrid-electric drive, a fully electric motor, and / or another type of drive system. In at least one embodiment, the drive system 1750 can be connected to a drivetrain of the vehicle 1700, which can include, without limitation, a transmission to enable the propulsion of the vehicle 1700. In at least one embodiment, the drive system 1750 can be controlled in response to the reception of signals from one or more throttle / accelerator pedals 1752.

[0116] In at least one embodiment, a steering system 1754, which may, for example, include a steering wheel, is used to steer the vehicle 1700 (for example, along a desired path or route) when the drive system 1750 is running (for example, when the vehicle 1700 is moving). In at least one embodiment, the steering system 1754 can receive signals from one or more steering actuators 1756. In at least one embodiment, a steering wheel may be optional for the fully automated functionality (Level 5). In at least one embodiment, a brake sensor system 1746 can be used to actuate vehicle brakes in response to receiving signals from one or more brake actuators 1748 and / or brake sensors.

[0117] In at least one embodiment, one or more controllers 1736 supply one or more system-on-chips (“SoCs”) without restriction (in Fig. 17A not shown) and / or may include one or more graphics processing units (“GPUs”), sending signals (e.g., representing commands) to one or more components and / or systems of the vehicle 1700. In at least one embodiment, the controller(s) 1736 may, for example, send signals to actuate the vehicle brakes via brake actuator(s) 1748, the steering system 1754 via steering actuator(s) 1756, and the drive system 1750 via throttle valve(s) / accelerator pedal(s) 1752. In at least one embodiment, the controller(s) 1736 may include one or more integrated computing units that process sensor signals and output operating commands (e.g., signals representing commands) to enable autonomous driving and / or to assist a human driver in driving the vehicle 1700.In at least one embodiment, the controller(s) 1736 may comprise a first controller for autonomous driving functions, a second controller for functional safety functions, a third controller for artificial intelligence functions (e.g., computer vision), a fourth controller for infotainment functions, a fifth controller for emergency redundancy, and / or other controllers. In at least one embodiment, a single controller may perform two or more of the above-mentioned functions, two or more controllers may perform a single function, and / or any combination thereof.

[0118] In at least one version, the control units 1736 provide signals for controlling one or more components and / or systems of the vehicle 1700 in response to sensor data received from one or more sensors (e.g. sensor inputs). In at least one embodiment, sensor data can be obtained, for example, and without limitation, from global navigation satellite systems (“GNSS”) sensors 1758 (e.g., Global Positioning System sensors), radar sensors 1760, ultrasonic sensors 1762, lidar sensors 1764, sensors of an inertial measurement unit (“IMU”) 1766 (e.g., accelerometers, gyroscopes, a magnetic compass or compasses, magnetometers, etc.), microphones 1796, stereo cameras 1768, wide-angle camera(s) 1770 (e.g., fisheye cameras), infrared camera(s) 1772, surround camera(s) 1774 (e.g., 360-degree cameras), long-range cameras (not in Fig. 17A shown), medium-range camera(s) (not shown in Fig. 17A shown), speed sensor(s) 1744 (e.g. for measuring the speed of the vehicle 1700), vibration sensor(s) 1742, steering sensor(s) 1740, brake sensor(s) (e.g. as part of the brake sensor system 1746) and / or other sensor types.

[0119] In at least one embodiment, one or more controllers 1736 can receive inputs (e.g., represented by input data) from a combination instrument 1732 of the vehicle 1700 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (“HMI”) display 1734, an acoustic signal generator, a loudspeaker, and / or via other components of the vehicle 1700. In at least one embodiment, the outputs can provide information such as vehicle speed, time, map data (e.g., a high-resolution map (in Fig. 17A not shown), location data (e.g., the location of vehicle 1700, e.g., on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and the status of objects as perceived by the controller(s) 1736, etc. For example, in at least one embodiment, the HMI display 1734 can show information about the presence of one or more objects (e.g., a road sign, a warning sign, a changing traffic light, etc.) and / or information about driving maneuvers that the vehicle has made, is currently making, or will make (e.g., changing lanes now, taking exit 34B in two miles, etc.).

[0120] In at least one embodiment, the vehicle 1700 also includes a network interface 1724, which can use wireless antennas 1726 and / or modems to communicate over one or more networks. For example, in at least one embodiment, the network interface 1724 can communicate over Long-Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communication (“GSM”), IMT-CDMA Multi-Carrier (“CDMA2000”) networks, etc. In at least one embodiment, the wireless antennas 1726 can also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using local area networks such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc., and / or Low Power Wide Area Networks (“LPWANs”) such as LoRaWAN, SigFox, etc.

[0121] Such components can be used to generate an alternative image, such as a bird's-eye view image, from disparity data with limited capacity, such as an embedded processor without access to external memory.

[0122] Fig. Figure 17B shows an example of camera positions and fields of view for the autonomous vehicle 1700. Fig. 17A, according to at least one embodiment. In at least one embodiment, cameras and their respective fields of view are an example of an embodiment and are not intended to be restrictive. For example, in at least one embodiment, additional and / or alternative cameras can be installed and / or cameras can be mounted at different locations on the vehicle 1700.

[0123] In at least one embodiment, camera types may include, among others, digital cameras that can be adapted for use with components and / or systems of the Vehicle 1700. In at least one embodiment, cameras may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. In at least one embodiment, camera types may have any frame rate, such as 60 frames per second (fps), 1220 fps, 240 fps, etc. In at least one embodiment, cameras may use rolling shutter, global shutter, another shutter type, or a combination thereof.In at least one embodiment, the color filter array may comprise a red-clear-clear-clear color filter array (“RCCC”), a red-clear-clear-blue color filter array (“RCCB”), a red-blue-green-clear color filter array (“RBGC”), a Foveon X3 color filter array, a Bayer sensor color filter array (“RGGB”), a monochrome sensor color filter array, and / or another type of color filter array. In at least one embodiment, clear pixel cameras, such as cameras with an RCCC, an RCCB, and / or an RBGC color filter array, may be used to increase light sensitivity.

[0124] In at least one embodiment, one or more cameras can be used to perform functions for advanced driver assistance systems (ADAS) (e.g., as part of a redundant or fail-safe design). For example, in at least one embodiment, a multi-function monocular camera can be integrated to provide functions such as lane keeping assist, traffic sign recognition, and intelligent headlight control. In at least one embodiment, one or more cameras (e.g., all cameras) can simultaneously capture and provide image data (e.g., video).

[0125] In at least one embodiment, one or more cameras can be mounted in a mounting device, such as a specially manufactured (three-dimensionally ("3D") printed device, to suppress stray light and reflections from inside the vehicle 1700 (e.g., reflections from the dashboard reflected in the windshield mirrors) that could impair the camera's image data acquisition capabilities. With regard to the mounting devices for exterior mirrors, in at least one embodiment the exterior mirror assemblies can be individually 3D printed so that a camera mounting plate conforms to the shape of an exterior mirror. In at least one embodiment, cameras can be integrated into the exterior mirrors. In at least one embodiment, for side cameras, the cameras can also be integrated into four pillars at each corner of a cabin.

[0126] In at least one embodiment, cameras with a field of view that includes parts of the environment in front of the vehicle 1700 (e.g., forward-facing cameras) can be used for all-around vision to identify forward paths and obstacles and, with the aid of one or more controllers 1736 and / or control SoCs, provide information that is crucial for generating an occupancy grid and / or determining preferred vehicle paths. In at least one embodiment, forward-facing cameras can be used to perform many similar ADAS functions as LiDAR, including, without limitation, emergency braking, pedestrian detection, and collision avoidance.In at least one version, forward-facing cameras can also be used for ADAS functions and systems, including, without limitation, lane departure warning (“LDW”), adaptive cruise control (“ACC”) and / or other functions such as traffic sign recognition.

[0127] In at least one embodiment, various cameras can be used in a forward-facing configuration, including, for example, a monocular camera platform comprising a CMOS color imager (Complementary Metal Oxide Semiconductor). In at least one embodiment, a wide-angle camera 1770 can be used to capture objects moving into the field of view from the periphery (e.g., pedestrians, crossing traffic, or bicycles). Although in Fig. While only one wide-angle camera 1770 is shown in Figure 17B, other embodiments may include any number (including zero) of wide-angle cameras on the vehicle 1700. In at least one embodiment, any number of long-range cameras 1798 (e.g., a pair of long-range stereo cameras) may be used for depth-based object detection, particularly for objects for which no neural network has yet been trained. In at least one embodiment, long-range cameras 1798 may also be used for object detection and classification, as well as for basic object tracking.

[0128] In at least one embodiment, any number of stereo cameras 1768 can also be included in a forward-facing configuration. In at least one embodiment, one or more stereo cameras 1768 can have an integrated control unit with a scalable processing unit that can provide programmable logic (“FPGA”) and a multi-core microprocessor with an integrated controller area network (“CAN”) or an Ethernet interface on a single chip. In at least one embodiment, such a unit can be used to generate a 3D map of the vehicle 1700's environment, including a distance estimate for all points in an image.In at least one embodiment, one or more stereo cameras 1768 can, without restriction, comprise compact stereo vision sensors, which can, without restriction, include two camera lenses (one each on the left and right side) and an image processing chip that can measure the distance from the vehicle 1700 to the target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. In at least one embodiment, the following types of stereo cameras 1768 can be used in addition to or as an alternative to the other types described herein.

[0129] In at least one embodiment, cameras with a field of view that includes parts of the surroundings to the sides of the vehicle (e.g., side cameras) can be used for all-around vision to provide information used to create and update an occupancy grid and to generate side-impact warnings. For example, in at least one embodiment, surround cameras (e.g., four surround cameras, as in Fig. (as shown in Figure 17B) are positioned on the vehicle 1700. In at least one embodiment, the surround-view camera(s) 1774 can comprise any number and combination of wide-angle cameras, fisheye cameras, 360-degree cameras, and / or similar cameras without restriction. In at least one embodiment, for example, four fisheye cameras can be mounted on the front, rear, and sides of the vehicle 1700. In at least one embodiment, the vehicle 1700 can use three surround-view cameras 1774 (e.g., left, right, and rear) and one or more other cameras (e.g., a forward-facing camera) as a fourth surround-view camera.

[0130] In at least one embodiment, cameras with a field of view that includes parts of the area behind the vehicle 1700 (e.g., reversing cameras) can be used for parking assistance, all-round visibility, collision warning, and the creation and updating of an occupancy grid. In at least one embodiment, a plurality of cameras can be used, including, but not limited to, cameras that are also suitable as forward-facing cameras (e.g., long-range cameras 1798 and / or medium-range cameras 1776, stereo cameras 1768, infrared cameras 1772, etc.), as described herein.

[0131] Such components can be used to generate an alternative image, such as a bird's-eye view image, from disparity data with limited capacity, such as an embedded processor without access to external memory.

[0132] Fig. 17C is a block diagram that shows an example of a system architecture for the autonomous vehicle 1700. Fig. 17A according to at least one embodiment. In at least one embodiment, all components, features and systems of the vehicle 1700 are shown. Fig. Figure 17C is shown to be connected via a bus 1702. In at least one embodiment, the bus 1702 can, without restriction, comprise a CAN data interface (here alternatively referred to as a "CAN bus"). In at least one embodiment, a CAN can be a network within the vehicle 1700, used to support the control of various features and functions of the vehicle 1700, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, etc. In at least one embodiment, the bus 1702 can be configured to have dozens or even hundreds of nodes, each of which has its own unique identifier (e.g., a CAN ID). In at least one embodiment, the bus 1702 can be read to determine the steering wheel angle, vehicle speed, engine revolutions per minute ("RPMs"), button positions, and / or other vehicle status indicators.In at least one embodiment, the 1702 bus can be a CAN bus that is ASIL B compliant.

[0133] In at least one embodiment, FlexRay and / or Ethernet protocols can be used in addition to or as an alternative to CAN. In at least one embodiment, there can be any number of buses forming the bus 1702, including, without limitation, zero or more CAN buses, zero or more FlexRay buses, zero or more Ethernet buses, and / or zero or more other types of buses using different protocols. In at least one embodiment, two or more buses can be used to perform different functions and / or to provide redundancy. For example, a first bus can be used for the collision avoidance function and a second bus for actuation control. In at least one embodiment, each bus of the bus 1702 can communicate with any component of the vehicle 1700, and two or more buses of the bus 1702 can communicate with corresponding components.In at least one embodiment, each of any number of system(s) on chip(s) (“SoC(s)”) 1704 (such as SoC 1704(A) and SoC 1704(B)), each of the controllers 1736 and / or each computer within the vehicle can have access to the same input data (e.g. inputs from sensors of the vehicle 1700) and can be connected to a common bus, such as a CAN bus.

[0134] In at least one version, the vehicle 1700 can contain one or more controllers 1736, as described here in relation to Fig. 17A. In at least one embodiment, the controllers 1736 can be used for a variety of functions. In at least one embodiment, the controller(s) 1736 can be coupled with various other components and systems of the vehicle 1700 and used for controlling the vehicle 1700, for the artificial intelligence of the vehicle 1700, for infotainment for the vehicle 1700 and / or for other functions.

[0135] In at least one embodiment, the vehicle 1700 can contain any number of SoCs 1704. In at least one embodiment, each of the SoCs 1704 can, without restriction, contain one or more central processing units (“CPU(s)”) 1706, graphics processing units (“GPU(s)”) 1708, processors 1710, caches 1712, accelerators 1714, data storage 1716, and / or other components and features not shown. In at least one embodiment, SoCs 1704 can be used to control the vehicle 1700 in a variety of platforms and systems. For example, in at least one embodiment, the SoC 1704 can be combined in a system (e.g., the system of the vehicle 1700) with a high-resolution (“HD”) card 1722, which is accessed via a network interface 1724 by one or more servers (in Fig. (17C not shown) may receive map updates and / or updates.

[0136] In at least one implementation, the CPU (or CPUs) 1706 can comprise a CPU cluster or a CPU complex (here alternatively referred to as "CCPLEX"). In at least one implementation, the CPU (or CPUs) 1706 can comprise multiple cores and / or Level 2 caches ("L2"). For example, the CPU (or CPUs) 1706 can have eight cores in a coherent multiprocessor configuration in at least one implementation. In at least one implementation, the CPU (or CPUs) 1706 can have four dual-core clusters, each cluster having a dedicated L2 cache (e.g., a 2-megabyte L2 cache). In at least one implementation, the CPU (e.g., CCPLEX) can be configured to support concurrent cluster operations, allowing any combination of CPU clusters to be active at any given time.

[0137] In at least one version, one or more CPUs can have 1706 power management features that include, for example, one or more of the following: individual hardware blocks can be automatically turned off when not in use to save power; each core clock can be turned off when the core is not executing instructions because it is waiting for "Wait for Interrupt" ("WFT") or "Wait for Event" ("WFE"); each core can be independently power-locked; each core cluster can be independently clock-lockable when all cores are clock-lockable or power-locked; and / or each core cluster can be independently power-locked when all cores are power-locked.In at least one implementation, the CPU(s) 1706 can also implement an improved algorithm for managing power states, specifying permissible power states and expected wake-up times, with the hardware / microcode determining which power state is most appropriate for the core, cluster, and CCPLEX. In at least one implementation, the processor cores can support simplified sequences for invoking power states in software, offloading the work to the microcode.

[0138] In at least one embodiment, the GPU(s) 1708 may include an integrated GPU (hereinafter referred to as the "iGPU"). In at least one embodiment, the GPU(s) 1708 may be programmable and efficient for parallel workloads. In at least one embodiment, the GPU(s) 1708 may use an extended Tensor instruction set. In at least one embodiment, the GPU(s) 1708 may include one or more streaming microprocessors, each streaming microprocessor including a Level 1 cache ("L1 cache") (e.g., an L1 cache with at least 96 KB of storage capacity), and two or more streaming microprocessors may share an L2 cache (e.g., an L2 cache with a storage capacity of 512 KB). In at least one embodiment, the GPU(s) 1708 may have at least eight streaming microprocessors.In at least one configuration, the GPU(s) 1708 can use one or more application programming interfaces (APIs). In at least one configuration, the GPU(s) 1708 can use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA model).

[0139] In at least one embodiment, one or more of the GPUs 1708 can be performance-optimized for optimal performance in automotive and embedded applications. For example, in at least one embodiment, the GPU(s) 1708 could be manufactured on Fin field-effect transistor (“FinFET”) circuits. In at least one embodiment, each streaming microprocessor can contain a number of mixed-precision processing cores divided into multiple blocks. For example, but not limited to, 64 PF32 cores and 32 PF64 cores could be divided into four processing blocks. In at least one embodiment, each processing block could be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor cores for deep learning matrix arithmetic, a level zero instruction cache (“L0”), a scheduler (e.g., warp scheduler) or sequencer, a dispatch unit, and / or a 64 KB register file.In at least one embodiment, streaming microprocessors can include independent parallel integer and floating-point data paths to enable efficient execution of workloads with a mix of computations and address calculations. In at least one embodiment, streaming microprocessors can include independent thread scheduling capability to enable finer synchronization and collaboration between parallel threads. In at least one embodiment, streaming microprocessors can include a combined L1 data cache and a shared memory unit to improve performance while simplifying programming.

[0140] In at least one embodiment, one or more GPUs 1708 can include high-bandwidth memory (“HBM”) and / or a 16 GB HBM2 memory subsystem to provide a maximum memory bandwidth of approximately 900 GB / second in some examples. In at least one embodiment, in addition to or as an alternative to the HBM memory, synchronous graphics random-access memory (“SGRAM”), such as Graphics Double Data Rate Type Five synchronous random-access memory (“GDDR5”), can be used.

[0141] In at least one embodiment, the GPU(s) 1708 can incorporate a unified memory technology. In at least one embodiment, an Address Translation Service (“ATS”) can be used to enable the GPU(s) 1708 of “ ” to directly access the page tables of the CPU(s) 1706. In at least one embodiment, if a GPU of the GPU(s) 1708 memory management unit (“MMU”) experiences a mismatch, an address translation request can be transmitted to the CPU(s) 1706. In response, the CPU 2 of the CPU(s) 1706 can search its page tables for a virtual-to-physical mapping for an address and send the translation back to the GPU(s) 1708, at least in one embodiment.In at least one embodiment, the unified memory technology can enable a single unified virtual address space for the memory of both the CPU(s) 1706 and the GPU(s) 1708, thereby simplifying the programming of the GPU(s) 1708 and the porting of applications to the GPU(s) 1708.

[0142] In at least one embodiment, the GPU(s) 1708 can include any number of access counters that can track the frequency of GPU(s) 1708 access to the memory of other processors. In at least one embodiment, the access counters can help ensure that memory pages are moved to the physical memory of a processor that accesses pages most frequently, thereby improving the efficiency of memory areas shared between processors.

[0143] In at least one embodiment, one or more SoCs 1704 can contain any number of caches 1712, including those described here. For example, in at least one embodiment, the cache(s) 1712 could contain a Level 3 cache (“L3”) that is available to both the CPU(s) 1706 and the GPU(s) 1708 (e.g., one connected to both the CPU(s) 1706 and the GPU(s) 1708). In at least one embodiment, the cache(s) 1712 can include a write-back cache capable of tracking the state of lines, for example, using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). In at least one embodiment, an L3 cache can comprise 4 MB of memory or more, depending on the embodiment, although smaller cache sizes can also be used.

[0144] In at least one embodiment, one or more of the SoCs 1704 can include one or more accelerators 1714 (e.g., hardware accelerators, software accelerators, or a combination thereof). In at least one embodiment, the SoC(s) 1704 can include a hardware acceleration cluster, which may include optimized hardware accelerators and / or a large on-chip memory. In at least one embodiment, a large on-chip memory (e.g., 4 MB SRAM) can enable a hardware acceleration cluster to accelerate neural networks and other computations. In at least one embodiment, a hardware acceleration cluster can be used to complement the GPU(s) 1708 and offload some of the GPU(s) 1708's tasks (e.g., to free up more GPU 1708 cycles for performing other tasks). In at least one embodiment, accelerators 1714 could be used for specific workloads (e.g.,Perception, convolutional neural networks (“CNNs”), recurrent neural networks (“RNNs”), etc.) are used that are stable enough to be suitable for acceleration. In at least one embodiment, a CNN may include a region-based or regional convolutional neural network (“RCNN”) and fast RCNNs (e.g., as used for object recognition) or another type of CNN.

[0145] In at least one embodiment, the accelerator(s) 1714 (e.g., hardware acceleration cluster) can comprise one or more deep learning accelerators (“DLA”). In at least one embodiment, the DLA(s) can, without limitation, comprise one or more tensor processing units (“TPUs”), which can be configured to provide an additional ten trillion operations per second for deep learning applications and inference. In at least one embodiment, TPUs can be accelerators configured and optimized for performing image processing functions (e.g., for CNNs, RCNNs, etc.). In at least one embodiment, the DLA(s) can further be optimized for a specific set of neural network types and floating-point operations, as well as for inference.In at least one implementation, the design of DLA(s) can offer more performance per millimeter than a typical general-purpose GPU and generally far surpasses the performance of a CPU. In at least one implementation, TPU(s) can perform multiple functions, including single-instance folding that supports, for example, INT8, INT16, and FP16 data types for both features and weights, as well as post-processing functions.In at least one embodiment, the DLA can execute neural networks, in particular CNNs, quickly and efficiently on processed or unprocessed data for a variety of functions, including, for example, but not limited to: a CNN for object identification and recognition using data from camera sensors; a CNN for distance estimation using data from camera sensors; a CNN for the detection and identification of emergency vehicles using data from microphones; a CNN for facial recognition and identification of vehicle owners using data from camera sensors; and / or a CNN for safety-related events.

[0146] In at least one embodiment, DLA(s) can execute any function of GPU(s) 1708, and by using an inference accelerator, a developer can, for example, select either DLA(s) or GPU(s) 1708 for each function. In at least one embodiment, a developer can, for example, concentrate the processing of CNNs and floating-point operations on DLA(s) and leave other functions to GPU(s) 1708 and / or accelerator(s) 1714.

[0147] In at least one embodiment, the accelerators 1714 may comprise a programmable image processing accelerator (“PVA”), which may alternatively be referred to herein as a computer vision accelerator. In at least one embodiment, the PVA may be designed and configured to accelerate computer vision algorithms for an advanced driver assistance system (“ADAS”) 1738, autonomous driving, augmented reality (“AR”) applications, and / or virtual reality (“VR”) applications. In at least one embodiment, the PVA may provide a balance between performance and flexibility. For example, in at least one embodiment, each PVA may, for example, without limitation, have any number of RISC (Reduced Instruction Set Computer) cores, DMA (Direct Memory Access) cores, and / or any number of vector processors.

[0148] In at least one embodiment, RISC cores can interact with image sensors (e.g., image sensors of any cameras described herein), image signal processors, etc. In at least one embodiment, each RISC core can include any amount of memory. In at least one embodiment, RISC cores can use one of several protocols, depending on the implementation. In at least one embodiment, RISC cores can run a real-time operating system (RTOS). In at least one embodiment, RISC cores can be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, in at least one embodiment, RISC cores could include an instruction cache and / or tightly coupled RAM.

[0149] In at least one embodiment, DMA can enable components of a PVA to access system memory independently of the CPU(s) 1706. In at least one embodiment, DMA can support any number of functions used to optimize a PVA, including, but not limited to, support for multidimensional addressing and / or circular addressing. In at least one embodiment, DMA can support up to six or more addressing dimensions, including, but not limited to, block width, block height, block depth, horizontal block spacing, vertical block spacing, and / or depth spacing.

[0150] In at least one embodiment, vector processors can be programmable processors designed to efficiently and flexibly execute computer vision algorithm programming and provide signal processing capabilities. In at least one embodiment, a PVA can comprise a PVA core and two vector processing subsystem partitions. In at least one embodiment, a PVA core can comprise a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. In at least one embodiment, a vector processing subsystem can function as the primary processing engine of a PVA and comprise a vector processing unit (“VPU”), an instruction cache, and / or a vector memory (e.g., “VMEM”).In at least one embodiment, the VPU core can include a digital signal processor, such as a single-instruction, multi-data (“SIMD”), and very-long instruction word (“VLIW”) digital signal processor. In at least one embodiment, a combination of SIMD and VLIW can improve throughput and speed.

[0151] In at least one version, each vector processor can have an instruction cache and be connected to its own memory. Therefore, in at least one version, each vector processor can be configured to run independently of the other vector processors. In at least one version, the vector processors belonging to a particular PVA can be configured to utilize data parallelism. For example, in at least one embodiment, multiple vector processors contained in a single PVA can execute a common computer vision algorithm, but on different regions of an image. In at least one embodiment, vector processors contained in a particular PVA can simultaneously execute different computer vision algorithms on an image, or even execute different algorithms on successive images or parts of an image.In at least one embodiment, an arbitrary number of PVAs can be contained in a hardware acceleration cluster, and an arbitrary number of vector processors can be contained in each PVA. In at least one embodiment, the PVA can include an additional error correction code (“ECC”) memory to improve the overall security of the system.

[0152] In at least one embodiment, the accelerator(s) 1714 can include an on-chip computer vision network and static random-access memory (“SRAM”) to provide high-bandwidth, low-latency SRAM for the accelerator(s). In at least one embodiment, the on-chip memory can comprise at least 4 MB of SRAM, which, for example, and without limitation, includes eight field-configurable memory blocks accessible to both a PVA and a DLA. In at least one embodiment, each pair of memory blocks can include an extended peripheral bus interface (“APB”), a configuration circuit, a controller, and a multiplexer. In at least one embodiment, any type of memory can be used. In at least one embodiment, a PVA and a DLA can access the memory via a backbone that enables fast access to the memory for both the PVA and the DLA.In at least one embodiment, a backbone can include an on-chip computer vision network that connects a PVA and a DLA to the memory (e.g., using APB).

[0153] In at least one embodiment, an on-chip computer vision network may include an interface that, prior to the transmission of a control signal / address / data, determines that both a PVA and a DLA provide ready and valid signals. In at least one embodiment, an interface may provide separate phases and channels for the transmission of control signals / addresses / data, as well as burst communication for continuous data transmission. In at least one embodiment, an interface may conform to the standards ISO 26262 of the International Organization for Standardization (“ISO”) or IEC 61508 of the International Electrotechnical Commission (“IEC”), although other standards and protocols may also be used.

[0154] In at least one embodiment, one or more of the SoCs 1704 can include a real-time ray tracing hardware accelerator. In at least one embodiment, a real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the positions and extents of objects (e.g., within a world model), to generate real-time visualization simulations, to interpret radar signals, to synthesize and / or analyze sound propagation, to simulate SONAR systems, for general wave propagation simulation, to compare with lidar data for localization purposes and / or other functions, and / or for other uses.

[0155] In at least one embodiment, the accelerator(s) 1714 can have a variety of uses for autonomous driving. In at least one embodiment, a PVA can be used for key processing stages in ADAS and autonomous vehicles. In at least one embodiment, the capabilities of a PVA are well suited for algorithmic domains that require predictable processing with low power consumption and low latency. In other words, a PVA performs well in semi-dense or dense regular computations, even with small datasets, which might require predictable runtimes with low latency and low power consumption. In at least one embodiment, such as in vehicle 1700, PVAs could be designed to execute classical computer vision algorithms, as they can be efficient in object detection and integer mathematics processing.

[0156] According to at least one embodiment of the technology, a PVA is used, for example, to perform computer stereo vision. In at least one embodiment, a semi-global, matching-based algorithm may be used in some examples, although this is not intended to be restrictive. In at least one embodiment, applications for Level 3 to 5 autonomous driving utilize real-time motion estimation / stereo matching (e.g., structure of motion, pedestrian detection, lane detection, etc.). In at least one embodiment, a PVA can perform computer stereo vision functions for input from two monocular cameras.

[0157] In at least one embodiment, a PVA can be used to perform a dense optical flow. For example, in at least one embodiment, a PVA could process raw RADAR data (e.g., using a 4D Fast Fourier Transform) to provide processed RADAR data. In at least one embodiment, a PVA is used for time-of-flight depth processing, for example, by processing raw time-of-flight data to provide processed time-of-flight data.

[0158] In at least one embodiment, a DLA can be used to operate any type of network to improve control and driving safety, including, for example, but not limited to, a neural network that outputs a measure of the reliability of each object detection. In at least one embodiment, the reliability can be represented or interpreted as a probability or as the relative "weight" of each detection compared to other detections. In at least one embodiment, a confidence measure enables a system to make further decisions about which detections should be considered true positives rather than false positives. In at least one embodiment, a system can set a confidence threshold and consider only detections that exceed the threshold as true positives.In an implementation using an automatic emergency braking system (“AEB”), false positive detections would cause the vehicle to automatically initiate emergency braking, which is obviously undesirable. In at least one implementation, high-confidence detections can be considered triggers for AEB. In at least one implementation, a DLA can execute a neural network to regress the confidence value. In at least one implementation, the neural network can take at least a subset of parameters as input, such as boundary frame dimensions, a ground plane estimate (e.g., obtained from another subsystem), the output of IMU sensors 1766 correlated with the vehicle's orientation 1700, the distance, 3D position estimates of objects obtained by the neural network and / or other sensors (e.g., LIDAR sensor(s) 1764 or RADAR sensor(s) 1760), among others.

[0159] In at least one embodiment, one or more of the SoCs 1704 can include data storage devices 1716 (e.g., memory). In at least one embodiment, the data storage devices 1716 can be on-chip memory of the SoC(s) 1704 that can store neural networks to be executed on GPU(s) 1708 and / or a DLA. In at least one embodiment, the data storage devices 1716 can have sufficient capacity to store multiple instances of neural networks for redundancy and security reasons. In at least one embodiment, the data storage device(s) 1716 can include an L2 or L3 cache.

[0160] In at least one embodiment, one or more of the SoCs 1704 can comprise any number of processors 1710 (e.g., embedded processors). In at least one embodiment, the processors 1710 can comprise a boot and power management processor, which can be a dedicated processor and a dedicated subsystem for handling boot power and management functions and the associated security enforcement. In at least one embodiment, a boot and power management processor can be part of a boot sequence of SoCs 1704 and provide runtime power management services. In at least one embodiment, a boot and power management processor can provide clock and voltage programming, support for system power-saving transitions, management of the thermals and temperature sensors of SoC(s) 1704, and / or management of the power states of SoC(s) 1704.In at least one embodiment, each temperature sensor can be implemented as a ring oscillator whose output frequency is proportional to the temperature, and SoC(s) 1704 can use ring oscillators to detect temperatures of CPU(s) 1706, GPU(s) 1708, and / or accelerator(s) 1714. In at least one embodiment, when a boot and power management processor detects that the temperatures exceed a threshold, it can invoke a temperature fault routine and put SoC(s) 1704 into a lower power state and / or put vehicle 1700 into a chauffeur-to-safety-stop mode (e.g., bring vehicle 1700 to a safe stop).

[0161] In at least one embodiment, the processor(s) 1710 may further comprise a set of embedded processors that can serve as an audio processing engine, which may be an audio subsystem that enables full hardware support for multi-channel audio over multiple interfaces and a wide and flexible range of audio I / O interfaces. In at least one embodiment, an audio processing engine is a dedicated processor core with a digital signal processor and dedicated RAM.

[0162] In at least one embodiment, the processor(s) 1710 may also include a continuously active processor engine that can provide the necessary hardware functions to support low-power use cases for sensor management and wake-up. In at least one embodiment, a continuously active processor engine may, without limitation, include a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0163] In at least one embodiment, the 1710 processor(s) may also include a security cluster engine, which without limitation comprises a dedicated processor subsystem for handling security management for automotive applications. In at least one embodiment, a security cluster engine may, without limitation, include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, an interrupt controller, etc.), and / or routing logic. In a security mode, in at least one embodiment, two or more cores may operate in a lockstep mode and function as a single core with comparison logic to detect differences between their operations. In at least one embodiment, the 1710 processor(s) may also include a real-time camera engine, which, without limitation, may include a dedicated processor subsystem for managing real-time camera operations.In at least one embodiment, the processor(s) 1710 may also include a high dynamic range signal processor, which may without limitation include an image signal processor that is a hardware engine that is part of a camera processing pipeline.

[0164] In at least one embodiment, the processor(s) 1710 may include a video image compositor, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to generate a final image for a player window. In at least one embodiment, a video image compositor may perform lens distortion correction on wide-angle cameras 1770, surround cameras 1774, and / or on indoor surveillance camera sensors. In at least one embodiment, indoor surveillance camera sensors are preferably monitored by a neural network running on another instance of SoC 1704 and configured to identify indoor events and respond accordingly.In at least one embodiment, a cabin system can perform unrestricted lip reading to activate mobile phone service and make a call, dictate emails, change a vehicle's destination, activate or change a vehicle's infotainment system and settings, or enable voice-controlled internet browsing. In at least one embodiment, certain functions are available to the driver when a vehicle is operating in autonomous mode and are otherwise deactivated.

[0165] In at least one version, a video image compositor can have improved temporal noise reduction for both spatial and temporal noise reduction. For example, in at least one implementation where motion occurs in a video, the noise reduction can appropriately weight spatial information and reduce the weighting of information from adjacent frames. In at least one implementation where a frame or part of a frame does not contain motion, the temporal noise reduction performed by the video image compositor can use information from a previous frame to reduce noise in the current frame.

[0166] In at least one embodiment, a video image compositor can also be configured to perform stereo correction on input stereo lens frames. In at least one embodiment, a video image compositor can also be used for user interface composition when an operating system desktop is used and the GPU(s) 1708 do not need to continuously render new surfaces. In at least one embodiment, a video image compositor can be used to offload the GPU(s) 1708, thereby improving performance and responsiveness when the GPU(s) 1708 are powered on and actively performing 3D rendering.

[0167] In at least one embodiment, one or more SoCs of SoCs 1704 may also include a mobile industrial processor interface (“MIPI”) with a serial camera interface for receiving video and camera inputs, a high-speed interface, and / or a video input block that can be used for a camera and associated pixel input functions. In at least one embodiment, one or more SoCs ( ) 1704 may also include one or more input / output controllers that can be controlled by software and used to receive I / O signals that are not assigned to a specific role.

[0168] In at least one embodiment, one or more SoCs of SoCs 1704 can also include a wide range of peripheral interfaces to enable communication with peripheral devices, audio codecs (“codecs”), power management, and / or other devices. In at least one embodiment, the SoC 1704 can be used to process data from cameras (e.g., connected via Gigabit multimedia serial link and Ethernet channels), sensors (e.g., LiDAR sensors 1764, radar sensors 1760, etc., which can be connected via Ethernet channels), data from the bus 1702 (e.g., vehicle speed 1700, steering wheel position, etc.), data from GNSS sensors 1754, etc. etc., which may be connected via Ethernet channels), data from bus 1702 (e.g. vehicle speed 1700, steering wheel position etc.), data from GNSS sensors 1758 (e.g. connected via an Ethernet bus or a CAN bus) etc.In at least one configuration, one or more SoC(s) 1704 may also have special high-performance mass storage controllers, which may have their own DMA engines and which can be used to relieve CPU(s) 1706 of routine data management tasks.

[0169] In at least one embodiment, the SoC(s) 1704 can be an end-to-end platform with a flexible architecture that covers automation levels 3-5, thereby providing a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS techniques for diversity and redundancy, and provides a platform for a flexible, reliable driving software stack along with deep learning tools. In at least one embodiment, the SoC(s) 1704 can be faster, more reliable, and even more energy- and space-efficient than conventional systems. For example, in at least one embodiment, accelerators 1714, in combination with CPUs 1706, GPUs 1708, and data storage devices 1716, can provide a fast, efficient platform for autonomous vehicles of levels 3 to 5.

[0170] In at least one embodiment, computer vision algorithms can be executed on CPUs that can be configured using a high-level programming language, such as C, to run a variety of processing algorithms on a variety of visual data. However, in at least one embodiment, CPUs are often unable to meet the performance requirements of many computer vision applications, for example, with regard to execution time and power consumption. In at least one embodiment, many CPUs are unable to execute complex object recognition algorithms in real time, as used in in-vehicle ADAS applications and in practical Level 3 to 5 autonomous vehicles.

[0171] The embodiments described here make it possible to execute multiple neural networks simultaneously and / or sequentially and to combine the results to enable autonomous driving functions of levels 3 to 5. In at least one embodiment, a CNN running on a DLA or a discrete GPU (e.g., GPU(s) 1720) can, for example, include text and word recognition that enables the reading and understanding of traffic signs, including signs for which a neural network has not been specifically trained. In at least one embodiment, a DLA can also include a neural network capable of identifying, interpreting, and semantically understanding a sign and passing this semantic understanding to path planning modules running on a CPU complex.

[0172] In at least one embodiment, multiple neural networks can be executed simultaneously, for example, for driving at levels 3, 4, or 5. In at least one embodiment, for instance, a warning sign with the inscription "Caution: Flashing lights indicate icy roads" along with an electric light can be interpreted independently or jointly by multiple neural networks. In at least one embodiment, such a warning sign itself can be identified as a traffic sign by a first neural network (e.g., a trained neural network), and the text "Flashing lights indicate icy roads" can be interpreted by a second neural network, which informs the vehicle's path planning software (preferably running on a CPU complex) that icy road conditions are present when flashing lights are detected.In at least one embodiment, a flashing light can be identified across multiple images by operating a third neural network, thereby informing a vehicle's route planning software about the presence (or absence) of flashing lights. In at least one embodiment, all three neural networks can run simultaneously, for example, within a DLA and / or on one or more GPUs 1708.

[0173] In at least one embodiment, a CNN for facial recognition and vehicle owner identification can use data from camera sensors to detect the presence of an authorized driver and / or owner of the vehicle 1700. In at least one embodiment, a continuously active sensor processing engine can be used to unlock a vehicle when an owner approaches the driver's door and turns on the lights, and to disable such a vehicle in a security mode when an owner leaves it. In this way, the SoC(s) 1704 provides security against theft and / or car robbery.

[0174] In at least one embodiment, a CNN for the detection and identification of emergency vehicles can use data from microphones 1796 to detect and identify the sirens of emergency vehicles. In at least one embodiment, SoC(s) 1704 use a CNN for classifying ambient and urban noise as well as for classifying visual data. In at least one embodiment, a CNN running on a DLA is trained to identify the relative approach speed of an emergency vehicle (e.g., using a Doppler effect). In at least one embodiment, a CNN can also be trained to identify emergency vehicles specific to a local area in which a vehicle operates, as identified by GNSS sensors 1758.In at least one embodiment, a CNN, when operating in Europe, attempts to detect European sirens, and in North America, a CNN attempts to identify only North American sirens. In at least one embodiment, once an emergency vehicle is detected, a control program can be used to execute an emergency vehicle safety routine, whereby a vehicle is slowed down, moved to the side of the road, parked, and / or idled, with the aid of ultrasonic sensors 1762, until the emergency vehicles have passed.

[0175] In at least one embodiment, the vehicle 1700 may include one or more CPUs 1718 (e.g., discrete CPUs or dCPUs) which may be coupled to one or more SoCs 1704 via a high-speed connection (e.g., PCIe). In at least one embodiment, the CPU(s) 1718 may, for example, comprise an x86 processor. The CPU(s) 1718 may be used to perform a variety of functions, including mediating potentially inconsistent results between ADAS sensors and SoC(s) 1704 and / or monitoring the status and integrity of controller(s) 1736 and / or an infotainment system on a single chip (“infotainment SoC”) 1730. In at least one embodiment, the SoC 1704 has one or more connections, and one connection may be a PCIe connection.

[0176] In at least one embodiment, the vehicle 1700 can include one or more GPUs 1720 (e.g., discrete GPUs or dGPUs) that can be coupled to the SoC(s) 1704 via a high-speed connection (e.g., NVIDIA's NVLINK channel). In at least one embodiment, the GPU(s) 1720 can provide additional artificial intelligence functions, for example, by running redundant and / or different neural networks, and can be used to train and / or update neural networks at least partially based on inputs (e.g., sensor data) from sensors of a vehicle 1700.

[0177] In at least one embodiment, the vehicle 1700 may further comprise a network interface 1724, which may, without limitation, include one or more wireless antennas 1726 (e.g., one or more wireless antennas for various communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). In at least one embodiment, the network interface 1724 may be used to establish a wireless connection to internet cloud services (e.g., with servers and / or other network devices), with other vehicles, and / or with computer devices (e.g., passenger client devices). In at least one embodiment, communication with other vehicles may be established via a direct connection between the vehicle 1700 and another vehicle and / or an indirect connection (e.g., via networks and the internet).In at least one embodiment, direct connections can be provided using a vehicle-to-vehicle communication link. In at least one embodiment, a vehicle-to-vehicle communication link can provide the vehicle 1700 with information about vehicles in the vicinity of the vehicle 1700 (e.g., vehicles in front of, beside, and / or behind the vehicle 1700). In at least one embodiment, the aforementioned functionality can be part of a cooperative adaptive cruise control functionality of the vehicle 1700.

[0178] In at least one embodiment, the network interface 1724 can include a SoC that provides modulation and demodulation functionality and enables the controller(s) 1736 to communicate over wireless networks. In at least one embodiment, the network interface 1724 can include a high-frequency front end for up-conversion from the baseband frequency to the high frequency and down-conversion from the high frequency to the baseband frequency. In at least one embodiment, frequency conversions can be performed in any technically feasible way. For example, frequency conversions could be performed by known methods and / or using superheterodyne methods. In at least one embodiment, the high-frequency front-end functionality can be provided by a separate chip.In at least one embodiment, network interfaces can include radio functionality for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN and / or other radio protocols.

[0179] In at least one embodiment, the vehicle 1700 may also include one or more data storage devices 1728, which may, without limitation, include off-chip memory (e.g., outside of SoC(s) 1704). In at least one embodiment, the data storage devices 1728 may, without limitation, include one or more memory elements, including RAM, SRAM, dynamic random-access memory (“DRAM”), video random-access memory (“VRAM”), flash memory, hard disks, and / or other components and / or devices capable of storing at least one data bit.

[0180] In at least one embodiment, the vehicle 1700 may also include GNSS sensors 1758 (e.g., GPS and / or assisted GPS sensors) to support functions such as mapping, perception, occupancy grid generation, and / or path planning. In at least one embodiment, any number of GNSS sensors 1758 may be used, including, for example, and without limitation, a GPS that uses a USB port with an Ethernet-to-serial bridge (e.g., RS-232).

[0181] In at least one version, the vehicle 1700 can also have RADAR sensors 1760. In at least one version, the RADAR sensors 1760 can be used by the vehicle 1700 to detect other vehicles from a distance, even in the dark and / or in bad weather. In at least one embodiment, the functional safety level of the RADAR can be ASIL B. In at least one embodiment, the RADAR sensors 1760 can use a CAN bus and / or bus 1702 (e.g., for transmitting data generated by the RADAR sensors 1760) to take over control and access object tracking data, with some examples involving access to Ethernet channels to access raw data. In at least one embodiment, a variety of RADAR sensor types can be used. For example, and without limitation, RADAR sensors 1760 can be suitable for use as front, rear, and side radar.In at least one version, one or more of the RADAR sensors 1760 are a pulse Doppler RADAR sensor.

[0182] In at least one embodiment, RADAR sensors 1760 can have different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range with side coverage, etc. In at least one embodiment, a long-range RADAR can be used for adaptive cruise control. In at least one embodiment, long-range RADAR systems can provide a wide field of view, achieved through two or more independent scans, for example, within a range of 250 m (meters). In at least one embodiment, RADAR sensors 1760 can help distinguish between stationary and moving objects and can be used by the ADAS system 1738 for emergency braking assistance and forward collision warning.In at least one embodiment, the sensors 1760 included in a long-range radar system can, without restriction, comprise a monostatic multimodal radar with multiple (e.g., six or more) fixed radar antennas and a high-speed CAN and FlexRay interface. In at least one embodiment with six antennas, four central antennas can generate a focused beam pattern designed to record the area around the vehicle 1700 at higher speeds with minimal interference from traffic in adjacent lanes. In at least one embodiment, two additional antennas can extend the field of view, enabling the rapid detection of vehicles entering or exiting the vehicle 1700's lane.

[0183] In at least one embodiment, medium-range radar systems can, for example, have a range of up to 160 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 150 degrees (rear). In at least one embodiment, short-range radar systems can, without limitation, include any number of radar sensors 1760 designed for installation at both ends of a rear bumper. When installed at both ends of a rear bumper, a radar sensor system in at least one embodiment can generate two beams that continuously monitor the blind spot in the reverse direction and alongside a vehicle. In at least one embodiment, short-range radar systems can be used in an ADAS system 1738 for blind spot detection and / or lane change assistance.

[0184] In at least one embodiment, the vehicle 1700 can also have ultrasonic sensors 1762. In at least one embodiment, ultrasonic sensors 1762, which can be mounted at the front, rear, and / or sides of the vehicle 1700, can be used for parking assistance and / or for creating and updating an occupancy grid. In at least one embodiment, a plurality of ultrasonic sensors 1762 can be used, and different ultrasonic sensors 1762 can be used for different detection ranges (e.g., 2.5 m, 4 m). In at least one embodiment, the ultrasonic sensors 1762 can be operated with functional safety levels of ASIL B.

[0185] In at least one embodiment, the vehicle can include 1700 LIDAR sensors 1764. In at least one embodiment, the LIDAR sensors 1764 can be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. In at least one embodiment, the LIDAR sensors 1764 can operate at functional safety level ASIL B. In at least one embodiment, the vehicle can include multiple LIDAR sensors 1764 (e.g., two, four, six, etc.) that can use an Ethernet channel (e.g., to deliver data to a Gigabit Ethernet switch).

[0186] In at least one embodiment, the LIDAR sensors 1764 can provide a list of objects and their distances for a 360-degree field of view. In at least one embodiment, commercially available LIDAR sensors 1764 can have a specified range of approximately 100 m with an accuracy of 2 cm to 3 cm and, for example, support a 100 Mbps Ethernet connection. In at least one embodiment, one or more non-protruding LIDAR sensors can be used. In such an embodiment, the LIDAR sensors 1764 can comprise a small device that can be embedded in a front, rear, side, and / or corner position of the vehicle 1700. In at least one embodiment, the LIDAR sensors 1764 in such a configuration can provide a horizontal field of view of up to 120 degrees and a vertical field of view of up to 35 degrees with a range of 200 m, even for objects with low reflectivity.In at least one version, the front-mounted LIDAR sensors 1764 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0187] In at least one embodiment, LIDAR technologies such as 3D flash LIDAR can also be used. In at least one embodiment, 3D flash LIDAR uses a laser pulse as a transmission source to illuminate the area around the vehicle 1700 up to a distance of approximately 200 m. In at least one embodiment, a flash LIDAR unit without limitation includes a receiver that records the laser pulse travel time and the reflected light on each pixel, which in turn corresponds to an area from the vehicle 1700 to objects. In at least one embodiment, flash LIDAR can generate highly precise and distortion-free images of the environment with each laser pulse. In at least one embodiment, four flash LIDAR sensors can be used, one on each side of the vehicle 1700. In at least one embodiment, 3D flash LIDAR systems include, among other things, a solid-state 3D staring array LIDAR camera without moving parts except for a fan (e.g., a fan).(a non-scanning device). In at least one embodiment, the Flash LIDAR device can use a 5-nanosecond Class I (eye-safe) laser pulse per image and capture reflected laser light as a 3D distance point cloud and co-registered intensity data.

[0188] In at least one embodiment, the vehicle 1700 may also have IMU sensors 1766. In at least one embodiment, the IMU sensors 1766 may be located in the center of a rear axle of the vehicle 1700. In at least one embodiment, the IMU sensors 1766 may, for example, and without limitation, include accelerometers, magnetometers, gyroscopes, a magnetic compass, magnetic compasses, and / or other sensor types. In at least one embodiment, for example, in six-axis applications, the IMU sensor(s) 1766 may, without limitation, include accelerometers and gyroscopes. In at least one embodiment, for example, in nine-axis applications, the IMU sensor(s) 1766 may, without limitation, include accelerometers, gyroscopes, and magnetometers.

[0189] In at least one embodiment, IMU sensors 1766 can be implemented as a miniaturized, high-performance GPS-based inertial navigation system (“GPS / INS”) that combines microelectromechanical systems (“MEMS”) inertial sensors, a high-sensitivity GPS receiver, and advanced Kalman filter algorithms to provide estimates of position, velocity, and orientation. In at least one embodiment, IMU sensors 1766 can enable the vehicle 1700 to estimate its course without requiring input from a magnetic sensor by observing and correlating changes in velocity directly from a GPS to IMU sensors 1766. In at least one embodiment, IMU sensor(s) 1766 and GNSS sensor(s) 1758 can be combined in a single integrated unit.

[0190] In at least one embodiment, the vehicle 1700 can include one or more microphones 1796 arranged in and / or around the vehicle 1700. In at least one embodiment, the microphones 1796 can be used, among other things, for the detection and identification of emergency vehicles.

[0191] In at least one embodiment, the vehicle 1700 may also include any number of camera types, including stereo cameras 1768, wide-angle cameras 1770, infrared cameras 1772, surround cameras 1774, long-range cameras 1798, medium-range cameras 1776, and / or other camera types. In at least one embodiment, cameras may be used to capture image data around the entire perimeter of the vehicle 1700. In at least one embodiment, the specific camera types used depend on the vehicle 1700. In at least one embodiment, any combination of camera types may be used to provide the required coverage around the vehicle 1700. In at least one embodiment, the number of cameras used may vary depending on the embodiment. For example, the vehicle 1700 in at least one embodiment could have six cameras, seven cameras, ten cameras, twelve cameras, or any other number of cameras.In at least one embodiment, the cameras can support, for example, Gigabit Multimedia Serial Link (“GMSL”) and / or Gigabit Ethernet communication. In at least one embodiment, each camera can, as before, with regard to… Fig. 17A and Fig. 17B will be described in more detail.

[0192] In at least one embodiment, the vehicle 1700 may also have one or more vibration sensors 1742. In at least one embodiment, the vibration sensors 1742 may measure vibrations of parts of the vehicle 1700, such as axles. In at least one embodiment, for example, changes in the vibrations may indicate a change in the road surface. In at least one embodiment, when using two or more vibration sensors 1742, differences between the vibrations may be used to determine the friction or slip of the road surface (e.g., if there is a difference in vibration between a driven axle and a freely rotating axle).

[0193] In at least one embodiment, the vehicle 1700 can have an ADAS system 1738. In at least one embodiment, the ADAS system 1738 can have a SoC in some examples without restriction. In at least one embodiment, the ADAS system 1738 can have any number and combination of an autonomous / adaptive / automatic cruise control system (“ACC”), a cooperative adaptive cruise control system (“CACC”), a forward collision warning system (“FCW”), an automatic emergency braking system (“AEB”), a lane keeping assist system (“In at least one embodiment, the ACC system can use radar sensors (1760), lidar sensors (1764), and / or any number of cameras).

[0194] In at least one embodiment, the ACC system can use RADAR sensors 1760, LIDAR sensors 1764, and / or any number of cameras. In at least one embodiment, the ACC system can include a longitudinal ACC system and / or a lateral ACC system. In at least one embodiment, a longitudinal ACC system monitors and controls the distance to another vehicle immediately in front of the vehicle 1700 and automatically adjusts the speed of the vehicle 1700 to maintain a safe distance from vehicles ahead. In at least one embodiment, a lateral ACC system ensures that the distance is maintained and, if necessary, prompts the vehicle 1700 to change lanes. In at least one embodiment, a lateral ACC system is integrated with other ADAS applications, such as LC and CW.

[0195] In at least one embodiment, a CACC system uses information from other vehicles that can be received via a network interface 1724 and / or one or more radio antennas 1726 from other vehicles via a radio link or indirectly via a network connection (e.g., via the Internet). In at least one embodiment, direct connections can be provided by a vehicle-to-vehicle communication link (“V2V”), while indirect connections can be provided by an infrastructure-to-vehicle communication link (“I2V”). In general, V2V communication provides information about vehicles immediately ahead (e.g., vehicles that are directly in front of the vehicle 1700 and in the same lane), while I2V communication provides information about traffic further ahead.In at least one embodiment, a CACC system can include either I2V or V2V information sources, or both. In at least one embodiment, a CACC system can be more reliable given the information about vehicles ahead of the vehicle (1700) and has the potential to improve traffic flow and reduce congestion on the road.

[0196] In at least one version, an FCW system is designed to alert the driver to a hazard so they can take action. In at least one version, an FCW system uses a forward-facing camera and / or RADAR 1760 sensors connected to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically connected to provide feedback to the driver, for example, via a display, a speaker, and / or a vibrating component. In at least one version, an FCW system can issue a warning, for example, in the form of an audible signal, a visual warning, a vibration, and / or a brief braking pulse.

[0197] In at least one embodiment, an AEB system detects an impending frontal collision with another vehicle or object and can brake automatically if the driver does not take corrective action within a certain time or distance parameter. In at least one embodiment, the AEB system can use one or more forward-facing cameras and / or radar sensors connected to a dedicated processor, DSP, FPGA, and / or ASIC. In at least one embodiment, when an AEB system detects a hazard, it typically first alerts the driver so that they can take corrective action to avoid a collision. If the driver fails to take corrective action, the AEB system can automatically apply the brakes to prevent a predicted collision or at least mitigate its effects.In at least one embodiment, an AEB system may include techniques such as dynamic brake support and / or emergency braking in the event of an impending collision.

[0198] In at least one embodiment, an LDW system provides visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to warn the driver if the vehicle crosses the lane markings. In at least one embodiment, an LDW system is not activated if the driver signals an intentional lane departure, for example, by activating the turn signal. In at least one embodiment, an LDW system may use forward-facing cameras coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to provide feedback to the driver, for example, via a display, a speaker, and / or a vibrating component. In at least one embodiment, a LKA system is a variant of an LDW system. In at least one embodiment, an LKA system provides steering or braking impulses to correct the vehicle if it begins to leave its lane.

[0199] In at least one embodiment, a BSW system detects vehicles in a car's blind spot and warns the driver. In at least one embodiment, a BSW system can issue a visual, audible, and / or tactile warning to indicate that merging or changing lanes is unsafe. In at least one embodiment, a BSW system can issue an additional warning when a driver activates the turn signal. In at least one embodiment, a BSW system can use rear-facing camera(s) and / or radar sensor(s) 1760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, a speaker, and / or a vibrating component.

[0200] In at least one embodiment, an RCTW system can provide a visual, audible, and / or tactile notification when an object is detected outside the field of view of the reversing camera while the vehicle is reversing. In at least one embodiment, an RCTW system includes an AEB system to ensure that the vehicle's brakes are applied to avoid a collision. In at least one embodiment, an RCTW system can use one or more rear-facing radar sensors connected to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to provide feedback to the driver, for example, via a display, a speaker, and / or a vibrating component.

[0201] In at least one embodiment, conventional ADAS systems can be prone to false alarms, which can be disruptive and distracting for a driver, but are generally not catastrophic, since conventional ADAS systems warn a driver and allow that driver to decide whether a safety condition actually exists and to act accordingly. In at least one embodiment, the vehicle 1700 itself decides, in the case of conflicting results, whether to consider the result of a primary computer or a secondary computer (e.g., a first control unit or a second control unit of the control units 1736). In at least one embodiment, the ADAS system 1738 can, for example, be a backup and / or secondary computer that provides perceptual information to a backup computer rationality module.In at least one embodiment, a backup computer rationality monitor can run redundant, diverse software on hardware components to detect errors in perception and dynamic driving tasks. In at least one embodiment, the outputs of the ADAS system 1738 can be provided to a higher-level MCU. In at least one embodiment, if the outputs of a primary computer and the outputs of a secondary computer conflict, a higher-level MCU determines how the conflict can be resolved to ensure safe operation.

[0202] In at least one embodiment, a primary computer can be configured to provide a higher-level MCU with a confidence score indicating the primary computer's confidence in a selected result. In at least one embodiment, if this confidence score exceeds a threshold, the higher-level MCU can follow the primary computer's instruction, regardless of whether the secondary computer provides a contradictory or inconsistent result. In at least one embodiment, if a confidence score does not reach a threshold and if the primary and secondary computers display different results (e.g., a conflict), a higher-level MCU can mediate between the computers to determine a suitable result.

[0203] In at least one embodiment, a higher-level MCU can be configured to execute a neural network or neural networks trained and configured to determine, based on outputs from a primary computer and outputs from a secondary computer, the conditions under which the latter will issue false alarms. In at least one embodiment, neural networks in a higher-level MCU can learn when the output of a secondary computer is reliable and when it is not. For example, in at least one embodiment, if that secondary computer is a radar-based FCW system, a neural network in that monitoring MCU can learn when an FCW system identifies metallic objects that do not actually pose a hazard, such as a drain grate or manhole cover, triggering an alarm.In at least one embodiment, a neural network in a monitoring MCU can learn to override LDW when cyclists or pedestrians are present and a lane change is indeed the safest maneuver, provided a secondary computer is a camera-based LDW system. In at least one embodiment, a monitoring MCU can include at least one DLA or a GPU suitable for running neural networks with associated memory. In at least one embodiment, a higher-level MCU can be included as a component of and / or contained within a 1704 SoC (System on Chip).

[0204] In at least one embodiment, the ADAS system 1738 can include a secondary computer that performs ADAS functions using traditional computer vision rules. In at least one embodiment, this secondary computer can use classical computer vision rules (if-then), and the presence of one or more neural networks in a higher-level MCU can improve reliability, safety, and performance. In at least one embodiment, for example, a diverse implementation and intentional non-identity make the overall system more fault-tolerant, particularly with respect to errors caused by the functionality of the software-hardware interface (or the software-hardware interface itself).In at least one embodiment, for example, if a software bug or a bug in the software running on a primary computer exists and non-identical software code running on a secondary computer produces a consistent overall result, a monitoring MCU can be more confident that the overall result is correct and that a bug in the software or hardware on that primary computer does not cause a material error.

[0205] In at least one embodiment, an output from the ADAS system 1738 can be fed into the perception block of a primary computer and / or into the dynamic driving task block of a primary computer. For example, if, in at least one embodiment, the ADAS system 1738 displays a forward collision warning due to an object directly in front of the vehicle, a perception block can use this information for object identification. In at least one embodiment, a secondary computer can have its own trained neural network, thus reducing the risk of false alarms, as described herein.

[0206] In at least one embodiment, the vehicle 1700 may also include an infotainment SoC 1730 (e.g., an in-vehicle infotainment system (IVI)). Although depicted and described as an SoC, in at least one embodiment the infotainment system SoC 1730 may not be an SoC and may, without limitation, comprise two or more discrete components. In at least one embodiment, the infotainment SoC 1730 may, without limitation, comprise a combination of hardware and software that can be used to provide audio (e.g., music, a personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., television, movies, streaming, etc.), telephony (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.), and / or information services (e.g., navigation systems, rear parking assistance, a radio data system, etc.).) and / or information services (e.g., navigation systems, rear parking assistance, a radio data system, vehicle-related information such as fuel level, total distance traveled, brake fluid level, oil level, door open / close status, air filter information, etc.) for the vehicle 1700. For example, the infotainment SoC 1730 could include radios, disc players, navigation systems, video players, USB and Bluetooth connectivity, car computers, in-vehicle entertainment systems, WiFi, steering wheel audio controls, hands-free voice control, a head-up display (“HUD”), an HMI display 1734, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, functions, and / or systems), and / or other components. In at least one embodiment, the infotainment SoC 1730 can also be used to provide information (e.g., visual and / or audible) to the users of the vehicle 1700, such as... B.Information from the ADAS system 1738, autonomous driving information such as planned vehicle maneuvers, trajectories, information about the environment (e.g. intersection information, vehicle information, road information, etc.) and / or other information.

[0207] In at least one embodiment, the infotainment SoC 1730 can include any quantity and type of GPU functionality. In at least one embodiment, the infotainment SoC 1730 can communicate with other devices, systems, and / or components of the vehicle 1700 via bus 1702. In at least one embodiment, the infotainment SoC 1730 can be coupled to a higher-level MCU so that a GPU of an infotainment system can perform some self-driving functions if the primary controller(s) 1736 (e.g., the primary and / or backup computer of the vehicle 1700) fails. In at least one embodiment, the infotainment SoC 1730 can place the vehicle 1700 into a driver-to-safety stop mode, as described herein.

[0208] In at least one embodiment, the vehicle 1700 may also include a combination instrument 1732 (e.g., a digital instrument panel, an electronic combination instrument, a digital instrument panel, etc.). In at least one embodiment, the combination instrument 1732 may, without limitation, include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). In at least one embodiment, the combination instrument 1732 may, without limitation, include any number and combination of a series of instruments, such as a speedometer, a fuel gauge, an oil pressure gauge, a tachometer, an odometer, turn signals, a gear position indicator, seat belt warning lights, parking brake warning light(s), engine malfunction light(s), additional restraint system information (e.g., airbag), lighting controls, safety controls, navigation information, etc.In some examples, information can be displayed and / or exchanged between the Infotainment SoC 1730 and the Instrument Cluster 1732. In at least one configuration, the Instrument Cluster 1732 can be included as part of the Infotainment SoC 1730, or vice versa.

[0209] Such components can be used to generate an alternative image, such as a bird's-eye view image, from disparity data with limited hardware capacity, such as an embedded processor without access to external memory.

[0210] Fig. Figure 17D shows a diagram of a system for communication between one or more cloud-based servers and the autonomous vehicle 1700. Fig.17A according to at least one embodiment. In at least one embodiment, the system may, without limitation, comprise one or more servers 1778, one or more networks 1790, and any number and type of vehicles, including the vehicle 1700. In at least one embodiment, the server(s) 1778 may, without limitation, comprise a plurality of GPUs 1784(A)-1784(H) (hereinafter collectively referred to as GPUs 1784), PCIe switches 1782(A)-1782(D) (hereinafter collectively referred to as PCIe switches 1782), and / or CPUs 1780(A)-1780(B) (hereinafter collectively referred to as CPUs 1780). In at least one embodiment, GPUs 1784, CPUs 1780 and PCIe switches 1782 can be interconnected via high-speed links, such as, but not limited to, NVLink interfaces 1788 developed by NVIDIA and / or PCIe links 1786.In at least one embodiment, the GPUs 1784 are connected via an NVLink and / or NVSwitch SoC, and the GPUs 1784 and PCIe switches 1782 are connected via PCIe links. Although eight GPUs 1784, two CPUs 1780, and four PCIe switches 1782 are shown, this is not intended to be a limitation. In at least one embodiment, each of the servers 1778 can, without limitation, include any number of GPUs 1784, CPUs 1780, and / or PCIe switches 1782 in any combination. For example, in at least one embodiment, each server 1778 could have eight, sixteen, thirty-two, and / or more GPUs 1784.

[0211] In at least one embodiment, the servers 1778 can receive image data via networks 1790 and from vehicles, showing images of unexpected or changed road conditions, such as recently commenced roadworks. In at least one embodiment, the servers 1778 can transmit neural networks 1792, updated or otherwise, and / or map information 1794 to vehicles via networks 1790 and, including without limitation, information about traffic and road conditions. In at least one embodiment, updates to the map information 1794 can, without limitation, include updates to the HD map 1722, such as information about construction sites, potholes, detours, floods, and / or other obstacles.In at least one embodiment, neural networks 1792 and / or map information 1794 can result from new training and / or experience represented in data received from any number of vehicles in an environment, and / or at least partially based on training performed in a data center (e.g. using servers 1778 and / or other servers).

[0212] In at least one embodiment, Server 1778 can be used to train machine learning models (e.g., neural networks) at least partially based on training data. In at least one embodiment, training data can be generated by vehicles and / or in a simulation (e.g., using a game engine). In at least one embodiment, all training data is tagged (e.g., if the associated neural network benefits from supervised learning) and / or subjected to other preprocessing. In at least one embodiment, not all training data is tagged and / or preprocessed (e.g., if the associated neural network does not require supervised learning). In at least one embodiment, once trained, machine learning models can be used by vehicles (e.g.,(transmitted to vehicles via networks 1790) and / or machine learning models can be used by servers 1778 to monitor vehicles remotely.

[0213] In at least one embodiment, Server 1778 can receive data from vehicles and apply this data to current real-time neural networks to draw intelligent real-time conclusions. In at least one embodiment, Server 1778 can include deep learning supercomputers and / or dedicated AI computers powered by GPUs 1784, such as DGX and DGX Station machines developed by NVIDIA. However, in at least one embodiment, Server 1778 can also include a deep learning infrastructure that utilizes CPU-based data centers.

[0214] In at least one embodiment, the deep learning infrastructure of the server(s) 1778 can enable fast real-time inference and use this capability to assess and verify the state of processors, software, and / or associated hardware in the vehicle 1700. For example, in at least one embodiment, the deep learning infrastructure can receive regular updates from the vehicle 1700, such as a series of images and / or objects that the vehicle 1700 has detected in that image sequence (e.g., via computer vision and / or other machine learning techniques for object classification).In at least one embodiment, the deep learning infrastructure can operate its own neural network to identify objects and compare them with the objects identified by the vehicle 1700, and if the results do not match and the deep learning infrastructure concludes that the AI ​​in the vehicle 1700 is malfunctioning, the server 1778 can send a signal to the vehicle 1700 instructing a fail-safe computer of the vehicle 1700 to take control, notify the passengers and perform a safe parking maneuver.

[0215] Such components can be used to generate an alternative image, such as a bird's-eye view image, from disparity data with limited capacity, such as an embedded processor without access to external memory.

[0216] Different embodiments can be described by the following clauses: 1. System, exhibiting: at least one embedded processor with direct memory access (DMA) capability for: Generating a two-dimensional (2D) histogram view of one or more objects in an environment, based in part on disparity data for the one or more objects, wherein the two-dimensional histogram view is a function of the angle and distance to a plane of at least one camera used to generate the stereo disparity data; and Creating a bird's-eye view image of one or more objects by transforming the 2D histogram view. 2. System according to claim 1, wherein the at least one embedded processor has no access to an external memory that could be used to generate the bird's-eye view image. 3. System according to clause 1 or 2, wherein the system is further configured to determine the disparity data using image data captured by the camera. 4. System according to any of the preceding clauses, wherein the camera is a stereoscopic camera unit or a pair of matched camera sensors. 5. System according to any of the preceding clauses, wherein the bird's-eye view image is partially generated by generating a list of object centers and statistics using the 2D histogram view and by converting the list into a corresponding list in a bird's-eye view coordinate system. 6. System according to any of the preceding clauses, wherein data transfers for the embedded processor are performed using DMA on rectangular areas of the image data. 7. System according to one of the preceding clauses, wherein the at least one embedded processor is further configured to: Receiving stereoscopic image data from at least one camera; and Generating a disparity map containing the disparity data for one or more objects using the stereoscopic image data. 8. System according to any of the preceding clauses, wherein the disparity data includes data obtained from at least one additional sensor. 9. System according to one of the preceding clauses, wherein the system comprises at least one of: a system for performing simulation processes; a system for performing simulations to test or validate autonomous machine applications; a system for performing digital twin operations; a system for performing light transport simulations; a system for rendering graphical output; a system for performing deep learning operations; a system for performing generative AI operations using a large language model (LLM), a system for performing generative AI operations using a visual model (VLM), a system for performing generative AI operations using a multimodal language model (MMLM); a system for providing one or more language models using an operating system-level (OS) virtualization container that communicates with the one or more language models via one or more application programming interfaces (APIs); a system implemented using an edge device; a system for creating or presenting virtual reality (VR) content; a system for creating or presenting augmented reality (AR) content; a system for creating or presenting mixed-reality (MR) content; a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; a system for performing hardware tests using simulation; a system for generating synthetic data; a platform for the collaborative creation of content for 3D assets; or a system that is implemented at least partially using cloud computing resources. 10. At least one embedded processor with direct memory access (DMA) capability to generate a bird's-eye view image of a scene by generating an intermediate histogram from disparity data for the scene as a function of the angle and distance to a camera plane and converting the intermediate histogram into the bird's-eye view image. 11. At least one embedded processor according to clause 10, wherein the intermediate histogram contains representations of one or more objects in the scene, and wherein the at least one embedded processor is configured to perform an analysis of connected components on the intermediate histogram in order to identify pixel positions that are associated with the one or more objects. 12. At least one embedded processor according to clause 11, wherein the at least one embedded processor is further configured to generate a list of object centers and statistics for the one or more objects using the intermediate histogram view, and to convert the list into a corresponding list in a coordinate system of the bird's-eye view image. 13. At least one embedded processor according to one of clauses 10 to 12, wherein the at least one embedded processor does not have access to a complete set of image data stored in an external memory for use in generating the intermediate histogram or the bird's-eye view image. 14. At least one embedded processor according to any of clauses 10 to 13, wherein data transfers for the at least one embedded processor are carried out using DMA on rectangular areas of image data. 15. At least one embedded processor according to any one of clauses 10 to 14, wherein the at least one embedded processor is further configured to determine the disparity data using image data acquired with at least one stereoscopic camera unit. 16. At least one embedded processor according to any one of clauses 10 to 15, wherein the at least one embedded processor is contained in at least one of: a system for performing simulation operations; a system for performing simulation processes to test or validate autonomous machine applications; a system for performing digital twin operations; a system for performing a light transport simulation; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using an edge device; a system for creating or presenting virtual reality (VR) content; a system for creating or presenting augmented reality (AR) content; a system for creating or presenting mixed-reality (MR) content; a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; a system for performing hardware tests using simulation; a system for generating synthetic data; a system for performing generative AI operations using a large language model (LLM), a system for performing generative AI operations using a visual model (VLM), a system for performing generative AI operations using a multimodal language model (MMLM); a system for providing one or more language models using an operating system-level (OS) virtualization container that communicates with the one or more language models via one or more application programming interfaces (APIs); a collaborative platform for creating content for 3D assets; or a system that is implemented at least partially using cloud computing resources. 17. Computer-implemented method, comprising: Generating an intermediate histogram representation of a disparity image using an embedded processor with DMA memory access; Identifying positions in the intermediate histogram representation that are associated with one or more objects; and Transforming the intermediate histogram representation using the embedded processor and partly based on the locations into a bird's-eye view image containing representations of one or more objects. 18. Computer-implemented method according to claim 17, wherein the embedded processor has no access to an external memory that could be used to generate the intermediate histogram representation or the bird's-eye view image. 19. Computer-implemented methods according to clause 17 or 18, which further includes: Performing an analysis of connected components using the embedded processor on the intermediate histogram display to identify the positions associated with the one or more objects. 20. Computer-implemented methods according to clause 18, which further includes: Generating a list of object centers and statistics for one or more objects from the intermediate histogram display using the embedded processor; and Transforming the list using the embedded processor into a corresponding list in a coordinate system of the bird's-eye view image. 21. Computer-implemented methods according to clause 18, wherein the bird's-eye view image is provided for use in a simulation of the one or more objects that were created at least partially using a three-dimensional (3D) content collaboration platform for 3D assets. 22. Computer-implemented methods according to clause 21, wherein the 3D content collaboration platform for 3D assets uses OpenUSD.

[0217] Other variations are within the scope of this disclosure. Although the disclosed techniques are subject to various modifications and alternative designs, certain illustrated embodiments are shown in the drawings and described in detail above. However, it should be understood that there is no intention to limit the disclosure to specific disclosed forms, but rather the intention to cover all modifications, alternative designs, and equivalents that are within the scope and meaning of the disclosure, as defined in the accompanying claims.

[0218] The use of the terms "a," "an," and "that," as well as similar reference words, in connection with the description of the disclosed embodiments (particularly in connection with the following claims) is to be understood as covering both the singular and plural forms, unless otherwise specified herein or clearly contradicted by the context, and not as a definition of a term. The terms "comprising," "having," "including," and "containing" are to be understood as open terms (in the sense of "including, but not limited to"), unless otherwise specified. The term "connected," when unmodified and referring to physical connections, is to be understood as something being partially or completely contained in, attached to, or connected with something else, even if something is in between.The specification of value ranges here serves only as a shorthand method to refer individually to each value within the range, unless otherwise stated herein and each individual value is included in the specification as if it were listed here individually. Unless otherwise stated or contradicted by the context, the use of the terms "set" (e.g., "a set of elements") or "subset" is to be understood as a non-empty collection containing one or more elements. Unless otherwise stated or contradicted by the context, the term "subset" of a corresponding set does not necessarily denote a proper subset of the corresponding set, but subset and corresponding set may be the same.

[0219] Conjunctive phrases such as "at least one of A, B, and C" or "at least one of A, B, and C" are, unless expressly stated otherwise or clearly contradicted by the context, to be understood in context as being generally used to indicate that an element, term, etc., can be either A, B, or C, or any non-empty subset of the set of A, B, and C. In an example of a set with three elements, the conjunctive phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to each of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Therefore, such a conjunctive phrase is generally not intended to imply that in certain embodiments, at least one of A, at least one of B, and at least one of C must be present.Furthermore, unless otherwise stated or contradicted by the context, the term "plurality" denotes a state of plurality (e.g., "a plurality of elements" denotes multiple elements). A plurality comprises at least two elements, but may include more if explicitly stated or indicated by the context. Unless otherwise stated or clear from the context, the phrase "based on" means "at least partly based on" and not "exclusively based on".

[0220] The process steps described herein can be performed in any order unless otherwise specified herein or clearly evident from the context. In at least one embodiment, a process, such as the processes described herein (or variations and / or combinations thereof), is executed under the control of one or more computer systems configured with executable instructions and is implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or combinations thereof. In at least one embodiment, the code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions that can be executed by one or more processors.In at least one embodiment, a computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transitory signals (e.g., a propagating transitory electrical or electromagnetic transmission) but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues) within transceivers for transitory signals. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media containing executable instructions (or other memory for storing executable instructions) which, when executed by one or more processors of a computer system (e.g., as a result of execution), cause the computer system to perform the operations described herein.In at least one embodiment, a group of non-transitory, computer-readable storage media comprises several non-transitory, computer-readable storage media, wherein one or more of the individual non-transitory storage media in the group of non-transitory, computer-readable storage media contain no code, while the group of non-transitory, computer-readable storage media as a whole stores all the code. In at least one embodiment, executable instructions are executed such that different instructions are executed by different processors—for example, a non-transitory, computer-readable storage medium stores instructions, and a central processing unit (CPU) executes some of the instructions, while a graphics processing unit (GPU) executes other instructions.In at least one embodiment, different components of a computer system have separate processors, and different processors execute different subsets of instructions.

[0221] Accordingly, in at least one embodiment, computer systems are configured to implement one or more services that individually or collectively perform the process operations described herein, and such computer systems are configured with suitable hardware and / or software that enables the execution of operations. Furthermore, a computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment, it is a distributed computer system comprising several devices that operate differently, such that the distributed computer system performs the operations described herein, and a single device does not perform all operations.

[0222] The use of all examples or illustrative phrases contained herein (e.g., "such as") serves only to better illustrate the embodiments of the disclosure and does not constitute a limitation of the scope of the disclosure unless otherwise stated. No wording in the description shall be construed as indicating that an unclaimed element is essential for the implementation of the disclosure.

[0223] All references cited herein, including publications, patent applications and patents, are hereby incorporated by reference to the same extent as if each reference were individually and expressly indicated as being incorporated by reference and reproduced in full herein.

[0224] The terms "coupled" and "connected," as well as their derivatives, may be used in the description and claims. It should be clear that these terms are not intended as synonyms. Rather, "connected" or "coupled" may be used in certain examples to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" can also mean that two or more elements are not in direct contact with each other but nevertheless work together or interact with each other.

[0225] Unless expressly stated otherwise, terms such as "processing", "calculating", "calculating", "determining" or similar terms can be understood to refer to actions and / or processes of a computer or computer system or similar electronic computing device that convert physically represented data, such as electronic quantities, in the registers and / or memories of the computer system into other data, also represented as physical quantities in the memories, registers or other information storage, transmission or display devices of the computer system.

[0226] Similarly, the term "processor" can refer to any device or part of a device that processes electronic data from registers and / or memories and converts that electronic data into other electronic data that can be stored in registers and / or memories. As non-restrictive examples, a "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used here, "software processes" can include, for example, software and / or hardware units that perform work over time, such as tasks, threads, and intelligent agents. Furthermore, each process can refer to multiple processes for executing instructions sequentially or in parallel, continuously or intermittently.The terms "system" and "procedure" are used synonymously here, since a system can embody one or more procedures and procedures can be considered as a system.

[0227] This document can be about fetching, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-controlled machine. Fetching, acquiring, receiving, or inputting analog and digital data can occur in various ways, for example, by receiving data as parameters of a function call or an application programming interface (API) call. In some implementations, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transferring data over a serial or parallel interface. In another implementation, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transferring data over a computer network from a providing unit to a receiving unit.It can also refer to the provision, output, transmission, sending, or presentation of analog or digital data. In various examples, the process of providing, outputting, transmission, sending, or presenting analog or digital data can be accomplished by transferring data as input or output parameters of a function call, a parameter of an application programming interface, or an interprocess communication mechanism.

[0228] Although example implementations of the described techniques are presented above, other architectures can also be used to implement the described functionality, and these are likewise intended to fall within the scope of this disclosure. While certain distributions of responsibilities are defined above for discussion purposes, various functions and responsibilities may be distributed and divided in different ways depending on the circumstances.

[0229] Although the subject matter was described in language relating to structural features and / or methodological actions, it should be noted that the subject matter claimed in the attached claims is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as exemplary forms of implementation of the claims.

[0230] It is understood that the aspects and embodiments described above serve only as examples and that detailed modifications may be made within the scope of the claims.

[0231] Each device, method and feature disclosed in the description and (where applicable) in the claims and drawings can be provided independently or in any suitable combination.

[0232] The reference numerals used in the claims are for illustrative purposes only and do not restrict the scope of the claims. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] Taxonomy and definitions for terms related to automated driving systems for road vehicles” (e.g. Standard No. J3016-201806, published on 15 June 2018, Standard No. J3016-201609, published on 30 September 2016

[0114]

Claims

[1] System, exhibiting: at least one embedded processor with direct memory access (DMA) capability for: Generating a two-dimensional (2D) histogram view of one or more objects in an environment, based in part on disparity data for the one or more objects, wherein the two-dimensional histogram view is a function of the angle and distance to a plane of at least one camera used to generate the stereo disparity data; and Creating a bird's-eye view image of one or more objects by transforming the 2D histogram view. [2] System according to claim 1, wherein the at least one embedded processor has no access to an external memory that could be used to generate the bird's-eye view image. [3] System according to claim 1 or 2, wherein the system is further configured to determine the disparity data using image data captured with the camera. [4] System according to any of the preceding claims, wherein the camera is a stereoscopic camera unit or a pair of matched camera sensors. [5] System according to one of the preceding claims, wherein the bird's-eye view image is partially generated by generating a list of object centers and statistics using the 2D histogram view and by converting the list into a corresponding list in a bird's-eye view coordinate system. [6] System according to one of the preceding claims, wherein data transfers for the embedded processor are carried out using DMA on rectangular areas of the image data. [7] System according to any of the preceding claims, wherein the at least one embedded processor is further configured to: Receiving stereoscopic image data from at least one camera; and Generating a disparity map containing the disparity data for one or more objects using the stereoscopic image data. [8] System according to any of the preceding claims, wherein the disparity data comprise data obtained from at least one additional sensor. [9] System according to any one of the preceding claims, wherein the system comprises at least one of: a system for performing simulation processes; a system for performing simulations to test or validate autonomous machine applications; a system for performing digital twin operations; a system for performing light transport simulations; a system for rendering graphical output; a system for performing deep learning operations; a system for performing generative AI operations using a large language model (LLM), a system for performing generative AI operations using a visual model (VLM), a system for performing generative AI operations using a multimodal language model (MMLM); a system for providing one or more language models using an operating system-level (OS) virtualization container that communicates with the one or more language models via one or more application programming interfaces (APIs); a system implemented using an edge device; a system for creating or presenting virtual reality (VR) content; a system for creating or presenting augmented reality (AR) content; a system for creating or presenting mixed-reality (MR) content; a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; a system for performing hardware tests using simulation; a system for generating synthetic data; a platform for the collaborative creation of content for 3D assets; or a system that is implemented at least partially using cloud computing resources. [10] At least one embedded processor with direct memory access (DMA) function to generate a bird's-eye view image of a scene by generating an intermediate histogram from disparity data for the scene as a function of the angle and distance to a camera plane and converting the intermediate histogram into the bird's-eye view image. [11] At least one embedded processor according to claim 10, wherein the intermediate histogram contains representations of one or more objects in the scene, and wherein the at least one embedded processor is configured to perform an analysis of connected components on the intermediate histogram in order to identify pixel positions that are associated with the one or more objects. [12] At least one embedded processor according to claim 11, wherein the at least one embedded processor is further configured to generate a list of object centers and statistics for the one or more objects using the intermediate histogram view, and to convert the list into a corresponding list in a coordinate system of the bird's-eye view image. [13] At least one embedded processor according to any one of claims 10 to 12, wherein the at least one embedded processor does not have access to a complete set of image data stored in an external memory for use in generating the intermediate histogram or the bird's-eye view image. [14] At least one embedded processor according to any one of claims 10 to 13, wherein data transfers for the at least one embedded processor are carried out using DMA on rectangular areas of image data. [15] At least one embedded processor according to any one of claims 10 to 14, wherein the at least one embedded processor is further configured to determine the disparity data using image data acquired with at least one stereoscopic camera unit. [16] At least one embedded processor according to any one of claims 10 to 15, wherein the at least one embedded processor is contained in at least one of: a system for performing simulation operations; a system for performing simulation processes to test or validate autonomous machine applications; a system for performing digital twin operations; a system for performing a light transport simulation; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using an edge device; a system for creating or presenting virtual reality (VR) content; a system for creating or presenting augmented reality (AR) content; a system for creating or presenting mixed-reality (MR) content; a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; a system for performing hardware tests using simulation; a system for generating synthetic data; a system for performing generative AI operations using a large language model (LLM), a system for performing generative AI operations using a visual model (VLM), a system for performing generative AI operations using a multimodal language model (MMLM); a system for providing one or more language models using an operating system-level (OS) virtualization container that communicates with the one or more language models via one or more application programming interfaces (APIs); a collaborative platform for creating content for 3D assets; or a system that is implemented at least partially using cloud computing resources. [17] Computer-implemented method which features: Generating an intermediate histogram representation of a disparity image using an embedded processor with DMA memory access; Identifying positions in the intermediate histogram representation that are associated with one or more objects; and Transforming the intermediate histogram representation using the embedded processor and partly based on the locations into a bird's-eye view image containing representations of one or more objects. [18] Computer-implemented methods according to claim 17, wherein the embedded processor does not have access to an external memory that could be used to generate the intermediate histogram representation or the bird's-eye view image. [19] Computer-implemented method according to claim 17 or 18, further comprising: Performing an analysis of connected components using the embedded processor on the intermediate histogram display to identify the positions associated with the one or more objects. [20] Computer-implemented method according to claim 18, further comprising: Generating a list of object centers and statistics for one or more objects from the intermediate histogram display using the embedded processor; and Transforming the list using the embedded processor into a corresponding list in a coordinate system of the bird's-eye view image. [21] Computer-implemented methods according to claim 18, wherein the bird's-eye view image is provided for use in a simulation of the one or more objects that were generated at least partially using a three-dimensional (3D) content collaboration platform for 3D assets. [22] Computer-implemented methods according to claim 21, wherein the 3D content collaboration platform for 3D assets uses OpenUSD.

Images