Extended bounding shape representation associated with three-dimensional object detection

Abstract

The invention relates to an extended bounding shape representation associated with three-dimensional object detection. In various examples, embodiments are directed to generating an extended bounding shape representation corresponding to an object in an environment in an efficient and efficient manner. In particular, various parameters may be used to represent a bounding shape associated with an object, including location parameters, size parameters, and orientation parameters that describe spatial attributes of the object. Advantageously, the orientation parameter includes a representation or indication of rotation about the x-axis, y-axis and z-axis. Orientation parameters associated with multiple orientations, such as angles of rotation about x-, y-, and z-axes, facilitate more comprehensive analysis of the environment, especially where sensors, such as a camera and LiDAR, are mounted on a wall or ceiling, or where angles of rotation may be present in association with multiple axes.

Description

BACKGROUND

[0001] Various sensors generate different types of sensor data. Generally, these sensor data complement each other. For example, LiDAR and camera sensors can provide mutually complementary sensor data for various computer vision tasks under different circumstances. By aggregating or fusing various sensor data, such as LiDAR and camera data, the strengths of both sensors can be leveraged to more reliably detect objects, even under challenging conditions. As one example, LiDAR data can be used to detect objects that are not visible via a camera (e.g., due to fog or water droplets). Thus, different types of sensor data can be aggregated or combined to facilitate object detection.

[0002] One approach to fusing different types of data, such as LiDAR and camera data, includes bird’s eye view (BEV) fusion of the different types of data. At a high level, this approach generates a set of fused or combined features in the form of a bird’s eye view. After generating the set of fused BEV features, these features can be used to perform object detection, including generating or identifying bounding boxes corresponding to objects using the fused BEV features.

[0003] In traditional implementations, LiDAR-camera fused BEV datasets are typically used in the context of autonomous driving. In the autonomous driving context, bounding boxes related to objects can be represented using a position parameter, a size parameter, and a single rotation parameter (e.g., rotation about the y-axis), thus representing seven degrees of freedom. Specifically, when an autonomous vehicle is moving perpendicular to the ground it is moving on, rotation about a single axis is typically used. Since the autonomous vehicle does not move up and down, rotation about the other axes is assumed to be zero and is not used. By way of example, assume that an autonomous vehicle includes multiple cameras (e.g., six or eight cameras) located around the vehicle and a LiDAR sensor located on the roof of the vehicle that rotates horizontally in all directions. In this case, data from the various cameras and LiDAR can be fused together for performing object detection, where a single rotation parameter is identified to define a bounding box around an object.

[0004] However, using such traditional approaches in other environmental applications can result in an inability to accurately define bounding boxes associated with objects. For example, in cases where sensors (such as LiDAR) are mounted on walls, ceilings, or other structures in an environment (e.g., for monitoring the environment), a bounding box defined using multiple degrees of freedom (e.g., seven degrees of freedom) can fail to adequately represent a particular object. For example, a single rotation parameter fails to capture all possible orientations of an object, which can include a combination of rotations about all three axes. As such, the actual orientation of an object can fail to be accurately represented, resulting in inaccuracies when performing object analysis such as collision detection, object manipulation, etc. In this regard, in cases where sensors (such as LiDAR sensors and / or cameras) are mounted on fixed structures (e.g., smart environment use cases), the assumption of zero degrees of rotation about two axes can be inaccurate.

[0005] Accordingly, using such traditional approaches that can fail to accurately reflect objects in various environments can consume significant computational resources. In particular, accurately identifying objects (such as three-dimensional objects) in an environment can reduce or eliminate the need for various potential subsequent computations, thereby reducing the utilization of computational resources. For example, accurate object identification can reduce the performance of subsequent searches or scans in the environment, improve the performance of additional post-processing tasks such as object location and boundary and false positive detection, etc. Accurate object identification can also enable efficient resource allocation (e.g., computer processing can be focused on particular areas), and enable enhanced tracking and prediction of objects.

[0006] Accordingly, traditional approaches that use fused feature data (e.g., data associated with LiDAR sensors and cameras) to generate or identify a single orientation parameter associated with a bounding box corresponding to an object can result in unnecessary use of computational resources to perform various data processing, particularly in cases where sensors are mounted to monitor an environment. The need to perform such additional data processing due to inaccurate object detection can reduce the efficiency of other processes being performed, and reduce overall system efficiency, thereby limiting the ability to efficiently and effectively analyze an environment. SUMMARY

[0007] Embodiments of the present disclosure relate to efficiently and effectively generating an extended bounding shape representation corresponding to a three-dimensional object in an environment. The disclosed systems and methods identify multiple orientation parameters associated with a bounding shape of an object, such that nine degrees of freedom can be used to define a bounding shape for the object (e.g., x position, y position, z position, width, height, depth, roll, pitch, and yaw). In this manner, an accurate bounding shape representing an object can be used for object tracking, manipulation, navigation, and / or other types of analysis of objects in an environment.

[0008] Unlike conventional systems, in some embodiments, spatial parameters (including multiple orientation parameters) are identified in association with a bounding shape corresponding to an object. In this regard, spatial parameters corresponding to rotations about the x-axis, the y-axis, and the z-axis can be identified via a machine learning model (e.g., an object detection model) based on a feature representation representing features associated with multiple sensors. To generate spatial parameters corresponding to rotations about multiple axes, an object detection model can be trained using a training dataset containing multiple orientation spatial parameters (e.g., roll, pitch, and yaw). In some cases, spatial parameters used for training (including ground truth orientation parameters) can be synthetically generated, thereby providing a high quality and efficiently generated training dataset. BRIEF DESCRIPTION OF DRAWINGS

[0009] The present systems and methods for generating extended bounding shape representations corresponding to three-dimensional objects in an environment are described in detail below with reference to the accompanying drawings, wherein:

[0010] Figure 1 is a dataflow diagram illustrating an example process for a three-dimensional object detection system in accordance with some embodiments of the present disclosure;

[0011] Figure 2 is an illustration providing one example implementation for generating a unified feature representation in accordance with some embodiments of the present disclosure;

[0012] Figure 3 One example method for generating a bounding shape representation for an object in accordance with some embodiments of the present disclosure is provided;

[0013] Figure 4 Another example method for generating a bounding shape representation for an object in accordance with some embodiments of the present disclosure is provided;

[0014] Figure 5 Another example method for generating a bounding shape representation for an object in accordance with some embodiments of the present disclosure is provided;

[0015] Figure 6 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0016] Figure 7 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] The systems and methods disclosed herein relate to generating an augmented or extended bounding shape representation corresponding to an object in an environment in an efficient and effective manner. In this regard, various parameters, including a plurality of orientation parameters, can be used to represent a bounding shape associated with an object, which describe spatial properties of the object. Specifically, these spatial parameters include a location parameter, a size parameter, and an orientation parameter associated with a three-dimensional environment. Advantageously, the orientation parameter includes a representation or indication of a rotation about an x-axis, a y-axis, and a z-axis. The orientation parameter associated with a plurality of orientations, such as the rotation angles about the x-axis, the y-axis, and the z-axis, facilitates a more comprehensive analysis of the environment, especially in cases where the sensor is mounted on a pole, a wall, or a ceiling, or in other cases where there can be rotation angles associated with multiple axes.

[0018] Accurate identification of objects, such as three-dimensional objects, in an environment reduces or eliminates the need for various potential subsequent computations, thereby reducing utilization of computational resources. For example, accurate object identification reduces the performance of subsequent searches or scans in the environment, improves the performance of object location and boundary and additional post-processing tasks for false positive detection, etc. Accurate object identification also enables efficient resource allocation (e.g., computer processing can focus on specific areas), and enables enhanced object tracking and prediction.

[0019] At a high level, the embodiments described herein aim to generate an extended bounding shape representation corresponding to an object in an environment in an efficient and effective manner. In this way, an object can be identified or detected in association with a set of extended spatial parameters that indicate a location, a size, and an orientation associated with a bounding shape for the object, and more specifically, an object captured in a unified feature representation that represents features associated with a plurality of different types of sensors (e.g., LiDAR sensors and cameras). Accordingly, a plurality of orientation parameters can be identified for a bounding shape associated with an object to define a first angle of rotation about an x-axis, a second angle of rotation about a y-axis, and a third angle of rotation about a z-axis.

[0020] In operation, sensor data can be acquired from a variety of sensors of different types. As noted above, in some cases, sensors can be located on walls, ceilings, poles, or other structures in an environment to capture sensor data. In some embodiments, sensors can be placed in a fixed or static manner to capture a particular or static environment, while objects can move within the environment. Such static environments with dynamic objects can include a physical layout that remains fixed (e.g., walls, floors, fixed furniture, and other immovable structures) and provide a consistent or stable frame of reference to observe motion therein. Objects that can move within such static environments include people, vehicles, robots, or other movable items. These objects can move position, change orientation, interact with static or other dynamic objects, and / or exhibit other behaviors over time. By way of example only, LiDAR sensors and cameras can be positioned (e.g., proximate to one another) and / or oriented to capture the same or similar portions of an environment. In some cases, sensors can be placed on fixed devices (e.g., walls, ceilings, or pillars) to capture an interior or exterior environment.

[0021] By acquiring sensor data (e.g., from cameras and LiDAR sensors), a representation of a set of features detected in association with objects in an environment can be generated. As used herein, a feature can refer to any characteristic that captures or indicates a spatial pattern or boundary associated with an object in an environment. In some embodiments, a unified representation of features is generated. A unified representation of features, or unified feature representation, generally refers to a representation of features identified in association with multiple sensors, such as sensors of different types. Thus, various features from different types of sensors, such as cameras and LiDAR, can be combined or fused into a single, unified representation of features. The unified feature representation can represent features in any number of perspectives or spaces. In this way, features can be converted to a single perspective or space. For example, in the case of LiDAR and camera features represented in a unified feature representation, the unified feature representation can take the form of a bird’s eye view (BEV), also referred to as a top-down view. In this way, features associated with a LiDAR sensor and features associated with a camera can be fused or aggregated in a unified BEV space or perspective to generate a unified feature representation. Generating a unified feature representation in the form of a BEV can more readily identify shapes and orientations. Advantageously, generating a unified feature representation with a BEV preserves both the geometry from LiDAR features and the semantic density from camera features.

[0022] The feature representation or unified feature representation can be used to detect three-dimensional objects in an environment. In this regard, a bounding shape corresponding to an object in the environment can be identified. The bounding shape (e.g., a box or cuboid shape) can be used to define a location of the object in an image or representation of the environment. The bounding shape can be represented via spatial parameters that indicate a location, size, and orientation of the bounding shape corresponding to the object in the environment. Accordingly, various spatial parameters can be generated or identified in association with the bounding shape of the object. In this way, location parameters, size parameters, and orientation parameters can be used to characterize or indicate the bounding shape corresponding to the object. The location parameters can include location parameters associated with x, y, and z coordinates. The size parameters generally define the physical extent or size of the bounding shape along three axes (length, width, and height). The orientation parameters generally refer to angles associated with the rotation of the bounding shape about or around an axis. The orientation parameters can include an orientation or rotation angle of the bounding shape defining a rotation of the bounding shape about a vertical axis (e.g., the y-axis), an orientation or rotation angle of the bounding shape defining a rotation of the bounding shape about a horizontal axis (e.g., the x-axis), and an orientation or rotation angle of the bounding shape defining a rotation of the bounding shape about a depth axis (e.g., the z-axis). In some cases, the orientation or rotation angles can be represented with sine and cosine components. In particular, the orientation (i.e., the rotation about an axis, typically represented as an angle) can be represented using the sine and cosine of the rotation angle, for example, to avoid issues of discontinuity and ambiguity. This approach is more robust and enables the model to learn the orientation in a more continuous manner.

[0023] To generate the spatial parameters, an object detection model can be used that outputs a set of spatial parameters describing or indicating an object in three-dimensional space. In one embodiment, the object detection model can be a deep learning network, such as a deep neural network (e.g., a convolutional neural network, such as a Faster R-CNN), containing various convolutional layers that processes a feature representation (e.g., fused BEV data) to detect a set of spatial parameters corresponding to an object in the environment. The object detection model can take a feature representation, such as a unified feature representation, as input and predict or provide various spatial parameters associated with a bounding box associated with the object as output. In one example, the output is in the form of a tensor containing such location, size, and orientation parameters. In some embodiments, the object detection model or portions thereof can predict sine and cosine components associated with rotation about each axis, predicting two independent components for each degree of freedom of orientation. In this way, the object detection model can generate 12 spatial parameters, such as location, size, and orientation parameters representing 9 degrees of freedom.

[0024] To predict or generate spatial parameters representing nine degrees of freedom, the object detection model can be trained using ground truth representations of nine degrees of freedom. As one example, the ground truth spatial parameters can include an x position label, a y position label, a z position label, a length label, a width label, a depth label, a rotation angle about the x axis, a rotation angle about the y axis, and a rotation angle about the z axis. As another example, the ground truth spatial parameters can include an x position label, a y position label, a z position label, a length label, a width label, a depth label, a sine of a rotation angle about the x axis, a cosine of a rotation angle about the x axis, a sine of a rotation angle about the y axis, a cosine of a rotation angle about the y axis, a sine of a rotation angle about the z axis, and a cosine of a rotation angle about the z axis.

[0025] In some embodiments, the ground truth labels are synthetically generated. For example, a simulator or graphics engine can be used to generate artificially and realistically images of different environments (e.g., warehouses) containing various objects (e.g., people, robots) therein. Using synthetically generated images, the spatial parameters associated with the various objects can be known or predefined. In this way, for one object, the position, size, and orientation (including three rotational degrees of freedom) for a camera image and LiDAR point cloud pair can be known (e.g., via the code that generates the graphics). In this way, human annotation of the ground truth spatial parameters is avoided.

[0026] After the spatial parameters are generated or predicted, one or more post-processing operations can be performed to refine, filter, and / or interpret the predicted spatial parameters. As one example, the orientation parameters represented via sine and cosine components can be converted back to rotation angles to represent the orientation of the object (e.g., for each orientation associated with a rotation axis). In this regard, an axis orientation represented by two components (e.g., a sine and a cosine of a rotation angle) can be converted or transformed to represent the axis orientation via a single angle representing the magnitude of the rotation about the axis. In this regard, the originally predicted six orientation parameters representing three degrees of freedom can be converted to three orientation parameters to represent the enclosing shape.

[0027] The refined spatial parameters or final spatial parameters can then represent the enclosing shape associated with the object. In this way, the output or refined spatial parameters can be used to represent the enclosing shape, including a nine degree of freedom representation (e.g., three position representations, three size representations, and three orientation representations). Advantageously, representing the enclosing shape in nine degrees of freedom, including three orientation representations associated with three axes in three-dimensional space, provides a more complete and accurate description of the rotation and orientation of the object and reduces or eliminates ambiguities that can arise from a more limited representation.

[0028] Such representations of enclosing shapes can be used in various environments, such as robotic environments (e.g., robotic arms, drones, and autonomous vehicles). Further representations of enclosing shapes associated with objects defined by spatial parameters can be used for precise localization and analysis of objects in three-dimensional environments. For example, spatial parameters can be used for object tracking, collision detection and avoidance, object interaction and manipulation, scene understanding, behavior analysis, data augmentation, object density estimation, anomaly detection, multi-modal integration, etc.

[0029] Accordingly, the techniques described herein can be used to identify spatial parameters representing or defining enclosing shapes of objects in an efficient and effective manner, including various orientation parameters. The identified spatial parameters representing nine degrees of freedom can be provided to assist in performing one or more operations, such as operations associated with localizing, tracking, and / or analyzing objects in an environment. Unlike traditional approaches, various embodiments provide a method for enabling generation of spatial parameters associated with a unified feature representation (e.g., in BEV form) including multiple orientation parameters. The use of nine degrees of freedom of the representation of enclosing shapes provides a more precise representation, allowing for more computationally resource efficient implementations. For example, fewer searches or environment scans can be performed based on accurate object identification, fewer post-processing tasks can be performed to refine the location and boundaries of objects and detect false positives, etc. Moreover, training using synthetically generated data can enable a more scalable process and provide high quality and consistent data, eliminating discrepancies and errors that can arise from human annotation.

[0030] While the present disclosure can be described with respect to example static environments with dynamic objects, this is not intended to be limiting. For example, the systems and methods described herein can be used in conjunction with, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more advanced driver assistance systems [ADAS]), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, watercraft, space shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, airplanes, engineering vehicles, trains, underwater vehicles, remote-controlled vehicles such as drones, and / or other types of vehicles. Moreover, while the present disclosure can be described with respect to smart environments, this is not intended to be limiting, the systems and methods described herein can be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technical field that can perform object detection.

[0031] Reference Figure 1 , Figure 1is a data flow diagram illustrating an example process 100 for a three-dimensional object detection system, in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be omitted altogether. Further, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location of components, can be implemented. Various functions described herein as being performed by entities can be stored in and executed by hardware, firmware and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be implemented using components, features, and / or functions similar to those of the example computing device 600 and / or the example data center 700 described below. Figure 6 Figure 7 may be performed using components, features, and / or functions similar to those of the example computing device 600 and / or the example data center 700 described below.

[0032] At a high level, the process 100 uses a three-dimensional object detector 110 to generate a representation of three-dimensional objects in an environment. In this regard, the three-dimensional object detector 110 can generate a representation of an enclosing shape corresponding to an object in the environment. According to embodiments described herein, the enclosing shape associated with an object can be represented using various parameters that describe the spatial properties of the detected object. These spatial parameters include location parameters, size parameters, and orientation parameters associated with the three-dimensional environment. Advantageously, the orientation parameters include a representation or indication of a rotation about the x-axis, the y-axis, and the z-axis. Orientation parameters associated with multiple orientations, such as rotation angles about the x-axis, the y-axis, and the z-axis, facilitate a more comprehensive analysis of the environment, especially in cases where the sensor is mounted on a wall or ceiling, or in other cases where the rotation angle can be associated with multiple axes.

[0033] In some embodiments, the three-dimensional object detector 110 can use the feature representation 108 to perform the generation or identification of the enclosing shape representation. As such, the three-dimensional object detector 110 can obtain the feature representation 108 and provide, as output, a corresponding enclosing shape representation representing an object in the environment.

[0034] In some embodiments, the feature representation generator 106 is configured (e.g., programmed) to generate or identify the feature representation 108. In particular, the feature representation generator 106 can generate or identify a representation of a set of features detected in association with an object in the environment. As used herein, a feature can refer to any feature that captures or indicates a spatial pattern or boundary associated with an object in the environment.

[0035] ​In some embodiments, the feature representation generator 106 generates a unified representation of features in the environment. The unified representation of features, or unified feature representation, generally refers to a representation of features identified in association with multiple sensors, such as different types of sensors. In this way, various features from different sensors or different views can be combined into a single, unified representation of features. In one embodiment, the unified representation of features represents features associated with a camera (camera features) and features associated with a LiDAR sensor (LiDAR features). In this regard, features identified in association with the camera and features identified in association with the LiDAR sensor can be combined or fused into a unified feature representation that represents features associated with both the camera and the LiDAR sensor.

[0036] The unified feature representation can represent features in any number of perspectives or spaces. Generally, different features can exist in different views. For example, camera features can exist in a perspective view, while LiDAR features can exist in a bird’s eye view (BEV). Moreover, camera features can correspond to different perspectives (e.g., front, back, left, right). This view disparity presents a challenge in generating a unified feature representation as the same element in different feature tensors corresponding to different spatial locations.

[0037] Accordingly, to generate the unified feature representation, the feature representation generator 106 can convert the features to a single perspective or space. The particular perspective or space for the unified feature representation can be selected to reduce or minimize information loss and be suitable for different types of tasks. In this regard, where LiDAR features and camera features are to be represented in the unified feature representation, the unified feature representation can take the form of a bird’s eye view (BEV), also referred to as a top-down view. For example, features associated with various sensors (e.g., LiDAR and camera) can be fused or aggregated in a unified BEV space or perspective to generate the unified feature representation. In this way, the unified feature representation is constructed in the form of BEV features, integrating data from both camera and LiDAR sensors to provide a comprehensive top-down view of the environment. Generating the unified feature representation in the form of a BEV can more readily discern shapes and orientations. Advantageously, generating the unified feature representation with a BEV preserves both the geometric structure from LiDAR features and the semantic density from camera features. Specifically, LiDAR-to-BEV projection flattens sparse LiDAR features along the height dimension, thereby avoiding geometric distortion, while camera-to-BEV projection projects each camera feature pixel back to a ray in 3D space, thereby generating a dense BEV feature map that preserves full semantic information from the camera. Moreover, since the output space is also in BEV, BEV is generally suitable for various perception tasks.

[0038] To generate a feature representation, such as a unified feature representation, the feature representation generator 106 can acquire and use sensor data 104. Sensor data generally refers to data collected by a sensor, such as the sensors 102. In some cases, the sensor data 104 can be pre-processed so that the data is in a format that can be accepted and processed by the feature representation generator 106. The sensor data 104 can be acquired from any number and type of sensors 102, such as but not limited to LiDAR sensors, cameras, and / or other types of sensors. For example, the sensors 102 can include a camera and a LiDAR sensor, and the sensors 102 can be used to generate sensor data 104 that represents objects in a 3D environment. In some cases, the sensor data 104 can be collected in association with any number of sensors. For example, a single LiDAR sensor and a single camera can capture sensor data for generating a unified feature representation. As another example, a single LiDAR and multiple cameras can be used to capture sensor data for generating a unified feature representation.

[0039] The sensors 102 can be placed in an environment in any of a variety of ways. As one example, the sensors 102 can be placed or mounted on a wall, a ceiling, a pole, or any type of structure to capture or collect data from an environment. Each type of sensor can provide different types of data. For example, a LiDAR sensor can provide precise distance measurements, while a camera can provide rich visual detail. In some cases, a LiDAR sensor and a camera can be placed close to one another.

[0040] By way of example only, a LiDAR sensor and a camera can be placed in an environment to capture sensor data. The LiDAR sensor and the camera can be positioned (e.g., close to one another) and / or oriented to capture the same or similar portions of an environment. In some cases, the sensors can be placed on a wall, a ceiling, or a column to capture an interior or exterior environment. The environment or a portion thereof that is captured can be an environment that is analyzed, for example, to facilitate smart cities, factories, retail, healthcare, etc. In some cases, the sensors are fixed sensors, and thus are positioned and / or oriented in a fixed or stationary manner. Although the examples provided herein generally describe the sensors as being mounted on a non-robotic structure, it can be appreciated that in some implementations one or more of the sensors can be mounted on a robotic machine.

[0041] In addition to being aligned or positioned to capture a particular area or range, the sensors 102 can also be aligned, coordinated, or synchronized in time. In this regard, the sensors can be aligned to maintain a clock, thereby generating synchronized sensor data. For example, a LiDAR sensor and a camera can be synchronized to capture space simultaneously. For example, assume that the LiDAR operates at 30 frames per second, while the camera also operates at 30 frames per second. In this case, the speed of the camera can be reduced, or one out of every three images can be selected to synchronize the captured space. In this way, sensors such as LiDAR and cameras can be synchronized with each other in time and space.

[0042] According to acquiring sensor data 104, such as data from LiDAR and cameras, the feature representation generator 106 can project the sensor data into a common space or perspective, such as a BEV space. Projecting sensor data into a BEV space can be performed differently based on the sensor data. For example, for LiDAR data, a LiDAR point cloud can be projected onto a two-dimensional grid representing a ground plane map. This projection can include converting the three-dimensional coordinates of each point to two-dimensional coordinates (x, y), and accumulating the height (z) or other attributes (e.g., intensity, reflectance, etc.) in the grid cells. For camera data, image features can be projected into the BEV space using geometric transformations.

[0043] According to the sensor data being projected into a particular or common space, such as a BEV space, features can then be extracted. For example, a convolutional neural network can be applied to the projected data to extract feature maps. In some cases, the feature extraction process generates multi-channel feature maps, where each channel captures a different aspect of the sensor data. The extracted features from different sensors can then be combined to generate a unified feature representation, such as a feature map. Combining the extracted features can be done in any of a variety of ways, such as performing concatenation, using attention mechanisms, using neural network-based fusion techniques, etc.

[0044] The feature representation 108 (such as the unified feature representation) generated via the feature representation generator 106 can take any of a variety of forms. As one example, the feature representation 108 (e.g., the unified feature representation) can take the form of a feature map, such as a BEV feature map. A feature map generally refers to a representation that encodes various characteristics or features of the input data. These features can include edges, textures, shapes, and other patterns or data that can be valuable for object detection. A BEV feature map can provide an aerial view of the environment, simplifying spatial relationships between objects and the ground plane. As noted above, this perspective can be helpful in understanding the layout of objects and their surrounding environment. In some cases, the BEV feature map contains multiple channels, each of which represents a different type of information, such as height, intensity, velocity, visual features, and the like. The channels can also encode features extracted at different levels of abstraction, thereby capturing both low-level details and high-level semantics.

[0045] Although a high-level approach for generating the feature representation 108 (such as the unified feature representation) is provided in association with the feature representation generator 106, any number of implementations or approaches can be used. For example, as described in greater detail below, Figure 2 One example implementation that can be used to generate a unified feature representation in accordance with the embodiments described herein is provided.

[0046] Turning to the three-dimensional object detector 110 in Figure 1 The three-dimensional object detector 110 is generally configured (e.g., programmed) to detect three-dimensional objects in an environment. In this regard, the three-dimensional object detector 110 identifies an enclosing shape that corresponds to an object in the environment. The enclosing shape can be used to define the location of the object in an image or representation of the environment. In some examples, the enclosing shape can be a rectangle, box, or cuboid shape, but is not limited thereto. As noted above, the enclosing shape can be represented via spatial parameters that indicate the location, size, and orientation of the enclosing shape that corresponds to the object in the environment.

[0047] The three-dimensional object detector 110 can contain any number of components to perform or carry out the functionality described herein. For example, the three-dimensional object detector 110 can include a feature representation obtainer 112, a spatial parameter generator 114, and a post-processor 116.

[0048] The feature representation obtainer 112 is generally configured (e.g., programmed) to obtain a feature representation, such as the feature representation 108. In accordance with the embodiments described herein, the feature representation obtainer 112 obtains a unified feature representation. For example, the unified feature representation can represent features associated with a LiDAR sensor and features associated with a camera in a single, cohesive representation, such as a BEV feature map.

[0049] The feature representation obtainer 112 can obtain the feature representations in any number of ways. For example, depending on how the feature representations 108 are generated, the feature representation generator 106 can directly provide the generated feature representations 108 to the feature representation obtainer 112. As another example, depending on how the feature representations 108 are generated, such feature representations can be stored in a data store for subsequent access. In this way, the feature representation obtainer 112 can obtain, access, or retrieve the feature representations from such a data store. In this case, the feature representations can be obtained in real-time, streamed, or alternatively in batched fashion.

[0050] The spatial parameter generator 114 is generally configured (e.g., programmed) to generate spatial parameters. As described herein, a spatial parameter generally refers to a parameter that describes or indicates a spatial attribute of an object in an environment. Such spatial parameters include location parameters, dimension parameters, and orientation parameters associated with a three-dimensional environment. In this way, the location parameters, dimension parameters, and orientation parameters can be used to characterize or indicate an enclosing shape corresponding to an object. The location parameters can include location parameters associated with x, y, and z coordinates. Such coordinates can correspond to any portion of the enclosing shape, such as the center of the enclosing shape. In some cases, the location coordinates can represent a location relative to a reference frame (e.g., the location of a sensor).

[0051] The dimension parameters generally define the physical extent or size of the enclosing shape along three axes (length, width, and height). The dimension parameters can include dimension parameters associated with the length of the enclosing shape, the width of the enclosing shape, and the height of the enclosing shape. These dimensions can be expressed using any unit of measurement.

[0052] The orientation parameters generally refer to angles associated with the rotation of the enclosing shape about or around certain axes (e.g., roll angle, pitch angle, yaw angle). The orientation parameters can include: an orientation or rotation angle of the enclosing shape that defines the rotation of the enclosing shape about a vertical axis (e.g., y-axis); an orientation or rotation angle of the enclosing shape that defines the rotation of the enclosing shape about a horizontal axis (e.g., x-axis); and an orientation or rotation angle of the enclosing shape that defines the rotation of the enclosing shape about a depth axis (e.g., z-axis). This angle or rotation angle generally describes the rotation of the enclosing shape about a particular axis, indicating which direction the object is facing. These rotation angles can also be referred to as roll angles, pitch angles, and yaw angles. In some cases, the orientation or rotation angle can be expressed in terms of sine and cosine components. Specifically, the orientation, i.e., the rotation about an axis (typically expressed as an angle), can be expressed using the sine and cosine of the rotation angle, for example, to avoid issues of discontinuity and ambiguity. This approach is more robust and enables the model to learn the orientation in a more continuous manner.

[0053] To generate the spatial parameters, the spatial parameter generator 114 can use or access an object detection model 118 (or spatial parameter model) that outputs a set of spatial parameters describing or indicative of an object in a three-dimensional space (e.g., captured via sensors such as cameras and lidar). The object detection model 118 can take any number of forms, e.g., in the manner of applying or incorporating artificial intelligence (AI) techniques. For example, the object detection model 118 can be one or more machine learning models, deep learning models, neural networks, etc. In one embodiment, the object detection model 118 can be a deep neural network (e.g., a convolutional neural network such as Faster R-CNN) that processes a feature representation (e.g., fused BEV data) to detect a set of spatial parameters corresponding to an object in the environment, incorporating various convolutional layers. For example, the object detection model 118 can process input data to propose candidate regions, refine spatial parameters, and / or assign a confidence score, resulting in an output (e.g., a tensor output) encapsulating such information for various identified objects.

[0054] The spatial parameters output from the object detection model 118 can take any number of forms. In one example, the output is in the form of a tensor containing such location parameters, size parameters, and orientation parameters. For example, in the case that the object detection model 118 detects multiple objects, the output tensor can have a structure or shape like (N, 9), where N is the number of detected objects. In this way, the contents of tensor[i] are reflected as [x i ,y i ,z i ,l i ,w i ,h i ,ψ i ,θ i , and φ i ] for the i-th detected object. Thus, for each detected object i, the tensor contains nine parameter values representing its coordinates (x center coordinate, y center coordinate, and z center coordinate), size (length, width, and height), and orientation (yaw angle, pitch angle, and roll angle). As noted above, in some cases the yaw angle, pitch angle, and roll angle are represented using sine and cosine values. In this way, in the case that the spatial parameter model detects multiple objects, the output tensor can have a structure or shape like (N, 12), where N is the number of objects. Thus, for the i-th detected object, the contents of tensor[i] are reflected as [x i ,y i ,z i ,l i, w i ,h i ,sin(ψ i ),cos(ψ i), sin(0 i ), cos(0 i ), sin(0 i ) and cos(0 i )] Thus, for each detected object i, the tensor includes 12 values representing the center coordinates, dimensions, and orientation (in terms of the sines and cosines of the yaw, pitch, and roll angles) of the enclosing shape.

[0055] In some cases, the object detection model 118 can also output a confidence score or class probability that indicates the likelihood that a detected object belongs to a certain class (e.g., human). In other words, the confidence score indicates the confidence of the spatial parameter model that the enclosing shape contains an object of interest. Thus, the confidence score can help filter out detections with low confidence. In some embodiments, the class score can indicate a single class (which can also be referred to as a binary classification), which provides an indication of whether a particular object type (e.g., human) is present. For example, for the class of humans, a higher class score (e.g., close to 1) can indicate a higher confidence that a human is present in the enclosing shape, while a lower score (e.g., close to 0) can indicate a lower confidence or that a human is not present. In other embodiments, there can be multiple object classes. In such cases, the class score can represent the probability of each possible class. For example, in a multi-class application that includes the classes of humans, vehicles, and animals, the class score associated with an enclosing shape can indicate a probability distribution over the three classes. The class with the highest score can be considered to represent the predicted class of the enclosing shape.

[0056] As described above, to generate spatial parameters in association with objects, the object detection model 118 can take as input a feature representation 108 (such as a unified feature representation associated with sensor data captured in association with sensors (e.g., cameras and LiDARs)). Based on this input, spatial parameters (such as a plurality of values (e.g., 12 values) representing coordinates, dimensions, and orientation associated with an enclosing shape corresponding to an object) can be provided as output.

[0057] In some embodiments, the object detection model 118 generates candidate regions or proposal regions that are identified as likely to contain objects. In some cases, a region proposal network (RPN) or other similar technique can be used to identify such candidate regions or proposal regions that are likely to contain objects. To this end, the feature representations (e.g., feature maps) can be fed into the RPN, which then slides over these feature representations to propose regions (or anchors) that are likely to contain objects. For example, the network can slide over the feature map to operate on each spatial location in the feature map. The candidate regions can be identified based on the extracted features that highlight potential object locations. In this way, candidate regions or anchor boxes can be generated. A candidate region or anchor typically refers to a reference region or box used to predict the presence and location of an object. In some cases, multiple anchor boxes can be generated for each location on the sliding window. These anchor boxes (or other shapes) can be predefined and have different scales and aspect ratios to cover various object sizes and shapes that can exist.

[0058] For various candidate regions, the RPN can predict an objectness score that measures how likely or probable the candidate region contains an object. The objectness score can help to distinguish between background and potential objects. The RPN can also generate or predict an adjustment or offset to the candidate region (e.g., anchor box) to better fit the potential object. For example, the RPN can predict four coordinates for each anchor box that indicate an offset to adjust the anchor to better fit the potential object. In some cases, the candidate region with the highest score, or the region with the highest objectness score, can be selected as the candidate region to be proposed. The number of candidate regions can vary and can be predetermined. In some cases, non-maximum suppression (NMS) is applied to the candidate regions to reduce redundancy in object detection. The spatial parameters associated with the proposed candidate regions generated by the RPN can be designated or considered as regions that are more likely to contain objects.

[0059] When the RPN generates the candidate regions, the candidate regions (e.g., four values representing an anchor box, such as two diagonal point values or other indications of the location and size of the candidate region) can then be provided to the head neural network. In this way, the head neural network can obtain the representation of the candidate regions (e.g., in the form of a feature map that was processed by the RPN) as input. Such a feature map can contain summarized information. Using the candidate regions, the head neural network can predict more accurate bounding shape coordinates for each proposal. In this way, the location and size of the bounding shape can be refined to better fit the detected object. Furthermore, the head neural network can further process the representation of the candidate regions to predict the orientation of the object.

[0060] More specifically, to generate the size, dimensions, and orientation of the detected object, the head neural network can operate through a series of layers. In one example, the head neural network uses region of interest (ROI) pooling or ROI alignment to extract a feature map corresponding to each candidate region. Such operations ensure that the extracted features have a fixed size so as to be able to be processed by fully connected layers. For size and dimensions, the head neural network can perform bounding shape regression. For example, the head neural network can take as input a fixed size feature map and use a fully connected layer to predict offsets relative to the candidate region proposed by the RPN. These offsets adjust the size and position of the anchor shape (e.g., a box) to tightly fit the detected object. This bounding shape regression performs a finer regression than the regression discussed with respect to the RPN, specifically, the candidate region is taken from the RPN and new offsets are predicted to further adjust the bounding shape and fine-tune the candidate region to closely match the actual object boundary. The head neural network can use additional context and information from the feature representation (such as a feature map) to make these adjustments more accurate. For orientation, the head neural network can use an additional regression layer to predict the angle or rotation of the object. Advantageously, the head neural network can predict the sine and cosine components associated with the rotation about each axis, thereby predicting two independent components for each degree of freedom of orientation. In this way, the head neural network can generate 12 spatial parameters, such as position parameters representing 9 degrees of freedom, dimension parameters, and orientation parameters. In some cases, the head neural network can perform regression for individual spatial parameters. In other cases, the head neural network can perform combined regression to predict size, position, and orientation concurrently. During the refinement process, the individual spatial parameters (e.g., 12 spatial parameters) are regressed to obtain a better fitting bounding shape.

[0061] In association with the spatial parameter prediction for the bounding shape, a class label can also be determined and / or assigned, which indicates the type of object represented by the bounding shape. For example, a confidence score can be provided for the bounding shape that reflects or indicates the likelihood that the bounding shape contains an object of the predicted class. In some cases, a softmax layer can be used to assign class probabilities.

[0062] In some cases, the head neural network can apply non-maximum suppression (NMS). For example, NMS can be applied to select a single best bounding shape for each object. For example, the degree of overlap between bounding shapes can be compared and the bounding shape with the highest degree of overlap can be suppressed.

[0063] In the example object detection model 118 described above, the object detection model 118 includes multiple networks, such as an RPN and a head neural network. These components can be part of a faster R-CNN. In some examples, these networks perform different functions in the object detection pipeline (e.g., the RPN generates coarse candidate regions, while the head neural network refines the candidate regions into final bounding shapes). In implementations, any number of networks can be used. For example, the object detection model 118 can include an integrated or single-stage approach, in which the functions performed by the RPN and head neural network are performed in a single network that can perform proposal generation and refinement simultaneously. Although examples are provided herein, the object detection model 118 is not intended to be limited thereto and can be or use any type of technology. By way of example only, an object detection model used to generate spatial parameters can include a single shot multibox detector (SSD) (e.g., using Inception V2, and optimized using TensorRT), you only look once (YOLO), etc.

[0064] Further, although the object detection model 118 is provided separately from the feature representation generator 106, the model can include aspects of both feature representation and object detection as described herein. For example, one portion of layers of the model can be used to perform feature extraction, while another portion of layers of the model can be used to perform object detection.

[0065] In some embodiments, the spatial parameter generator 114 or other component can facilitate training of the object detection model 118. Training the object detection model helps to generate suitable spatial features that represent the enclosing shapes associated with objects. To train the object detection model 118, ground truth spatial parameters are obtained or generated and used for training. Ground truth spatial parameters generally refer to labels or annotations that provide reference data for spatial measurements. According to embodiments described herein, the ground truth spatial parameters can include various position parameters, size parameters, and orientation parameters. As one example, the ground truth spatial parameters can include an x-position label, a y-position label, a z-position label, a length label, a width label, a depth label, a rotation angle about the x-axis, a rotation angle about the y-axis, and a rotation angle about the z-axis. As another example, the ground truth spatial parameters can include an x-position label, a y-position label, a z-position label, a length label, a width label, a depth label, a sine of a rotation angle about the x-axis, a cosine of a rotation angle about the x-axis, a sine of a rotation angle about the y-axis, a cosine of a rotation angle about the y-axis, a sine of a rotation angle about the z-axis, and a cosine of a rotation angle about the z-axis. As described above, such ground truth spatial parameters represent the spatial parameters associated with the enclosing shapes corresponding to objects. In addition to the ground truth spatial parameters, the ground truth labels or annotations can also include respective class labels, e.g., class labels that indicate the types or classes of objects.

[0066] At a high level, the training process uses the ground truth labels to teach the object detection model 118 to generate spatial parameters, including position parameters, size parameters, and orientation parameters. In this way, the object detection model 118 can learn to predict the enclosing shapes of various objects and the classes associated therewith. For example, the generated unified feature representation can be used to predict the position parameters, size parameters, and orientation parameters. These predictions are then compared to respective ground truth labels (e.g., position, size, and orientation ground truth labels) to adjust the object detection model parameters and improve their accuracy. According to embodiments described herein, when training or optimizing the object detection model, the spatial parameters, including the orientation parameters associated with rotations about the x-axis, y-axis, and z-axis, are used to optimize a loss function. In some embodiments, the trained orientation parameters include sine components and cosine components. For example, instead of directly using angles to represent orientations, each orientation parameter (e.g., associated with an axis) can be represented using a sine component and a cosine component associated with a respective angle or rotation about the axis, thereby transforming the orientation associated with the axis into two separate values that the model can learn more effectively.

[0067] When applying a loss function, an object detection model can be trained to minimize the difference between the predicted parameters and the corresponding ground truth labels. Specifically, the loss function measures the difference between the predicted parameters generated by the object detection model and the ground truth parameters. The object detection model can then use this loss to understand its performance and adjust it to minimize error. Examples of loss functions that can be used for training include smoothing L1 loss (Huber loss), L2 loss (mean squared error), and intersection-over-union (IoU) loss.

[0068] In some embodiments, truth labels are synthesized. For example, simulators or graphics engines can be used to generate artificial and realistic images of different environments (e.g., a warehouse) containing various objects (e.g., people). An example of a simulator is NVIDIA. NVIDIA SAAC This is used to provide a highly realistic and scalable simulation environment for, for example, developing, testing, and training robots and autonomous systems. Using synthetically generated images, spatial parameters associated with various objects can be known or predefined. Thus, for each object, its position, size, and orientation (including three rotational degrees of freedom) relative to the image and LiDAR point cloud pair can be known (e.g., via the code that generated the graphics). This avoids the need for manual annotation of ground-value spatial parameters.

[0069] Post-processor 116 is typically configured (e.g., programmed to) refine, filter, and / or interpret the results output by spatial parameter generator 114 or object detection model 118. In this way, post-processor 116 can apply various techniques to the output to refine, filter, and / or interpret the results, thereby transforming the output into meaningful detection results that can be used in practical applications. The post-processor can execute any number of techniques to perform various tasks.

[0070] According to the embodiments described herein, post-processor 116 can be configured (e.g., programmed to) convert the sine and cosine components back to rotation angles to represent the orientation of an object (e.g., for each orientation associated with a rotation axis). In this respect, the axis orientation represented by two components (e.g., the sine and cosine of the rotation angle) can be converted or transformed to represent the axis orientation via a single angle representing the magnitude of rotation about the axis. In this respect, six orientation parameters representing three degrees of freedom can be converted into three orientation parameters. In one example, this conversion technique can be performed using the "atan2" function, which calculates the angle based on the sine and cosine components as follows:

[0071] [Angle = Text{atan2}(sin(Angle),cos(Angle))]

[0072] ( [angle = atan2 (sin (angle), cos (angle) )])

[0073] Additionally or alternatively, the post-processor 116 can perform various other tasks. For example, the post-processor 116 can delete duplicate detections and keep the best enclosing shape for each object. In some examples, non-maximum suppression can be performed to delete duplicate detections. In this regard, for each detected object class, the enclosing shapes can be sorted by the respective confidence scores. The highest scoring enclosing shape can be selected iteratively, and other enclosing shapes with significant overlap (e.g., using an Intersection over Union (IoU) threshold) can be suppressed to delete duplicates.

[0074] Further, the post-processor 116 can perform enclosing shape adjustment to refine the enclosing shape spatial parameters. To this end, corrections or adjustments can be applied based on additional heuristics or rules to improve the alignment and accuracy of the enclosing boxes.

[0075] The post-processor 116 can also perform confidence thresholding to filter out low confidence detections. For example, assume a confidence threshold is set. Any enclosing shape associated with a confidence score below the threshold will be discarded or removed to reduce false positives.

[0076] Other post-processing techniques or tasks that the post-processor 116 can perform include, for example, assigning class labels, performing clustering, transforming to global coordinates, performing visualization, and / or temporal smoothing. Assigning class labels to detected objects can be performed by assigning the most likely class label to each detected enclosing box using the class scores from the object detection model 118 output. Clustering can be performed (e.g., for a particular application) to group multiple detections that belong to the same object. In some embodiments, a clustering algorithm (e.g., DBSCAN, MeanShift) can be applied to group proximate detections into a single object representation, which is particularly useful in dense environments. Transforming to global coordinates is applied to convert local coordinates to global coordinates. For example, if the detections are in the coordinate system of a sensor, a pose or transformation matrix can be used to transform the detections to a global coordinate system. Performing visualization is typically applied to generate visual representations of the detections for verification and debugging. In some embodiments, a visual overlay layer (e.g., enclosing boxes on images, points in 3D space) can be created on the original sensor data to help verify the accuracy and performance of the detections. Temporal smoothing can be performed to ensure consistency of the detections across data frames. For example, temporal smoothing techniques can be applied to reduce jitter and improve the stability of the detections over time.

[0077] Post-processing in three-dimensional object detection is critical to refine the raw output from the spatial parameter generator 114 and / or the object detection model 118. Techniques for performing angle conversion, NMS, enclosing shape adjustment, confidence thresholding, and / or class label assignment ensure that the final detection results, such as spatial parameters, are accurate and reliable. Techniques for performing clustering, global coordinate transformation, visualization, temporal smoothing, and / or sensor data aggregation further enhance the quality and applicability of the detection results in real-world scenarios. Thus, the processes that can be performed by the post-processor 116 ensure that the three-dimensional object detector 110 performs well and produces results that are suitable for practical applications such as autonomous driving, robotics, and augmented reality.

[0078] In this way, the three-dimensional object detector 110 generates a representation of an enclosing shape corresponding to an object (e.g., a person, a machine, etc.). Such an enclosing shape can be represented using the output or refined spatial parameters, including a nine-degree-of-freedom representation (e.g., three position representations, three size representations, and three orientation representations). Advantageously, representing the enclosing shape in nine degrees of freedom, including three orientation representations associated with three axes in three-dimensional space, can provide a more comprehensive and more accurate description of the rotation and orientation of the object and reduce or eliminate ambiguities that can arise from a more limited representation. For example, orientation parameters that include all three axes can ensure that even the smallest rotation is accurately captured and represented, allowing for more precise control and manipulation of the object (e.g., in robotics and simulation environments). As another example, using three axes for orientation ensures that the transformation is consistent and predictable, which can be valuable for tasks such as animation, physics simulation, and navigation. The enclosing shape representation can also include a class associated with the enclosing shape or an object associated therewith.

[0079] Such a representation of an enclosing shape can be used in various environments, including robotics environments (e.g., robotic arms, drones, and autonomous vehicles). For example, assume that a robot is navigating within a warehouse and sensors are distributed around the warehouse. Thus, generating or determining a representation of an enclosing shape in the warehouse can be valuable to monitor various aspects of the warehouse, such as the moving locations of people or robots. For example, knowing the object locations and movements can allow the robot to plan a path (e.g., to avoid congestion). As another example, such a representation of an enclosing shape can be used for traffic monitoring (e.g., monitoring a road intersection) or autonomous vehicle navigation.

[0080] Representations of encompassing shapes can be used to perform a variety of operations. As one example, encompassing shape representations can be used to perform various monitoring and security analysis or operations. For instance, encompassing shape representations can be used for intrusion detection (e.g., to identify and / or track unauthorized individuals) and / or crowd monitoring (e.g., to prevent overcrowding or to reinforce crowd control measures). As another example, encompassing shape representations can be used to perform various traffic management tasks. For instance, such representations can be used to monitor the location and movement of vehicles (e.g., to facilitate real-time traffic management and traffic light optimization), to perform accident detection, etc. As another example, encompassing shape representations can be used to perform robot navigation or interaction tasks. For instance, such representations can be used to plan efficient collision-free paths for robots, to identify or locate objects that a robot can need to move or interact with, etc. Other examples include public safety and emergency response tasks, city planning and management tasks, environmental monitoring tasks, retail and business analysis tasks, AR / VR tasks, etc.

[0081] Turning to Figure 2 , Figure 2 One example implementation that can be used to generate a unified feature representation is provided in accordance with embodiments described herein. In this example, various features are extracted from multi-modal inputs and converted into a unified feature representation in the form of a shared BEV space (e.g., using view transformations). The unified BEV features can be fused with a fully convolutional BEV encoder. More specifically, camera images 202 can be encoded via a camera encoder 204 to extract camera features 206. In this way, the camera encoder 204 (e.g., a neural network or other algorithm) processes the images (e.g., raw images) to produce a set of camera features 206. The camera images can be generated via cameras that are mounted or placed in the environment (e.g., affixed to a wall / ceiling / pole, etc.). At block 208, the camera features are transformed into a BEV view to produce a set of camera features in a BEV 210. Transforming the camera features into a BEV to produce a set of camera features in a BEV can include performing a technique that is capable of projecting 2D image features onto a 3D plane that simulates a top-down perspective.

[0082] For the LiDAR point cloud 214, the LiDAR point cloud 214 can be encoded via a LiDAR encoder 216 to extract LiDAR features 218. In this way, the LiDAR encoder 216 (e.g., a neural network or other algorithm) processes the point cloud to produce a set of LiDAR features 218. For example, the LiDAR encoder 216 can transform the raw point cloud data into a more compact and informationally richer representation. Such LiDAR point clouds can be generated via a LiDAR mounted or placed in the environment (e.g., fixed to a wall / ceiling / pole, etc.). At block 220, the LiDAR features are flattened (e.g., along the z-axis) to produce LiDAR features in the BEV 222. The camera features in the BEV 210 and the LiDAR features in the BEV 222 are aggregated, as shown at 224. A BEV encoder 226 performs encoding to generate a set of fused BEV features 228, generating a unified feature representation. This set of fused BEV features 228 is provided as the unified feature representation to a three-dimensional object detector 230. In some embodiments, the three-dimensional object detector 230 is configured similarly to the three-dimensional object detector 110.

[0083] Referring now to Figures 3 to 5 Each block of the methods 300, 400, and 500 described herein contains one or more processes that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. The methods can also be embodied as computer-usable instructions stored on computer storage media. The methods can be provided by a standalone application, a service or hosted service (standalone or in combination with other hosted services), a plug-in to another product, or some other entity / combination, just to name a few. Further, the methods 300, 400, and 500 can be described in the general context of a system, which can include software applications, services, hosted services, or other products that execute on computer storage media. Figure 1 However, these methods can also be performed additionally or alternatively by any one system or any combination of systems, including but not limited to the systems described herein.

[0084] Figure 3is a flowchart illustrating a method 300 for generating a bounding shape representation of an object according to some embodiments of the present disclosure. The method 300 includes, at block B302, obtaining a representation of features associated with one or more sensors. In some embodiments, the representation of features includes a unified feature representation that aggregates features associated with a LiDAR sensor and features associated with a camera in an environment. The LiDAR sensor and the camera can be placed at different locations. For example, the camera and the LiDAR sensor are mounted on a fixed structure in an environment with limited field of view (e.g., indoors or outdoors). Such an environment can be fixed in space and contain any number of objects that move dynamically within the space. The objects can also be static and do not need to move in the space. In some embodiments, the unified feature representation corresponds to an overhead view.

[0085] The method 300 includes, at block B304, generating a representation of a bounding shape corresponding to an object in the environment based at least on the representation of features associated with the one or more sensors, including a plurality of orientation parameters. In some cases, the representation of the bounding shape includes an x-coordinate, a y-coordinate, a z-coordinate, a length, a width, a height, a yaw angle, a pitch angle, and a roll angle. In other cases, the representation of the bounding shape includes an x-coordinate, a y-coordinate, a z-coordinate, a length, a width, a height, a sine of a rotation angle about the x-axis, a cosine of the rotation angle about the x-axis, a sine of a rotation angle about the y-axis, a cosine of the rotation angle about the y-axis, a sine of a rotation angle about the z-axis, and a cosine of the rotation angle about the z-axis.

[0086] The representation of the bounding shape can be generated via an object detection model (e.g., the object detection model 118). Such an object detection model can be a neural network with one or more layers for predicting the plurality of orientation parameters associated with the bounding shape. To detect the plurality of orientation parameters, the object detection model can be trained using synthetic space parameters representing nine degrees of freedom, including an orientation associated with the x-axis, an orientation associated with the y-axis, and an orientation associated with the z-axis. In some cases, the representation of the bounding shape can be generated or identified by predicting, via the object detection model, a set of initial space parameters including parameters representing sine and cosine components of rotation angles about the x-axis, the y-axis, and the z-axis. Thereafter, a post-processor (e.g., the post-processor 116) can generate the plurality of orientation parameters representing rotation angles about the x-axis, the y-axis, and the z-axis based on the set of initial space parameters.

[0087] The method 300 includes, at block B306, performing one or more operations corresponding to the environment based at least on the representation of the bounding shape. Any operation can be performed, including, for example, operations associated with analyzing the environment.

[0088] Figure 4 is a flowchart of a method 400 for generating a bounding shape representation of an object according to some embodiments of the present disclosure. The method 400 includes, at block B402, generating a representation of a bounding shape corresponding to an object in an environment based at least on a representation of features associated with one or more sensors mounted or positioned in the environment, the representation of the bounding shape including a plurality of orientation parameters. In some embodiments, the environment can include a static background with dynamic objects present. The representation of the features can be in any number of formats, such as a unified representation of features captured by LiDAR sensors and cameras. In some embodiments, the orientation parameters can include a first parameter indicating a first angle of rotation about a first axis, a second parameter indicating a second angle of rotation about a second axis, and a third parameter indicating a second angle of rotation about a third axis.

[0089] The method 400 includes, at block B404, performing one or more operations corresponding to the environment based at least on the representation of the bounding shape. Any operation can be performed, including, for example, operations associated with analyzing the environment.

[0090] Figure 5 is a flowchart illustrating a method 500 for generating a bounding shape representation of an object according to some embodiments of the present disclosure. The method 500 includes, at block B502, obtaining a representation of features associated with one or more sensors in an environment as input to a model. In some embodiments, the model can be trained using synthetically generated ground truth orientation parameters associated with x, y, and z axes.

[0091] The method 500 includes, at block B504, generating a representation of a bounding shape including a plurality of orientation parameters based on the input, the bounding shape corresponding to an object in the environment. In some embodiments, the plurality of orientation parameters can include a first representation of a first angle of rotation about a first axis, a second representation of a second angle of rotation about a second axis, and a third representation of a third angle of rotation about a third axis. In some cases, the representations can be angles of rotation (e.g., an angle of rotation about the x-axis, an angle of rotation about the y-axis, and an angle of rotation about the z-axis). In other cases, the representations can include sine and cosine components of the angles of rotation.

[0092] The systems and methods described herein can be used by non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driving assistance systems [ADAS]), manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles connected to one or more trailers, aircraft, watercraft, space shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, airplanes, engineering vehicles, trains, underwater vehicles, remotely operated vehicles such as drones, and / or other types of vehicles, without limitation. Moreover, the systems and methods described herein can be used for various uses such as, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twin, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twin, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, and / or any other suitable application.

[0093] The disclosed embodiments can be included in various different systems such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines, etc.), systems implemented using robots, aviation systems, medical systems, boating systems, smart area surveillance systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems for performing teleoperations, systems for performing real-time streaming, systems for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content, systems implementing one or more large language models, systems implementing one or more visual language models, systems implementing one or more multi-modal language models, systems for generating synthetic data, systems for generating synthetic data using AI, systems including one or more virtual machines, systems implemented at least in part in a data center, systems for performing conversational AI operations, systems for performing light transport simulation, systems for performing collaborative content creation of 3D assets, systems implemented at least in part using cloud computing resources, and / or other types of systems.

[0094] Example computing device

[0095] Figure 6A block diagram of an example computing device 600 suitable for implementing some embodiments of the present disclosure is shown. The computing device 600 can include an interconnection system 602 coupling the following components: a memory 604, one or more central processing units (CPU) 606, one or more graphics processing units (GPU) 608, a communication interface 610, an input / output (I / O) port 612, an input / output component 614, a power supply 616, one or more presentation components 618 (e.g., a display), and one or more logic units 620. In at least one embodiment, the computing device 600 can include one or more virtual machines (VMs), and / or any component thereof can include a virtual component (e.g., a virtual hardware component). For a non-limiting example, the one or more GPUs 608 can include one or more vGPUs, the one or more CPUs 606 can include one or more vCPUs, and / or the one or more logic units 620 can include one or more virtual logic units. Thus, the computing device 600 can include discrete components (e.g., a full GPU dedicated to the computing device 600), virtual components (e.g., a portion of a GPU dedicated to the computing device 600), or a combination thereof.

[0096] Although Figure 6 various blocks of the computing device are shown as being connected via the interconnection system 602 having a bus, this is not intended to be limiting and is for clarity only. For example, the presentation component 618, such as a display device, can be considered an I / O component 614 (e.g., if the display is a touch screen). As another example, the CPU 606 and / or GPU 608 can include memory (e.g., the memory 604 can represent a storage device in addition to the memory of the GPU 608, CPU 606, and / or other components). In other words, the components of the computing device are merely illustrative and not restrictive. Figure 6 The computing device is merely illustrative of one implementation and is not intended to limit the scope of use or functionality of any implementation of the present disclosure. Neither should the computing device be interpreted as having any dependency or requirement relating to one or more components included in the computing device. Figure 6 The computing device is merely illustrative of one implementation and is not intended to limit the scope of use or functionality of any implementation of the present disclosure. Neither should the computing device be interpreted as having any dependency or requirement relating to one or more components included in the computing device.

[0097] The interconnection system 602 can represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 602 can include one or more link or bus types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU 606 can be directly connected to the memory 604. Also, the CPU 606 can be directly connected to the GPU 608. Where there are direct or point-to-point connections between components, the interconnection system 602 can include a PCIe link to perform the connection. In these examples, a PCI bus need not be included in the computing device 600.

[0098] The memory 604 can include any of a wide variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computing device 600. Computer-readable media can include both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media.

[0099] Computer storage media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, and / or other data types. For example, the memory 604 can store computer readable instructions such as those representing programs and / or program elements, e.g., an operating system. Computer storage media can include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computing device 600. Computer storage media, as used herein, does not include signals per se.

[0100] Computer storage media can include computer-readable instructions, data structures, program modules, and / or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" can refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of the above should also be included within the scope of computer readable media.

[0101] The CPUs 606 can be configured to execute at least some of the computer-readable instructions in order to control one or more components of the computing device 600 to perform one or more of the methods and / or processes described herein. Each of the CPUs 606 can include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling a large number of software threads concurrently. The CPUs 606 can include any type of processors and can include different types of processors depending on the type of computing device 600 being implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 600, the processors can be Advanced RISC Machines (ARM) processors implemented using reduced instruction set computing (RISC) or x86 processors implemented using complex instruction set computing (CISC). The computing device 600 can include one or more CPUs 606 in addition to one or more microprocessors or complementary co-processors such as math co-processors.

[0102] In addition or alternatively to CPU 606, GPU 608 can be configured to execute at least some computer-readable instructions to control one or more components of computing device 600 to perform one or more methods and / or processes described herein. GPU(s) 608 can be integrated GPUs (e.g., with CPU(s) 606) and / or GPU(s) 608 can be discrete GPUs. In embodiments, GPU(s) 608 can be co-processors to CPU(s) 606. Computing device 600 can use GPU(s) 608 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPU(s) 608 can be used for general-purpose computing on GPUs (GPGPU). GPU(s) 608 can include hundreds or thousands of cores capable of processing hundreds or thousands of software threads concurrently. GPU(s) 608 can generate pixel data for output images in response to rendering commands (e.g., rendering commands from CPU(s) 606 received via a host interface). GPU(s) 608 can include graphics memory, such as display memory, for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory can be included as part of memory 604. GPU(s) 608 can include two or more GPUs operating in parallel (e.g., via a link). The link can connect the GPUs directly (e.g., using NVLINK) or through a switch (e.g., using NVSwitch). When combined together, each GPU 608 can generate different portions of pixel data or GPGPU data for output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can include its own memory or can share memory with other GPUs.

[0103] In addition or alternatively to CPU 606 and / or GPU 608, logic unit(s) 620 can be configured to execute at least some computer-readable instructions to control one or more components of computing device 600 to perform one or more methods and / or processes described herein. In embodiments, CPU(s) 606, GPU(s) 608, and / or logic unit(s) 620 can execute any combination of methods, processes, and / or portions thereof discretely or jointly. Logic unit(s) 620 can be part of and / or integrated with CPU(s) 606 and / or GPU(s) 608 and / or logic unit(s) 620 can be discrete components of or otherwise external to CPU(s) 606 and / or GPU(s) 608. In embodiments, logic unit(s) 620 can be processors of CPU(s) 606 and / or GPU(s) 608.

[0104] Examples of logic units 620 include one or more processing cores and / or components thereof, such as data processing units (DPUs), tensor cores (TCs), tensor processing units (TPUs), pixel visual cores (PVCs), visual processing units (VPUs), graphics processing clusters (GPCs), texture processing clusters (TPCs), streaming multi-processors (SMs), tree traversal units (TTUs), artificial intelligence accelerators (AIAs), deep learning accelerators (DLAs), arithmetic logic units (ALUs), application specific integrated circuits (ASICs), floating point units (FPUs), input / output (I / O) elements, peripheral component interconnects (PCI) or peripheral component interconnect express (PCIe) elements, and the like.

[0105] Communication interface 610 can include one or more receivers, transmitters, and / or transceivers that enable computing device 600 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communication. Communication interface 610 can include components and functionality enabling communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, and the like), wired networks (e.g., communication over Ethernet or InfiniBand), low power wide area networks (e.g., LoRaWAN, SigFox, and the like), and / or the Internet. In one or more embodiments, logic units 620 and / or communication interface 610 can include one or more data processing units (DPUs) to transfer data received over a network and / or over interconnect system 602 directly to one or more GPUs 608 (e.g., memory thereof).

[0106] I / O ports 612 can enable the computing device 600 to logically couple to other devices including I / O components 614, presentation components 618, and / or other components, some of which can be built in to (e.g., integrated in) the computing device 600. Illustrative I / O components 614 include a microphone, mouse, keyboard, joystick, game pad, game controller, satellite dish, scanner, printer, wireless device, etc. The I / O components 614 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs can be transmitted to an appropriate network element for further processing. A NUI can implement any combination of speech recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with a display of the computing device 600. The computing device 600 can include depth cameras, infrared cameras, RGB cameras, touch screens, and combinations of these, such as a stereoscopic camera system to provide a depth map.

[0107] A power supply 616 can include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply 616 can supply power to the computing device 600 to enable the components of the computing device 600 to operate.

[0108] The presentation components 618 can include a display (e.g., a monitor, a touch screen, a television, a heads-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation components 618 can receive data from other components (e.g., the GPU 608, the CPU 606, the DPU, etc.) and output the data (e.g., as images, video, sound, etc.).

[0109] Example data center

[0110] Figure 7 An example data center 700 is shown, which can be used in at least one embodiment of the present disclosure. The data center 700 can include a data center infrastructure layer 710, a framework layer 720, a software layer 730, and an application layer 740.

[0111] As Figure 7As shown, the data center infrastructure layer 710 may include a resource coordinator 712, grouped computing resources 714, and node computing resources (“nodes CR”) 716(1)-716(N), where “N” represents any complete positive integer. In at least one embodiment, nodes CR 716(1)-716(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state drives or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power modules and cooling modules, etc. In some embodiments, one or more node CRs of nodes CR 716(1)-716(N) may correspond to servers having one or more of the aforementioned computing resources. Furthermore, in some embodiments, node CRs

[0112] 716(1)-716(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more of the nodes CR716(1)-716(N) may correspond to virtual machines (VMs).

[0113] In at least one embodiment, the grouped computing resources 714 may include individual groups (not shown) of nodes CR716 housed in one or more racks, or a plurality of racks (also not shown) housed in data centers in various geographic locations. Individual groups of nodes CR716 within the grouped computing resources 714 may include computing, networking, memory, or storage resources that can be configured or allocated to support groups of one or more workloads. In at least one embodiment, several nodes CR716, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0114] Resource coordinator 712 may be configured or otherwise controlled to control one or more nodes CR716(1)-716(N) and / or grouped computing resources 714. In at least one embodiment, resource coordinator 712 may include a Software Design Infrastructure (“SDI”) management entity for data center 700. Resource coordinator 712 may include hardware, software, or some combination thereof.

[0115] In at least one embodiment, such as Figure 7As shown, framework layer 720 may include a job scheduler 733, a configuration manager 734, a resource manager 736, and a distributed file system 738. Framework layer 720 may include a framework for software 732 supporting software layer 730 and / or one or more applications 742 of application layer 740. Software 732 or application 742 may respectively include web-based service software or applications, such as service software or applications provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 720 may be, but is not limited to, a free and open-source software web application framework, such as Apache Spark, which can utilize distributed file system 738 for large-scale data processing (e.g., "big data"). TM (Hereinafter referred to as "Spark"). In at least one embodiment, the job scheduler 733 may include a Spark driver for facilitating the scheduling of workloads supported by various layers of data center 700. In at least one embodiment, the configuration manager 734 may be able to configure different layers, such as software layer 730 and framework layer 720 including Spark and a distributed file system 738 for supporting large-scale data processing. The resource manager 736 is able to manage cluster or grouped computing resources mapped to or allocated to support the distributed file system 738 and the job scheduler 733. In at least one embodiment, the cluster or grouped computing resources may include grouped computing resources 714 at data center infrastructure layer 710. The resource manager 736 may coordinate with the resource coordinator 712 to manage these mapped or allocated computing resources.

[0116] In at least one embodiment, the software 732 included in the software layer 730 may include software used by at least a portion of the nodes CR716(1)-716(N), the grouped computing resources 714, and / or the distributed file system 738 of the framework layer 720. One or more types of software may include, but are not limited to, Internet web page search software, email virus browsing software, database software, and streaming video content software.

[0117] In at least one embodiment, one or more application programs 742 included in application layer 740 can include one or more types of application programs used by at least portions of node C.R.s 716(1)-716(N), grouped computing resources 714, and / or distributed file system 738 of framework layer 720. One or more types of application programs can include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications, including training or inferencing software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0118] In at least one embodiment, any of configuration manager 734, resource manager 736, and resource orchestrator 712 can implement any number and type of self-modification actions based on any number and type of data acquired in any technically feasible manner. Self-modification actions can relieve data center operators of data center 700 from making possibly poor configuration decisions and can avoid underutilized and / or poorly performing portions of a data center.

[0119] Data center 700 can include tools, services, software, or other resources for training one or more machine learning models or using one or more machine learning models to predict or infer information in accordance with one or more embodiments described herein. For example, a machine learning model can be trained in accordance with a neural network architecture by computing weight parameters using software and computing resources described above with respect to data center 700. In at least one embodiment, using weight parameters computed through one or more training techniques, a trained machine learning model corresponding to one or more neural networks can be used to infer or predict information using resources described above with respect to data center 700, such as but not limited to those described herein.

[0120] In at least one embodiment, data center 700 can use CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or virtual computing resources corresponding thereto) to perform training and / or inference using resources described above. Moreover, one or more software and / or hardware resources described above can be configured as a service to allow users to train or perform information inference, such as image recognition, speech recognition, or other artificial intelligence services.

[0121] Example network environment

[0122] Network environments suitable for implementing embodiments of the present disclosure can include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) can be implemented on one or more instances of the computing device 600— e.g., each device can include similar components, features, and / or functionality of the computing device 600. Moreover, where backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices can be included as part of the data center 700, an example of which is described herein with respect to FIG. 7, in more detail. Figure 6 Figure 7

[0123] Components of the network environment can communicate with each other over a network, which can be wired, wireless, or both. The network can include multiple networks, or a network of multiple networks. By way of example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (e.g., the Internet and / or the public switched telephone network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (among other components) can provide wireless connectivity.

[0124] Compatible network environments can include one or more peer-to-peer network environments (in which case servers can not be included in the network environment), as well as one or more client-server network environments (in which case one or more servers can be included in the network environment). In a peer-to-peer network environment, functionality described herein with respect to servers can be implemented on any number of client devices.

[0125] In at least one embodiment, the network environment can include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. A cloud-based network environment can include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which can include one or more core network servers and / or edge servers. The framework layer can include a framework for supporting one or more applications of a software layer and / or an application layer. The software or applications can include network-based service software or applications, respectively. In embodiments, the one or more client devices can use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer can be, without limitation, a type of free and open-source software web application framework, such as can be used for large-scale data processing (e.g., “big data”) using the distributed file system. ​​

[0126] The cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions described herein (or one or more portions thereof). Any of these various functions can be distributed across multiple locations from a central or core server (e.g., across one or more data centers in a state, region, country, globally, etc.). The core server can designate at least a portion of the functions to an edge server if the connection to the user (e.g., client device) is relatively close to the edge server. The cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0127] The client device can include at least some components, features, and functionality of the example computing device 600 described herein with respect to Figure 6 As examples and not by way of limitation, the client device can embody a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smartwatch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a video camera, a surveillance device or system, a vehicle, a watercraft, an aircraft, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or system, an entertainment system, an in-vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of these described devices, or any other suitable device.

[0128] The present disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, being executed by a computer or other machine, such as a personal digital assistant or other handheld devices. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The present disclosure can be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general- purpose computers, more specialty computing devices, and the like. The present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0129] As used herein, the term "and / or," with respect to a listing of two or more elements, means that at least one of the listed elements is present. For example, "A and / or B" is intended to cover A only, B only, or both A and B. Further, "at least one of A or B" is intended to cover at least one of A, at least one of B, or at least one of both A and B.

[0130] The subject matter of the present disclosure is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have contemplated that the claimed subject matter might also be embodied in other ways, to include different steps or combinations of steps similar to the ones described in this document, in conjunction with other present or future technologies. Moreover, although the terms "step" and / or "block" might be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described.

Claims

1. A method comprising: obtaining a representation of features associated with one or more sensors; generating, based at least on the representation of features associated with the one or more sensors, a representation of an enclosing shape corresponding to an object in an environment, the representation of the enclosing shape comprising a plurality of orientation parameters; and performing one or more operations corresponding to the environment based at least on the representation of the enclosing shape.

2. The method of claim 1, wherein, The representation of features comprises a unified feature representation that aggregates features associated with a LiDAR sensor and features associated with a camera in the environment.

3. The method of claim 1, wherein, The representation of features comprises a unified feature representation corresponding to an overhead view of the environment.

4. The method of claim 1, wherein, The environment is fixed in space and comprises at least one of one or more static objects or one or more dynamic objects that move within the space.

5. The method of claim 1, wherein, The representation of the enclosing shape comprises an x-coordinate, a y-coordinate, a z-coordinate, a length, a width, a height, a yaw angle, a pitch angle, and a roll angle.

6. The method of claim 1, wherein, The representation of the enclosing shape comprises an x-coordinate, a y-coordinate, a z-coordinate, a length, a width, a height, a sine of a rotation angle about an x-axis, a cosine of a rotation angle about the x-axis, a sine of a rotation angle about a y-axis, a cosine of a rotation angle about the y-axis, a sine of a rotation angle about a z-axis, and a cosine of a rotation angle about the z-axis.

7. The method of claim 1, wherein, The representation of the enclosing shape is generated using an object detection model that predicts the representation of the enclosing shape based on the representation of features input to the object detection model.

8. The method of claim 1, wherein, The representation of the enclosing shape is generated using an object detection model that comprises a neural network having one or more layers, the neural network configured to predict an orientation associated with an x-axis, an orientation associated with a y-axis, and an orientation associated with a z-axis.

9. The method of claim 1, wherein, The representation of the enclosing shape is generated using an object detection model that comprises a neural network trained using synthetic space parameters representing nine degrees of freedom, the space parameters comprising an orientation associated with an x-axis, an orientation associated with a y-axis, and an orientation associated with a z-axis.

10. The method of claim 1, wherein, The representation of the enclosing shape is generated by: predicting, via an object detection model, a set of initial space parameters comprising parameters representing sine and cosine components of rotation angles about an x-axis, a y-axis, and a z-axis; and generating, via a post-processor, the plurality of orientation parameters representing rotation angles about the x-axis, the y-axis, and the z-axis, the plurality of orientation parameters generated based on the set of initial space parameters.

11. The method of claim 1, wherein, The method is performed using at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing optical transport simulations; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system for performing teleoperations; a system for performing real-time streaming; A system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; A system implemented using edge devices; A system implemented using robots; A system for performing conversational AI operations; A system implementing one or more language models; A system implementing one or more large language models (LLMs); A system implementing one or more visual language models (VLMs); A system implementing one or more multi-modal language models; A system for generating synthetic data; A system for generating synthetic data using AI; A system including one or more virtual machines (VMs); A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

12. One or more processors comprising processing circuitry to: generate, based at least on a representation of features associated with one or more sensors located in an environment, a representation of an enclosing shape corresponding to an object in the environment, the representation of the enclosing shape comprising a plurality of orientation parameters; and perform one or more operations corresponding to the environment based at least on the representation of the enclosing shape.

13. The one or more processors of claim 12, wherein, The environment comprises a static background with dynamic objects.

14. The one or more processors of claim 12, wherein, The representation of the features comprises a unified representation of features captured by a LiDAR sensor and a camera.

15. The one or more processors of claim 12, wherein, The plurality of orientation parameters comprises a first parameter indicating a first angle of rotation about a first axis, a second parameter indicating a second angle of rotation about a second axis, and a third parameter indicating a third angle of rotation about a third axis.

16. The one or more processors of claim 12, wherein, The one or more processors are included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing optical transport simulations; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system for performing teleoperations; a system for performing real-time streaming; A system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; A system implemented using edge devices; A system implemented using robots; A system for performing conversational AI operations; A system implementing one or more language models; A system implementing one or more large language models (LLMs); A system implementing one or more visual language models (VLMs); A system implementing one or more multi-modal language models; A system for generating synthetic data; A system for generating synthetic data using AI; A system including one or more virtual machines (VMs); A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

17. A system comprising one or more processors to: obtain, as input to a deep learning model, a representation of features associated with one or more sensors in an environment; generate, based on the input, a representation of an enclosing shape comprising a plurality of orientation parameters, the enclosing shape corresponding to an object in the environment; and performing one or more operations corresponding to the environment based at least on the representation of the enclosing shape.

18. The system of claim 17, wherein, The deep learning model is trained using synthetically generated ground truth orientation parameters associated with x, y, and z axes.

19. The system of claim 17, wherein, The plurality of orientation parameters includes a first representation of a first angle of rotation about a first axis, a second representation of a second angle of rotation about a second axis, and a third representation of a third angle of rotation about a third axis.

20. The system of claim 18, wherein, The system is included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing optical transport simulation; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system for performing teleoperations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using edge devices; a system implemented using robots; a system for performing conversational AI operations; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system implementing one or more visual language models (VLMs); a system implementing one or more multi-modal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

Images