3D auto-labeling with structural and physical constraints

Abstract

To provide a method for 3D auto-labeling of objects with predetermined structural and physical constraints.SOLUTION: The method includes identifying initial object-seeds for all frames from a given frame sequence of a scene. The method also includes refining each of the initial object-seeds over 2D / 3D data, while complying with the predetermined structural and physical constraints to auto-label 3D object vehicles within the scene. The method further includes linking the auto-labeled 3D object vehicles over time to trajectories while respecting the predetermined structural and physical constraints.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 62 / 935,246, entitled "Automatic Labeling of 3D Objects via Differentiable Rendering of SDF Shape Priors," filed November 14, 2019, the disclosure of which is expressly incorporated by reference in its entirety.

[0002] Certain aspects of the present disclosure relate generally to machine learning, and more specifically to the automated 3D labeling of objects constrained by structural and physical constraints. [Background technology]

[0003] Autonomous actors (e.g., vehicles, robots, etc.) rely on machine vision to sense their surroundings by analyzing areas of interest within images of the surrounding environment. Although scientists have spent decades studying the human visual system, a solution to achieving comparable machine vision remains elusive. Achieving comparable machine vision is the goal for truly autonomous actors. Machine vision differs from the field of digital image processing because it requires recovering three-dimensional (3D) structure on the ground from images and using the 3D structure to achieve a holistic understanding of the scene. That is, machine vision aims to provide a high-level understanding of the surrounding environment, similar to that achieved by the human visual system.

[0004] During operation, an autonomous agent may rely on a trained deep neural network (DNN) to identify objects within an area of interest within an image of a scene surrounding the autonomous agent. For example, the DNN may be trained to identify and track objects captured by one or more sensors, such as a light detection and ranging (LIDAR) sensor, a sonar sensor, a red-green-blue (RGB) camera, an RGB-depth (RGB-D) camera, etc. In particular, the DNN may be trained to understand a scene from video input based on annotations of vehicles within the scene. Unfortunately, annotating video is a challenging task that involves deep understanding of the visual scene. Summary of the Invention

[0005] A method for 3D automatic labeling of objects with predetermined structural and physical constraints includes identifying initial object seeds for all frames from a given sequence of frames of a scene. The method also includes refining each of the initial object seeds on 2D / 3D data while automatically labeling 3D object vehicles in the scene in accordance with the predetermined structural and physical constraints. The method includes associating the automatically labeled 3D object vehicles with trajectories over time while respecting the predetermined structural and physical constraints.

[0006] A non-transitory computer-readable medium has program code recorded thereon, the program code being executed by a processor, for 3D automatic labeling of objects with predetermined structural and physical constraints. The non-transitory computer-readable medium includes program code for identifying initial object seeds for all frames from a given sequence of frames of a scene. The non-transitory computer-readable medium also includes program code for refining each of the initial object seeds on the 2D / 3D data while automatically labeling 3D object vehicles in the scene in accordance with the predetermined structural and physical constraints. The non-transitory computer-readable medium further includes program code for associating the automatically labeled 3D object vehicles with trajectories over time while respecting the predetermined structural and physical constraints.

[0007] A system for 3D automatic labeling of objects with predetermined structural and physical constraints includes an object seed detection module trained to identify initial object seeds for all frames from a given sequence of frames of a scene. The system also includes an object seed refinement module trained to refine each of the initial object seeds on the 2D / 3D data while automatically labeling 3D object vehicles in the scene in accordance with the predetermined structural and physical constraints. The system further includes a 3D automatic labeling module trained to associate the automatically labeled 3D object vehicles with trajectories over time while respecting the predetermined structural and physical constraints.

[0008] This has outlined rather broadly the features and technical advantages of the present disclosure in order that the detailed description that follows may be more fully understood. Additional features and advantages of the present disclosure are described below. Those skilled in the art will appreciate that this disclosure may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. Moreover, those skilled in the art will realize that such equivalent constructions do not depart from the teachings of the present disclosure as set forth in the appended claims. The novel features believed to be characteristic of the present disclosure, both as to its organization and method of operation, together with further objects and advantages, will be more fully understood from the following description when considered in conjunction with the accompanying figures. It is to be expressly understood, however, that each of the figures is provided for the purpose of illustration and description only and is not intended as a definition of the limits of the present disclosure. [Brief explanation of the drawings]

[0009] The features, nature, and advantages of the present disclosure will become more apparent from the detailed description set forth below when taken in conjunction with the drawings in which like reference characters identify correspondingly throughout.

[0010] [Figure 1] FIG. 1 illustrates an example implementation of designing a system using a system-on-chip (SOC) for 3D automated labeling with structural and physical constraints according to an aspect of the present disclosure. [Figure 2] FIG. 1 is a block diagram illustrating a software architecture that may modularize functionality for 3D automated labeling with architectural and physical constraints, according to an embodiment of the present disclosure. [Figure 3] FIG. 1 illustrates an example of a hardware implementation of a 3D automated labeling system with structural and physical constraints, according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a block diagram of a 3D auto-labeling pipeline for the 3D auto-labeling system of FIG. 3 according to an embodiment of the present disclosure. [Figures 5A-5C] FIG. 1 illustrates a surface projection of an object using a signed distance field (SDF), according to aspects of the present disclosure. [Figure 6] 4 is a diagram of an initialization portion of a 3D auto-labeling pipeline for the 3D auto-labeling system of FIG. 3 according to an embodiment of the present disclosure. [Figure 7] 4A and 4B are diagrams illustrating example 3D labels output by a 3D auto-labeling pipeline for the 3D auto-labeling system of FIG. 3 in accordance with an exemplary configuration of the present disclosure. [Figure 8] FIG. 4 illustrates a system architecture of a 3D auto-labeling pipeline for the 3D auto-labeling system of FIG. 3 according to an embodiment of the present disclosure. [Figure 9] 1 is a flowchart illustrating a method for 3D automatic labeling of objects with structural and physical constraints, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] The detailed description set forth below in connection with the accompanying drawings is intended to illustrate various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. The detailed description includes specific details for a thorough understanding of various concepts. However, it will be apparent to those skilled in the art that such concepts may be practiced without such specific details. In some instances, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0012] Based on the teachings, one skilled in the art should understand that the scope of the present disclosure is intended to cover any aspect of the present disclosure, whether implemented independently or in combination with any other aspect of the present disclosure. For example, an apparatus may be implemented or a method may be practiced using any number of the described aspects. Furthermore, the scope of the present disclosure is intended to cover such an apparatus or method implemented using other structure, function, or structure and function in addition to or other than the various aspects of the present disclosure. It should be understood that any aspect of the present disclosure as disclosed may be embodied by one or more elements of a claim.

[0013] While particular aspects are described herein, many variations and permutations of such aspects are within the scope of the present disclosure. While certain benefits and advantages of the preferred aspects are described, the scope of the present disclosure is not intended to be limited to particular benefits, uses, or purposes. Rather, aspects of the present disclosure are intended to be broadly applicable to different technologies, system configurations, networks, and protocols, some of which are listed herein as examples in the drawings of the preferred aspects and the following description. The detailed description and drawings are merely illustrative of the present disclosure, rather than limiting the scope of the present disclosure, which is defined by the appended claims and equivalents thereof.

[0014] Deep learning often requires large labeled datasets to reach state-of-the-art performance. In the context of three-dimensional (3D) object detection for robotics applications, including autonomous vehicles, 3D cuboids are one annotation type because they enable proper inference across all nine degrees of freedom (position, orientation, and three degrees of freedom per example: meter range). Unfortunately, obtaining enough labels to train a 3D object detector can be laborious and costly, often relying on a large number of human annotators. Previous efforts to expand annotation pipelines include improved tooling, active learning, or a combination thereof. Such efforts often rely on heuristic problem-solving methods, bringing humans into the loop to correct semi-automated labeling, especially for challenging edge cases.

[0015] In particular, traditional methods in the domain of deep learning rely heavily on supervised training regimes. While traditional training can provide instant learning of input-to-output mappings, supervision requires large annotated datasets to accomplish the task. Unfortunately, acquiring such annotated datasets is laborious and costly. Furthermore, annotation costs vary significantly depending on the type of annotation, as 2D bounding boxes can be annotated much cheaper and faster than, for example, instance segments or cuboids.

[0016] Aspects of the present disclosure provide an improvement over conventional annotation methods by using several different priors for automatically labeling objects (e.g., vehicles and non-vehicles). Such priors include that a vehicle must be on the ground, that a vehicle cannot penetrate other vehicles, that a vehicle has four wheels, etc. This aspect of the present disclosure effectively uses a shape prior before automatically labeling an object. The shape prior includes specific constraints such as that a vehicle must be rigid, that a vehicle must be on the ground, that a vehicle cannot penetrate other vehicles, etc.

[0017] In one aspect of the present disclosure, three-dimensional (3D) annotation and object detection systems are improved by relying on differentiable rendering of shape priors. In this aspect of the disclosure, differentiable rendering of shape priors enables the recovery of metric scale, pose, and shape of objects in the field (e.g., vehicles in the case of autonomous driving systems). In one configuration, a 3D auto-labeling pipeline takes as input 2D detections (e.g., bounding boxes or instance masks) and sparse lidar point clouds. Lidar point clouds are now widely used in 3D robotics applications. In fact, object detections themselves can also be generated by off-the-shelf 2D object detectors. This configuration demonstrates that differentiable visual registration (e.g., also known as "analysis by synthesis" or "render and compare") is an effective approach to auto-labeling. That is, differentiable visual registration provides an effective approach for applications such as autonomous driving, including 3D robotics applications, including humanoid robots.

[0018] One configuration of a 3D annotation and object detection system includes a continuously traversable coordinate shape space (CSS) that combines a signed distance field (SDF) shape space (e.g., the "DeepSDF" shape space framework) with normalized object coordinates (NOCS). This combination allows object shapes to correspond, thereby facilitating deformable shape fitting. The 3D annotation and object detection system employs a differentiable SDF renderer for comparative scene analysis across the prescribed shape space. Additionally, the 3D annotation and object detection system includes a training curriculum for an auto-labeling pipeline that starts with synthetic data (e.g., computer-aided design (CAD) models of vehicles and driving scenes in the context of autonomous driving). In one configuration, the auto-labeling pipeline blends synthetic and real data in subsequent training loops, gradually increasing the difficulty of the input data throughout the training loops.

[0019] In some configurations, the auto-labeling pipeline begins with a CSS neural network trained to predict 2D NOCS maps as well as shape vectors from image patches. To bootstrap the initial version, the CSS network is trained using synthetic data where ground truth NOCS and shape vector targets can be easily obtained, and augmentation is applied to minimize the domain gap (e.g., sim2real). In such configurations, the auto-labeling loop includes (1) localizing instances using 2D annotations, (2) running the CSS network on the extracted patches of the input image, (3) reprojecting the NOCS onto the scene via LIDAR, (4) recovering object models from the CSS, (5) computing approximate poses via 3D-3D correspondences, and (6) performing projection and geometric registration to refine the initial guess.

[0020] After processing images in the training set, recovered auto-labels are collected and the CSS prediction network is retrained to gradually expand to new domains. This process is then repeated, resulting in increasingly better CSS predictions and improved auto-labeling of objects (e.g., 3D cuboid bounding boxes). To avoid drift due to noisy auto-labeling, a training curriculum is employed that initially focuses on simple examples and increases in difficulty with each training loop. In embodiments of the present disclosure, vehicle annotation (e.g., auto-labeling) is performed inexpensively by leveraging powerful priors such as car shape, metric size, road topography, maps, and similar shape priors.

[0021] This disclosure extends the use of shape priors to perform automated labeling. As explained, a "shape prior" is information known in advance about the shape of an object. For example, shape prior information may recognize that a vehicle should have an inflexible shape. This shape prior information may be extended to improve the accuracy of automated labeling. For example, a shape prior may include information such as that a vehicle must have four or more wheels, that the vehicle must be on the ground, and that vehicles must not interpenetrate.

[0022] In one aspect of the present disclosure, a method for automatically labeling 3D objects includes identifying initial object seeds for all frames from a given frame sequence of a scene using an object detector that uses 2D / 3D data. For example, the object seeds are objects that may be vehicles, but may also be non-vehicle objects. Once identified, an optimization procedure refines each initial seed against the 2D / 3D information while respecting map and road constraints. This part of the method involves a "shape prior." Additional shape prior information is used in this part of the process, including that vehicles must have wheels, be positioned on the ground, and not interpenetrate. Another optimization associates the 3D objects with each other over time to create smooth trajectories while respecting road and physical boundaries.

[0023] FIG. 1 illustrates an exemplary implementation of the above-described systems and methods for 3D automated labeling with structural and physical constraints using a system-on-chip (SOC) 100 in an ego vehicle 150. The SOC 100 may include a single processor or a multi-core processor (e.g., a central processing unit) according to certain aspects of the present disclosure. Variables (e.g., neural signals and synaptic weights), system parameters associated with the computing device (e.g., neural networks with weights), delays, frequency bin information, and task information may be stored in memory blocks. The memory blocks may be associated with a neural processing unit (NPU) 108, a CPU 102, a graphics processing unit (GPU) 104, a digital signal processor (DSP) 106, a dedicated memory block 118, or may be distributed across multiple blocks. Instructions executed by a processor (e.g., the CPU 102) may be retrieved from a program memory associated with the CPU 102 or from the dedicated memory block 118.

[0024] The SOC 100 may also include additional processing blocks configured to perform specific functions, such as a GPU 104, a DSP 106, and a connectivity block 110. The processing blocks may include a Fourth Generation Long Term Evolution (4G LTE) connection, an unlicensed Wi-Fi connection, a USB connection, a Bluetooth® connection, etc. Additionally, a multimedia processor 112 in combination with a display 130 may classify and categorize the pose of objects in an area of interest, for example, according to the display 130 showing the vehicle's field of view. In some aspects, the NPU 108 may be implemented in the CPU 102, the DSP 106, and / or the GPU 104. The SOC 100 may further include a sensor processor 114, an image signal processor (ISP) 116, and / or a navigation system 120, which may include, for example, a global positioning system.

[0025] SOC 100 may be based on an Advanced Risk Machine (ARM) instruction set, or the like. In another aspect of the present disclosure, SOC 100 may be a server computer in communication with ego vehicle 150. In this arrangement, ego vehicle 150 may include mechanisms including the processor of SOC 100. In this aspect of the disclosure, instructions loaded into the processor (e.g., CPU 102) or NPU 108 of ego vehicle 150 may include code for 3D automatic labeling with structural and physical constraints of objects (e.g., vehicles and non-vehicle and non-vehicle objects) in images captured by sensor processor 114. The instructions loaded into the processor (e.g., CPU 102) may also include code for planning and controlling (e.g., the ego vehicle) in response to associating 3D objects over time and creating a smooth trajectory while respecting road and physical boundaries from images captured by sensor processor 114.

[0026] The instructions loaded into the processor (e.g., CPU 102) may also include code for identifying initial object seeds for all frames from a given sequence of frames of a scene. The instructions loaded into the processor (e.g., CPU 102) may also include code for refining each of the initial object seeds on the 2D / 3D data while auto-labeling 3D object vehicles in the scene in accordance with predetermined structural and physical constraints. The instructions loaded into the processor (e.g., CPU 102) may further include code for associating the auto-labeled 3D object vehicles with trajectories over time while respecting the predetermined structural and physical constraints.

[0027] 2 is a block diagram illustrating a software architecture 200 that may modularize functionality for planning and control of an ego vehicle using a 3D automatic labeling system with architectural and physical constraints, according to an aspect of the present disclosure. Using the architecture, a controller application 202 may be designed to have various processing blocks of the SOC 220 (e.g., CPU 222, DSP 224, GPU 226, and / or NPU 228) assist with computations during runtime operation of the controller application 202.

[0028] Controller application 202 may be configured to invoke functions defined in user space 204 that may, for example, analyze a video scene captured by the ego vehicle's monocular camera based on 3D auto-labeling of objects within the scene. In aspects of the present disclosure, 3D auto-labeling of video objects (e.g., vehicles and non-vehicle objects) is improved by using structural and physical constraints as shape priors. Controller application 202 may request compilation of program code associated with a library defined in 3D auto-labeling application programming interface (API) 206 to label vehicles within the video scene captured by the ego vehicle's monocular camera using structural and physical constraints as shape priors.

[0029] A runtime engine 208, which may be compiled code of a runtime framework, may also be accessible to the controller application 202. The controller application 202 may cause the runtime engine 208 to perform, for example, monocular (single camera) 3D detection and auto-labeling. When an object is detected within a predetermined distance from the ego vehicle, the runtime engine 208 may then send a signal to an operating system 210, such as a Linux kernel 212 running on the SOC 220. The operating system 210 may then cause computations to be performed by the CPU 222, the DSP 224, the GPU 226, the NPU 228, or some combination thereof. The CPU 222 may be accessed directly by the operating system 210, and other processing blocks may be accessed via drivers, such as drivers 214-218 for the DSP 224, the GPU 226, or the NPU 228. In the illustrated example, the deep neural network is configured to run on a combination of processing blocks such as a CPU 222 and a GPU 226, or may run on an NPU 228, if provided.

[0030] FIG. 3 illustrates an example hardware implementation of a 3D auto-labeling system 300 that uses structural and physical constraints as shape priors in accordance with embodiments of the present disclosure. The 3D auto-labeling system 300 may be configured to understand a scene and enable planning and control of an ego vehicle 350 in response to images from video captured through a camera during operation of the vehicle 350. The 3D auto-labeling system 300 may be a component of a vehicle, a robotic device, or other device. For example, as shown in FIG. 3 , the 3D auto-labeling system 300 is a component of the vehicle 350. It is contemplated that the 3D auto-labeling system 300 may also be used in other devices, such as buses, motorcycles, and other vehicles, and therefore embodiments of the present disclosure are not limited to the 3D auto-labeling system 300 being a component of the vehicle 350. The vehicle 350 may be autonomous or semi-autonomous.

[0031] The 3D automatic labeling system 300 may be implemented with an interconnect architecture generally represented by interconnect 308. Interconnect 308 may comprise any number of point-to-point interconnects, buses, and / or bridges, depending on the particular application of the 3D automatic labeling system 300 and the overall design constraints of the vehicle 350. Interconnect 308 associates various circuits with one another, including one or more processors and / or hardware modules represented by sensor module 302, vehicle perception module 310, processor 320, computer-readable medium 322, communication module 324, movement module 326, position module 328, planner module 330, and controller module 340. Interconnect 308 may also associate various other circuits, such as timing sources, peripherals, voltage regulators, and power management circuits. The listed circuits are well known in the art and will not be described further.

[0032] The 3D automatic labeling system 300 includes a transceiver 332 coupled to a sensor module 302, a vehicle perception module 310, a processor 320, a computer-readable medium 322, a communication module 324, a movement module 326, a position module 328, a planner module 330, and a controller module 340. The transceiver 332 is coupled to an antenna 334. The transceiver 332 communicates with various other devices via a transmission medium. For example, the transceiver 332 may receive commands via transmission from a user or a remote device. As discussed herein, the user may be located remotely from the location of the automobile 350. As another example, the transceiver 332 may transmit automatically labeled 3D objects and / or planned actions in a video from the vehicle perception module 310 to a server (not shown).

[0033] The 3D auto-labeling system 300 includes a processor 320 coupled to a computer-readable medium 322. The processor 320 performs processes, including the execution of software stored on the computer-readable medium 322, to provide functionality according to the present disclosure. The software, when executed by the processor 320, causes the 3D auto-labeling system 300 to perform various functions described for the ego vehicle's perception of an auto-labeled scene in video captured by a single camera of an ego vehicle, such as automobile 350, or any of the modules (e.g., 302, 310, 324, 326, 328, 330, and / or 340). The computer-readable medium 322 may also be used to store data manipulated by the processor 320 when executing the software.

[0034] The sensor module 302 may acquire images via various sensors, such as a first sensor 304 and a second sensor 306. The first sensor 304 may be a visual sensor (e.g., a stereoscopic camera or a red-green-blue (RGB) camera) for capturing 2D RGB images. The second sensor 306 may be a ranging sensor, such as a light detection and ranging (LIDAR) sensor or a radio detection and ranging (RADAR) sensor. Of course, aspects of the present disclosure are not limited to the above sensors, as other types of sensors (e.g., thermal, sonar, and / or laser) are also contemplated for either the first sensor 304 or the second sensor 306.

[0035] Images from the first sensor 304 and / or the second sensor 306 may be processed by the processor 320, the sensor module 302, the vehicle perception module 310, the communication module 324, the movement module 326, the position module 328, and the controller module 340. In conjunction with the computer-readable medium 322, images from the first sensor 304 and / or the second sensor 306 are processed to perform the functions described herein. In one configuration, 3D object information captured and detected by the first sensor 304 and / or the second sensor 306 may be transmitted via the transceiver 332. The first sensor 304 and the second sensor 306 may be coupled to or in communication with the vehicle 350.

[0036] Scene understanding from video input based on automatic labeling of 3D objects in the scene is an important perceptual task in the field of autonomous driving, such as automobile 350. The present disclosure extends the use of shape priors to perform automatic labeling. As described, a “shape prior” is information known a priori about the shape of an object. For example, shape prior information may recognize that a vehicle must have an inflexible shape. This shape prior information may be extended to improve the accuracy of automatic labeling. For example, a shape prior may include information such as that a vehicle must have four or more wheels, that the vehicle must be on the ground, and that vehicles must not interpenetrate. In aspects of the present disclosure, vehicle annotation (e.g., automatic labeling) is performed inexpensively by leveraging powerful priors such as car shape, metric size, road topography, maps, and similar structural and physical shape prior constraints.

[0037] The location module 328 may determine the location of the vehicle 350. For example, the location module 328 may use a global positioning system (GPS) to determine the location of the vehicle 350. The location module 328 may implement a dedicated short-range communications (DSRC) compliant GPS unit. A DSRC compliant GPS unit includes hardware and software that causes the vehicle 350 and / or the location module 328 to comply with one or more of the following DSRC standards, including their derivatives or offshoots: DSRC standards include EN 12253:2004 Dedicated Short Range Communications - Physical Layer using 5.9 GHz Microwave (Review), EN 12795:2002 Dedicated Short Range Communications (DSRC) - DSRC Data Link Layer: Medium Access and Logical Link Control (Review), EN 12834:2002 Dedicated Short Range Communications - Application Layer (Review), EN 13372:2004 Dedicated Short Range Communications (DSRC) - DSRC Profile for RTTT Applications (Review), and EN ISO 14906:2004 Electronic Toll Collection - Application Interface.

[0038] The DSRC-compatible GPS unit in the location module 328 is operable to provide GPS data describing the location of the vehicle 350 with spatial accuracy to precisely orient the vehicle 350 to a desired location. For example, the vehicle 350 may desire partial sensor data while driving to a predetermined location. Spatial accuracy means that the location of the vehicle 350 is described by enough GPS data to locate a parking spot for the vehicle 350. That is, the location of the vehicle 350 is precisely determined with spatial accuracy based on the GPS data from the vehicle 350.

[0039] The communications module 324 may facilitate communications via the transceiver 332. For example, the communications module 324 may be configured to provide communications capabilities via various wireless protocols, such as Wi-Fi, Long Term Evolution (LTE), 3G, etc. The communications module 324 may also communicate with other components of the vehicle 350 that are not modules of the 3D automated labeling system 300. The transceiver 332 may be a communications channel via a network access point 360. The communications channel may include DSRC, LTE, LTE-D2D, mmWave, Wi-Fi (fundamental mode), Wi-Fi (ad hoc mode), visible light communications, TV-free band communications, satellite communications, full-duplex wireless communications, or any other wireless communications protocol, such as those described herein.

[0040] In some configurations, the network access point 360 includes a Bluetooth® or cellular communication network for transmitting and receiving data, including networks via Short Message Service (SMS), Multimedia Message Service (MMS), Hypertext Transfer Protocol (HTTP), direct data connection, Wireless Application Protocol (WAP), email, DSRC, full-duplex radio, mmWave, Wi-Fi (infrastructure mode), Wi-Fi (ad hoc mode), visible light, TV-free bands, and satellite communications. The network access point 360 may also include a mobile data network, which may include 3G, 4G, 5G, LTE, LTE-V2X, LTE-D2D, VoLTE, or any other mobile data network or combination of mobile data networks. Additionally, the network access point 360 may include one or more IEEE 802.11 wireless networks.

[0041] The 3D automated labeling system 300 also includes a planner module 330 for planning a selected trajectory to execute a route / action (e.g., collision avoidance) for the vehicle 350, and a controller module 340 for controlling movement of the vehicle 350. The controller module 340 may implement the selected action via the movement module 326, for example, for autonomous operation of the vehicle 350 along the selected route. In one configuration, the planner module 330 and the controller module 340 may collectively override user inputs when the user inputs are likely (e.g., predicted) to cause a collision according to the autonomy level of the vehicle 350. The modules may be software modules executing on the processor 320, software modules resident / stored on the computer-readable medium 322, and / or hardware modules coupled to the processor 320, or some combination thereof.

[0042] The National Highway Traffic Safety Administration (NHTSA) has defined various "levels" of autonomous vehicles (e.g., Level 0, Level 1, Level 2, Level 3, Level 4, Level 5). For example, if one autonomous vehicle has a higher level number than another autonomous vehicle (e.g., Level 3 is a higher level number than Level 2 or 1), the autonomous vehicle with the higher level number offers a greater combination and quantity of autonomous features than vehicles with lower level numbers. These various levels of autonomous vehicles are briefly described below.

[0043] Level 0: In a Level 0 vehicle, the set of Advanced Driver Assistance Systems (ADAS) features installed in the vehicle do not provide vehicle control but may provide alerts to the vehicle driver. Level 0 vehicles are neither autonomous nor semi-autonomous.

[0044] Level 1: In a Level 1 vehicle, the driver is ready to assume driving control of the autonomous vehicle at any time. The set of ADAS features installed in the autonomous vehicle may provide autonomous functions such as any combination of Adaptive Cruise Control (ACC), Park Assist with Autopilot, and Lane Keeping Assist (LKA) Type II.

[0045] Level 2: In a Level 2 vehicle, the driver must detect objects and events in the road environment and respond if the set of ADAS features installed in the autonomous vehicle does not respond appropriately (based on the driver's subjective judgment). The set of ADAS features installed in the autonomous vehicle may include acceleration, braking, and steering. In a Level 2 vehicle, the set of ADAS features installed in the autonomous vehicle can be disabled immediately after the driver takes over.

[0046] Level 3: In a Level 3 ADAS vehicle, the driver can safely divert their attention from the driving task within a known and limited environment (e.g., a highway), but must still be prepared to take control of the autonomous vehicle if necessary.

[0047] Level 4: In a Level 4 vehicle, the set of ADAS features onboard the autonomous vehicle allows the autonomous vehicle to be controlled in all environments except for a few, such as inclement weather. The driver of a Level 4 vehicle enables the automated system (consisting of the set of ADAS features onboard the vehicle) only when it is safe to do so. When an automated Level 4 vehicle is enabled, driver attention is not required for the autonomous vehicle to operate safely and consistently within acceptance criteria.

[0048] Level 5: A Level 5 vehicle requires no human intervention other than setting the destination and activating the system. The automated system can drive anywhere where driving is legal and can make independent decisions (which may vary depending on the jurisdiction in which the vehicle is located).

[0049] A highly autonomous vehicle (HAV) is an autonomous vehicle that is Level 3 or higher. Thus, in some configurations, motor vehicle 350 is any of a Level 0 non-autonomous vehicle, a Level 1 autonomous vehicle, a Level 2 autonomous vehicle, a Level 3 autonomous vehicle, a Level 4 autonomous vehicle, a Level 5 autonomous vehicle, and an HAV.

[0050] The vehicle perception module 310 may be in communication with the sensor module 302, the processor 320, the computer-readable medium 322, the communication module 324, the movement module 326, the position module 328, the planner module 330, the transceiver 332, and the controller module 340. In one configuration, the vehicle perception module 310 receives sensor data from the sensor module 302. The sensor module 302 may receive sensor data from the first sensor 304 and the second sensor 306. According to aspects of the present disclosure, the vehicle perception module 310 may receive sensor data directly from the first sensor 304 or the second sensor 306 to perform 3D automatic labeling of vehicles and non-vehicle objects from images captured by the first sensor 304 or the second sensor 306 of the car 350.

[0051] As shown in FIG. 3 , the vehicle perception module 310 includes an object seed detection module 312, an object seed refinement module 314, a 3D auto-labeling module 316, and a vehicle trajectory module 318 (e.g., based on video auto-labeling). The object seed detection module 312, the object seed refinement module 314, the 3D auto-labeling module 316, and the vehicle trajectory module 318 may be components of the same artificial neural network, such as a deep neural network (DNN), or different artificial neural networks. The object seed models of the object seed detection module 312 and / or the object seed refinement module 314 are not limited to deep neural networks. During operation, the vehicle perception module 310 receives a data stream from the first sensor 304 and / or the second sensor 306. The data stream may include a 2D RGB image from the first sensor 304 and LIDAR data points from the second sensor 306. The data stream may include multiple frames, such as an image frame. In this configuration, the first sensor 304 captures a monocular (single camera) 2D RGB image.

[0052] The vehicle perception module 310 is configured to understand a scene from video input (e.g., a sensor module) based on 3D automatic labels describing objects (e.g., vehicles) in the scene as a perception task during autonomous driving of the automobile 350. Aspects of the present disclosure are directed to a method for automatically labeling 3D objects, including identifying initial object seeds for all frames from a given frame sequence of a scene by an object seed detection module 312. For example, the object seeds are objects that may be vehicles, but may also be non-vehicle objects. Once identified, the object seed refinement module 314 performs an optimization procedure to refine each initial seed on 2D / 3D information while respecting physical and architectural constraints (e.g., map and road constraints).

[0053] In aspects of the present disclosure, this portion of the 3D auto-labeling method involves physical and structural shape prior constraints. Additional shape prior information is applied during this portion of the process, including that the vehicle must have wheels, must be positioned on the ground, and must not penetrate another vehicle. The 3D auto-labeling module completes the labeling of 3D vehicles and non-vehicle objects. Another optimization is provided by the vehicle trajectory module 318, which associates 3D objects with each other over time to create smooth trajectories while respecting road and physical boundaries, for example, as shown in FIG. 4.

[0054] Overview of the 3D Automated Labeling Pipeline FIG. 4 is a block diagram of a 3D auto-labeling pipeline 400 for the 3D auto-labeling system 300 according to an exemplary configuration of the present disclosure. In FIG. 4, a dataset 410 potentially includes both real imagery and LIDAR data as well as synthetic input data. As described, the synthetic input data can include computer-rendered driving scenes and CAD models of various types of vehicles with ground truth annotations. In such a configuration, the synthetic input data is used to train a coordinate shape space (CSS) network 430. In the example of FIG. 4, a 2D object detector (e.g., the vehicle perception module 310 of FIG. 3) detects three vehicles in the input image 405 and labels them with 2D labels 420a, 420b, and 420c, respectively. In this example, the 2D labels are 2D bounding boxes. The vehicle perception module 310 inputs the 2D labels 420a, 420b, and 420c to the CSS network 430.

[0055] In this aspect of the disclosure, for each 2D labeled object, the vehicle perception module 310 generates a 2D normalized object coordinates (NOCS) image and a shape vector. The vehicle perception module 310 decodes the 2D NOCS image and shape vector into an object model within a CSS network 430 (e.g., a continuously traversable CSS network). The vehicle perception module 310 then back-projects the 2D NOCS image to the corresponding LIDAR point cloud within a frustum. The vehicle perception module 310 also identifies one or more correspondences between the LIDAR point cloud and the object model to generate an initial estimate of an affine transformation between the LIDAR point cloud and the object model.

[0056] In this aspect of the disclosure, the object seed refinement module 314 performs an optimization process 440, which involves iteratively refining the estimate of the affine transformation through differentiable geometric and visual registration using a differentiable signed distance field (SDF) renderer. The 3D auto-labeling module 316 may then perform an auto-label verification process 450 to discard obviously inaccurate auto-labels before collecting such labels in a CSS label pool 460. Once any frames have been processed in a particular training loop, the CSS network 430 can be retrained (retraining 470 in FIG. 4 ) to begin the next training loop for the dataset 410. Various aspects of the 3D auto-labeling pipeline 400 are discussed in more detail in connection with FIGS. 5A-8 below.

[0057] Coordinate Shape-Space (CSS) Such a configuration employs a coordinate-space framework known in the literature as "DeepSDF" to embed a (waterproof) vehicle model into a shared, compact shape-space representation with a single neural network (e.g., CSS network 430). The concept is to convert the input model into an SDF, where each value indicates the distance to the nearest face, with positive and negative values representing exterior and interior regions, respectively. The SDF representation is desirable because it is generally easy for neural networks to learn. Ultimately, DeepSDF forms the shape-space of an implicit surface using a decoder that can be queried at spatially continuous 3D locations x = {x1, ... , xN} using a provided latent code z (the shape vector discussed above) to obtain the SDF value s = {s1, ... , sN}: f(x; z) = s.

[0058] To facilitate approximate deformable shape matching, such a construction combines the shape space with the NOCS to form a continuously traversable CSS, as discussed above. To do so, such a construction resizes the model to unit diameter and interprets the 3D coordinates of the 0 level set as a dense surface description.

[0059] To train the function f, such a configuration uses synthetic input data, including multiple CAD models of vehicles, as well as rendered traffic scenes with ground truth labels. Such a configuration follows the original DeepSDF approach to training, but latent vectors (e.g., shape vectors) are projected onto a unit sphere after each iteration (e.g., each pass through a training loop). In CSS, each vehicle corresponds to a single shape vector. For example, (0,1,1) might be an SUV, (0,1,0) might be a convertible, and (0,0,1) might be a Porsche. The vectors are continuous, meaning that CSS continuously traverses from one vehicle to another (as if one vehicle "morphs" into another as the shape space moves). In such a configuration, CSS is three-dimensional, but other configurations can have shape spaces of different dimensions.

[0060] Differentiable SDF Rendering One component of the 3D automatic labeling pipeline 400 is the ability to optimize objects with respect to pose, scale, and shape. Such functionality is performed by the object seed refinement module 314, as discussed above. To that end, such a configuration includes a differentiable SDF renderer that avoids mesh-related issues such as connectivity or intersection and provides an alternative way of sampling the representation. Such a configuration also employs an alternative formulation for rendering implicit surfaces that are suitable for backpropagation.

[0061] One aspect of a differentiable SDF renderer is the projection of a zero isosurface. Such a construction, comprising a query point (Equation 1) and an associated signed distance value (Equation 2), provides a differentiable method for accessing an implicit surface. Simply selecting a query point based on the signed distance value does not form a derivative with respect to the latent vector. Furthermore, the periodically sampled locations are approximately on the surface. Such a construction exploits the property that deriving an SDF for that location generates the normal at this point, which is actually calculated in a backward pass (Equation 3).

number

number

number

[0062] Since the normal provides the direction to the nearest surface, the signed distance value provides the exact distance. In this example, the query position can be projected onto the 3D surface position pi (Equation 4).

number

[0063] To obtain a clear surface projection, such a configuration requires that all points x outside the narrow band of the surface (Eq. 5) i The schematic diagram is shown in Figures 5A to 5C.

number

[0064] 5A-5C illustrate a surface projection 500 of an object using SDF in accordance with an exemplary configuration of the present disclosure. FIG. 5A illustrates an object surface 510 within a query grid 520. As shown in FIG. 5A, locations inside the object surface 510 have negative SDF values, and locations outside the object surface 510 have positive SDF values. FIG. 5B illustrates a normal to a point 530 outside the object surface 510. FIG. 5C illustrates a projected object surface point 540 that lies on the object surface 510.

[0065] Another aspect of the differentiable SDF renderer is the use of surface tangent disks. In the field of computer graphics, the concept of surface elements (surfels) is an established alternative to connected triangle primitives. A differentiable SDF representation in such a configuration produces oriented points that can be immediately used to render a surface disk. To obtain a watertight surface, a disk diameter is chosen that is large enough to cover the holes. The surface disk can be constructed as follows: 1. If the normal of the projection point satisfies Equation 6, estimate the 3D coordinates of the resulting tangent plane displayed on the screen.

number

number

number

number

[0066] 3D Automated Labeling Pipeline Another aspect of a differentiable SDF renderer is the rendering function. To produce the final rendering, such a construction employs a function that constructs a layer of 2D projected disks onto the image plane. This involves combining colors from various point primitives based on their depth values. The closer a primitive is to the camera, the stronger its contribution. Such a construction uses softmax to ensure that all contributions of primitives sum to 1 at each pixel. More specifically, the rendering function is I=Σ i NOCS(p i )* w i where I is the resulting image, NOCS returns the coordinate coloring, and w i is a weighting mask that defines the contribution of each disk (Equation 10).

number

number

number

number

[0067] 3D Object Detection One of the fundamental principles of the auto-labeling approach in this configuration is to utilize weak labels and strong differentiable priors to recover more complex labels. While this concept has broad applicability, this configuration focuses specifically on the auto-labeling of cuboids in driving scenes. As discussed above in connection with Figures 3 and 4, the 3D auto-labeling module 316 can run multiple loops (iterations) of the 3D auto-labeling pipeline 400 during the training phase. In the first training loop, the CSS label pool 460 contains entirely synthetic labels, and the CSS network 430 (e.g., the trained CSS network) has not yet adequately adapted to real images. The result can be noisy NOCS predictions that can only be trusted for well-behaved object examples in the scene.

[0068] In one aspect of the present disclosure, the vehicle perception module 310 prescribes a predetermined training curriculum in which the CSS network 430 is initially exposed to easy annotations, and the vehicle perception module 310 increases the difficulty throughout subsequent training loops. In such a configuration, the difficulty of an annotation can be defined by measuring the pixel size of the 2D label, the amount of intersection with other 2D labels, and whether the 2D label touches the image boundary (often indicating object truncation). The vehicle perception module 310 includes thresholds for these criteria to define curriculums of increasing difficulty.

[0069] For example, the CSS network 430 is derived from a ResNet18 backbone, follows an encoder-decoder structure, processes 128x128 input patches, and outputs NOCS maps of the same size and 3D shape vectors. Additional details regarding the structure of the CSS network 430 are provided below in conjunction with the discussion of Figure 8. Prior to the first annotation loop, the vehicle perception module 310 trains the CSS network 430 to infer 2D NOCS maps and shape vectors from patches. Such mappings can be bootstrapped from synthetic input data, as described above.

[0070] 6 is a diagram of the initialization portion of a 3D auto-labeling pipeline 600 for the 3D auto-labeling system 300 of FIG. 3, in accordance with an exemplary configuration of the present disclosure. For a given image patch (see image patch from input image 660 bounded by 2D labels 420 in FIG. 6), the vehicle perception module 310 uses the CSS network 430 to infer a 2D NOCS map M (610) and a shape vector z (620). The vehicle perception module 310 decodes z into an SDF, finds 3D surface points p={p,...,p} of the object model 630 in its local frame, and finds the NOCS coordinates p c is calculated using Equation 14.

number

[0071] To estimate the initial pose and scale, in this configuration, the vehicle perception module 310 establishes a 3D-3D correspondence between p and l to estimate an initial affine transformation between a point l of the LIDAR point cloud and a point p of the object model. To do so, the vehicle perception module 310 calculates the initial pose and scale for each p i For , find its nearest neighbor using NOCS distance (equation 15), and keep the result if equation 16 holds.

number

number

[0072] At this point, the vehicle perception module 310 can begin differentiable optimization against complementary 2D and 3D evidence. While the projected 2D information provides strong clues to orientation and shape, the 3D points allow for inference of scale and translation. At each iteration, the vehicle perception module 310 decodes the current shape vector estimate (Equation 17) and projects the surface points p i and transform the surface points using the current estimates of pose and scale (Equation 18).

number

number

[0073] Given the surface model points in the scene frame, the individual 2D and 3D losses are computed as follows: For the 2D loss, the optimization process 440 employs the differentiable SDF renderer discussed above to generate a rendering R that seeks maximum alignment with M. Because the predicted M may be noisy (especially during the initial training loop), minimizing the dissimilarity (Equation 19) may yield a poor solution.

number

number

[0074] In the case of 3D loss, for each of the equations 21, the vehicle perception module 310 determines the closest neighbor from l and keeps the neighbor if it is closer than 0.25 m.

number

number

[0075] Overall, the final criterion is the sum of both losses in such a configuration: loss = loss 2D + loss 3DIn such a configuration, both loss terms function with approximately the same magnitude, and therefore the terms are not balanced (e.g., weighted). While described with reference to 2D and 3D losses, the additional losses include losses due to structural and physical constraints. For example, the object seed refinement module 314 is configured to access vehicle shape prior information with respect to road boundaries and physical boundaries. In this example, the object seed refinement module 314 is configured to adjust the 3D object-vehicle association over time by applying road boundaries and physical boundaries to the trajectory, which may be optimized based on the additional losses from the imposed structural and physical constraints.

[0076] Referring back to the automatic label verification process 450 in FIG. 4, the optimization framework can sometimes produce inaccurate results, thereby reducing the impact of incorrectly inferred automatic labels. To that end, in this configuration, the object seed refinement module 314 performs geometric and projection verification to eliminate the worst automatic labels (e.g., cuboids). The object seed refinement module 314 measures the number of LIDAR points within a narrow band (0.2 m) around the surface of the automatic label and rejects them if less than 60% are outside this band. Furthermore, the object seed refinement module 314 defines projection constraints such that an automatic label is rejected if the intersection-over-union (IoU) of a mask rendered with the provided 2D label falls below 70%.

[0077] In such a configuration, the auto-labels that survive the auto-label verification process 450 are collected and added to the CSS label pool 460. After the first training loop, subsequent training loops use a mix of synthetic and real samples and are used to retrain the CSS network 430. Over multiple self-improving training loops, the CSS network 430 is retrained, resulting in better initialization and more accurate auto-labeling.

[0078] 7 illustrates example 3D labels output by the 3D auto-labeling pipeline 400 of FIG. 4 for the 3D auto-labeling system 300 of FIG. 3 according to an exemplary configuration of the present disclosure. The scene depicted in input image 710 includes object 720 (a vehicle) and object 730 (another vehicle). In this example, the 3D auto-labeling module 316 extracted 3D label 740a for object 720 and 3D label 740b for object 730. FIG. 7 illustrates that in this configuration, the 3D labels output to the 3D object detectors (e.g., object seed detection module 312 and object seed refinement module 314) are cuboids (e.g., 3D bounding boxes).

[0079] 3D Object Detection In such a configuration, the 3D cuboid auto-label is not an end goal, but rather a means to an end, i.e., 3D object detection. As those skilled in the art know, once the 3D auto-labeling module 316 automatically extracts a 3D label (e.g., a cuboid) for an object, it is a relatively straightforward matter for the vehicle perception module 310 to perform 3D object detection of the object based at least in part on the extracted 3D label for the object. In aspects of the present disclosure, the vehicle trajectory module 318 is trained to plan a trajectory of the ego vehicle according to the associated trajectory of the auto-labeled 3D object vehicle, while respecting road boundaries and physical boundaries.

[0080] Additional implementation details about pipeline components FIG. 8 illustrates a system architecture of a 3D auto-labeling pipeline 800 for the 3D auto-labeling system 300 of FIG. 3 according to an exemplary configuration of the present disclosure. As described above, in this configuration, the CSS network 430 comprises a ResNet18 backbone architecture. In this configuration, the decoder uses bilinear interpolation as an upsampling operation rather than deconvolution to reduce the number of parameters and computations. After each upsampling, the output feature map is concatenated with the feature map from the previous level and one convolutional layer. Because the CSS network 430 is trained on synthetic input data, it can be initialized with ImageNet weights, and the first five layers are frozen to prevent overfitting to the particularities of the rendered data. In this configuration, the five heads 805 of the CSS network 430 are responsible for the outputs of the U, V, and W channels of the NOCS, as well as the object's mask (610) and its latent vector (e.g., shape vector 620), and encode its DeepSDF shape.

[0081] The pose estimation block 810 is based on 3D-3D correspondence estimation. In one aspect of the present disclosure, the procedure is defined as follows: The CSS network 430 outputs a NOCS, mapping each RGB pixel to a 3D position on the surface of the object. The NOCS is back-projected to the LIDAR frustum points 650 using the provided camera parameters. The CSS network 430 then outputs a latent vector (e.g., shape vector 620), which is then fed into the Deep SDF network 820 (DSDF) and converted to a surface point cloud using a zero isosurface projection, as discussed above. The Deep SDF network 820 is trained to output a normalized model placed at the origin, so each point on the resulting model surface represents a NOCS. At this point, the system is ready to proceed with pose estimation.

[0082] NOCS is used to establish correspondences between frustum points and model points. The backprojected frustum NOCS are compared to the predicted model coordinates to estimate the nearest neighbors of each frustum point. RANSAC can be used for robust rejection of outliers. At each iteration, four random points (n) are selected from the set of correspondences and fed into the Procrustes algorithm to provide an initial estimate of the model pose and scale (i.e., an initial estimate of the affine transformation). In such a configuration, the following RANSAC parameters can be used: The number of iterations, k, is based on a standard function of the desired probability of success, p, using theoretical results (Equation 23):

number

[0083] In such a configuration, a threshold of 0.2 m is used to estimate the inliers and select the best fit. The final pose and scale of the initial affine transformation discussed above are calculated based on the inliers of the best fit.

[0084] Given the output of the CSS network 430 and pose initialization, the optimization process 440 proceeds with the optimization phase (see again FIG. 8 ). An input is formed for the DeepSDF network 820 by concatenating the latent vector z (620) with the 3D query grid x (520). The DeepSDF network 820 outputs an SDF value for each query point on the query grid 520, which is used in a zero isosurface projection to provide a dense surface point cloud. The resulting point cloud is then transformed using the estimated pose and scale obtained from the pose estimation block 810. Points that are not visible from a particular camera field of view can be filtered using simple backface culling, since the surface normals have already been calculated for the zero isosurface projection. At this stage, the vehicle perception module 310 can apply a 3D loss between the resulting transformed point cloud and the input LIDAR frustum points. The surface point cloud is also used as input to a differentiable renderer 860, which renders the NOCS as RGB and applies a 2D loss between the NOCS predictions of the CSS network 430 and the output NOCS of the differentiable renderer 860. The latent vectors (e.g., shape vector 620) and pose 830 are then updated, and the process is repeated until complete.

[0085] The 3D losses considered above support obtaining accurate pose / shape alignment with frustum points. However, in some cases, few points are available, leading to poor alignment results. On the other hand, the 2D loss allows accurate alignment in screen space on dense pixels, but is generally inadequate for optimizing 3D scale and translation, and is highly dependent on its initial estimate. The combination of the two losses (2D and 3D) provides the best of both worlds: dense 2D alignment and robust scale / translation estimation.

[0086] 9 is a flowchart illustrating a method for 3D automatic labeling of objects with structural and physical constraints according to an aspect of the present disclosure. Method 900 begins at block 902, where an object detector uses 2D / 3D data to identify initial object seeds for every frame from a given sequence of frames of a scene. For example, as shown in FIG. 3, the object seed detection module 312 is trained to identify initial object seeds for every frame from a given sequence of frames of a scene from the sensor module 302. For example, the object seeds are objects that may be vehicles, but may also be objects other than vehicles, as shown in FIG.

[0087] In block 904, each of the initial object seeds is refined on the 2D / 3D data while conforming to predetermined physical and structural constraints to automatically label 3D object vehicles in the scene. For example, as shown in FIG. 3, the object seed refinement module 314 is trained to refine the initial object seeds from the object seed detection module 312 by applying predetermined shape prior information (e.g., map and road constraints). Once identified, an optimization procedure in the object seed refinement module improves each initial object seed relative to the 2D / 3D information while respecting the map and road constraints. This portion of the method 900 involves shape priors. In this portion of the method 900, additional shape prior information (e.g., vehicles must have wheels, must be positioned on the ground, and must not penetrate another vehicle) is applied to constrain 3D object detection and enable the 3D auto-labeling module 316 to automatically label 3D object vehicles in the scene, as shown in FIG. 4.

[0088] In block 906, the auto-labeled 3D object vehicles are associated with trajectories over time while respecting predetermined structural and physical constraints. For example, as shown in FIG. 3 , the vehicle trajectory module 318 is trained to associate the auto-labeled 3D object vehicles with trajectories over time while conforming to predetermined structural and physical constraints. The method 900 also includes accessing vehicle shape prior information related to road boundaries and physical boundaries. The method 900 further includes adjusting the 3D object vehicle association over time by applying the road boundaries and physical boundaries to the trajectories. This enables the method 900 to associate the auto-labeled 3D object vehicles of block 906.

[0089] At block 908, a trajectory of the ego vehicle is planned according to the associated trajectory of the auto-labeled 3D object vehicle while respecting road boundaries and physical boundaries. For example, as shown in FIG. 3 , the vehicle trajectory module 318 is configured to plan a trajectory of the ego vehicle (e.g., automobile 350) according to the associated trajectory of the auto-labeled 3D object vehicle while respecting road boundaries and physical boundaries. Further, the controller module 340 is configured to select a vehicle control action (e.g., accelerate, brake, steer, etc.). The method 900 further includes performing three-dimensional object detection of the auto-labeled 3D vehicle object within the scene. The method 900 also includes performing three-dimensional pose detection of the auto-labeled 3D vehicle object within the scene.

[0090] In some aspects of the present disclosure, method 900 may be implemented by SOC 100 (FIG. 1) or software architecture 200 (FIG. 2) of ego vehicle 150 (FIG. 1). That is, each element of method 900 may be implemented by, for example, but not limited to, SOC 100, software architecture 200, or a processor (e.g., CPU 102) and / or other components of ego vehicle 150.

[0091] The various operations of the methods described above may be implemented by any suitable means capable of performing the corresponding functions, which may comprise various hardware and / or software components and / or modules, including, but not limited to, circuits, application specific integrated circuits (ASICs), or processors. Generally, where there are illustrated operations, such operations may include corresponding equivalent means and functional components that are similarly numbered.

[0092] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" can include calculating, computing, processing, deriving, examining, looking up (e.g., looking up in a table, database, or another data structure), ascertaining, and the like. Furthermore, "determining" can include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Furthermore, "determining" can include resolving, selecting, choosing, establishing, and the like.

[0093] As used herein, a phrase referring to "at least one" of a list of items refers to any combination of such items, including single members. By way of example, "at least one of a, b, or c" is intended to cover a, b, c, ab, ac, bc, and abc.

[0094] The various illustrative logic blocks, modules, and circuits described in connection with this disclosure may be implemented or performed using a processor configured in accordance with the present disclosure, a digital signal processor (DSP), an ASIC, a field programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A processor may be a microprocessor, but alternatively, the processor may be any commercially available processor, controller, microcontroller, or state machine specially configured as described herein. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration.

[0095] The steps of a method or algorithm described in connection with the present disclosure may be embodied directly in hardware, in a software module executed by a processor, or a combination of the two. A software module may reside on any form of storage medium known in the art. Some examples of storage media include random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, removable disk, CD-ROM, etc. A software module may comprise a single instruction, or many instructions, and may be distributed across several different code segments, among different programs, and across multiple storage media. A storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral to the processor.

[0096] The methods disclosed herein comprise one or more steps or actions for achieving the described method. Method steps and / or actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be changed without departing from the scope of the claims.

[0097] The described functionality may be implemented in hardware, software, firmware, or any combination thereof. When implemented in hardware, an example hardware configuration may include a processing system within a device. The processing system may be implemented using a bus architecture. The bus may include any number of interconnecting buses and bridges, depending on the particular application and overall design constraints of the processing system. The bus may associate various circuits with each other, including a processor, a machine-readable medium, and a bus interface. The bus interface may connect a network adapter to the processing system via the bus, among other things. The network adapter may implement signal processing functions. In certain aspects, a user interface (e.g., keypad, display, mouse, joystick, etc.) may also be connected to the bus. The bus may also associate various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, etc., that are well known in the art and will not be described further.

[0098] The processor may be responsible for managing buses and performing processes, including the execution of software stored on a machine-readable medium. Examples of processors that may be specially configured according to the present disclosure include microprocessors, microcontrollers, DSP processors, and other circuitry capable of executing software. Software shall be broadly construed to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. The machine-readable medium may include, by way of example, random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, magnetic disk, optical disk, hard drive, or any other suitable storage medium, or any combination thereof. The machine-readable medium may be incorporated into a computer program product. The computer program product may include packaging materials.

[0099] In a hardware implementation, the machine-readable medium may be part of a processing system separate from the processor. However, as one skilled in the art will readily appreciate, the machine-readable medium, or any portion thereof, may be external to the processing system. By way of example, the machine-readable medium may include a transmission line, a carrier wave modulated by data, and / or a computer product separate from the device. Any of the foregoing may be accessed by the processor via a bus interface. Alternatively, or in addition, the machine-readable medium, or any portion thereof, may be integrated into the processor, such as with cache and / or special register files. While the various components discussed may be described as having a particular location, such as a local component, they may also be configured in various ways, such as with certain components configured as part of a distributed computing system.

[0100] A processing system may be comprised of one or more microprocessors providing processor functionality and external memory providing at least a portion of the machine-readable medium, all of which are associated with other support circuitry via an external bus architecture. Alternatively, a processing system may include one or more neuromorphic processors for implementing the neuron models and models of neuronal systems described herein. Alternatively, a processing system may be implemented as an ASIC with the processor, bus interface, user interface, support circuitry, and at least a portion of the machine-readable medium integrated on a single chip, or one or more PGAs, PLDs, controllers, state machines, gate logic, discrete hardware components, or any other suitable circuitry, or any combination of circuitry capable of performing the various functions described throughout this disclosure. Those skilled in the art will recognize how best to implement the described functionality for a processing system depending on the particular application and the overall design constraints imposed on the entire system.

[0101] The machine-readable medium may comprise multiple software modules. The software modules include instructions that, when executed by a processor, cause the processing system to perform various functions. The software modules may include a transmitting module and a receiving module. Each software module may reside on a single storage device or be distributed across multiple storage devices. As an example, a software module may be loaded into RAM from a hard drive when a triggering event occurs. During execution of a software module, the processor may load some of the instructions into a cache to increase access speed. One or more cache lines may then be loaded into a special-purpose register file for execution by the processor. When referring to the functionality of a software module below, it will be understood that such functionality is implemented by the processor when executing instructions from that software module. It will further be understood that aspects of the present disclosure provide improved functionality for a processor, computer, machine, or other system implementing such aspects.

[0102] If implemented in software, the functions may be stored on or transmitted as one or more instructions or code on a non-transitory computer-readable medium. Computer-readable media includes both computer storage media and communication media, including any medium that facilitates transfer of a computer program from one place to another. A storage medium may be any available medium that can be accessed by a computer. By way of example, and not limitation, such computer-readable media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can carry or store desired program code in the form of instructions or data structures that can be accessed by a computer. Additionally, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using wireless technology, such as coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or infrared (IR), radio, microwave, or other wireless technology, then the coaxial cable, fiber optic cable, twisted pair, DSL or infrared, radio, microwave, or other wireless technology is included in the definition of medium. As used herein, disk and disc include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy discs, and Blu-ray discs. While disks typically reproduce data magnetically, discs reproduce data optically using lasers. Thus, in some aspects, computer-readable media may include non-transitory computer-readable media (e.g., tangible media). Furthermore, in other aspects, computer-readable media may include transitory computer-readable media (e.g., signals). Additionally, combinations of the above should be included within the scope of computer-readable media.

[0103] Thus, certain aspects may comprise a computer program product for performing the operations presented herein. For example, such a computer program product may comprise a computer-readable medium having instructions stored (and / or encoded) thereon. The instructions are executable by one or more processors to perform the operations described herein. In certain aspects, the computer program product may comprise packaging materials.

[0104] Furthermore, it should be understood that modules and / or other suitable means for implementing the methods and techniques described herein can be downloaded and / or otherwise obtained by a user terminal and / or base station, as appropriate. For example, such devices can be coupled to a server to facilitate the transfer of means for implementing the methods described herein. Alternatively, the various methods described herein can be provided via storage means (e.g., physical storage media such as RAM, ROM, CD, or floppy disk), such that the user terminal and / or base station can obtain the various methods when the user terminal and / or base station couples or provides the storage means to the device. Furthermore, any other suitable technology for providing the methods and techniques described herein to a device can be utilized.

[0105] It is to be understood that the claims are not limited to the precise configuration and components described above. Various modifications, changes and variations may be made in the arrangement, operation and details of the methods and apparatus described above without departing from the scope of the claims.

Claims

1. A method for 3D automatic labeling of objects with predetermined structural and physical constraints, comprising: Identifying initial object seeds for all frames from a given sequence of frames of a scene; refining each of the initial object seeds on the 2D / 3D data while automatically labeling 3D object vehicles in the scene in accordance with the predetermined structural and physical constraints; Associating the automatically labeled 3D object vehicles with trajectories over time while respecting the predetermined structural and physical constraints; A method comprising:

2. The method of claim 1 , further comprising planning a trajectory of an ego vehicle according to the associated trajectory of the auto-labeled 3D object vehicle while respecting road boundaries and physical boundaries.

3. The method of claim 1 , wherein identifying the initial object seeds is performed by a vehicle perception module using 2D / 3D data.

4. refining the initial object seed comprises: accessing vehicle shape prior information; discarding an incorrect automatic label of the initial object seed if the initial object seed is identified as being inconsistent with the vehicle shape prior information; The method of claim 1 , comprising:

5. Associating the automatically labeled 3D object vehicle includes: accessing vehicle shape prior information relative to road boundaries and physical boundaries; adjusting the 3D object-vehicle association over time by applying the road boundaries and physical boundaries to the trajectory; The method of claim 1 , comprising:

6. and planning a trajectory of an ego vehicle according to a perception of the scene from video captured by the ego vehicle. The method of claim 1.

7. The method of claim 1 , further comprising performing three-dimensional object detection of the automatically labeled 3D vehicle objects in the scene.

8. The method of claim 1 , further comprising: performing three-dimensional pose detection of the automatically labeled 3D vehicle objects in the scene.

9. 1. A non-transitory computer-readable medium having program code recorded thereon for 3D automated labeling of objects with predetermined structural and physical constraints, the program code being executed by a processor; program code for identifying initial object seeds for all frames from a given sequence of frames of a scene; program code for refining each of the initial object seeds on the 2D / 3D data while automatically labeling 3D object vehicles in the scene in accordance with the predetermined structural and physical constraints; and program code for associating the automatically labeled 3D object vehicle with a trajectory over time while respecting the predetermined structural and physical constraints. Non-transitory computer-readable medium.

10. 10. The non-transitory computer readable medium of claim 9, further comprising program code for planning a trajectory of an ego vehicle according to an associated trajectory of the auto-labeled 3D object vehicle while respecting road boundaries and physical boundaries.

11. 10. The non-transitory computer-readable medium of claim 9, wherein the program code for identifying the initial object seeds is executed by a vehicle perception module using 2D / 3D data.

12. The program code for refining the initial object seed comprises: program code for accessing vehicle shape prior information; program code for discarding an incorrect automatic label of the initial object seed if the initial object seed is identified as being inconsistent with the vehicle shape prior; 10. The non-transitory computer-readable medium of claim 9, comprising:

13. a program code for associating the automatically labeled 3D object vehicle, program code for accessing vehicle shape prior information with respect to road boundaries and physical boundaries; program code for adjusting the 3D object-vehicle association over time by applying the road boundaries and physical boundaries to the trajectory; 10. The non-transitory computer-readable medium of claim 9, comprising:

14. 10. The non-transitory computer-readable medium of claim 9, further comprising program code for planning a trajectory of an ego vehicle according to a perception of the scene from video captured by the ego vehicle.

15. 10. The non-transitory computer readable medium of claim 9, further comprising program code for performing three-dimensional object detection of the automatically labeled 3D vehicle objects in the scene.

16. The non-transitory computer readable medium of claim 9 , further comprising program code for performing three-dimensional pose detection of the automatically labeled 3D vehicle object in the scene.

17. 1. A system for 3D automatic labeling of objects with predetermined structural and physical constraints, comprising: an object seed detection module trained to identify initial object seeds for all frames from a given sequence of frames of a scene; an object seed refinement module trained to refine each of the initial object seeds on the 2D / 3D data while automatically labeling 3D object vehicles in the scene in accordance with the predetermined structural and physical constraints; a 3D auto-labeling module trained to associate the auto-labeled 3D object-vehicles with trajectories over time while respecting the predetermined structural and physical constraints; A system comprising:

18. 20. The system of claim 17, further comprising a vehicle trajectory module trained to plan a trajectory of an ego vehicle according to an associated trajectory of the auto-labeled 3D object vehicle while respecting road boundaries and physical boundaries.

19. 20. The system of claim 17, further comprising a vehicle perception module executed using 2D / 3D data and trained to identify the initial object seeds.

20. The object seed refinement module: Access vehicle shape prior information, 20. The system of claim 17, further trained to discard erroneous automatic labels of the initial object seeds if the initial object seeds are identified as inconsistent with the vehicle shape prior information.

Images