Systems and methods for dataset and model management for multimodal automated labeling and active learning

Abstract

To provide a method and system for managing datasets.SOLUTION: Datasets for autonomous driving systems and multi-modal scenes may be automatically labeled using previously trained models to mitigate the limitations of conventional manual data labeling. Properly version-managed models including model weights and knowledge of the dataset on which the model has been trained may be used to run an inference operation on unlabeled data, thus automatically labeling the dataset. The newly labeled dataset may then be used to train new models including sparse data sets, in a semi-supervised or weakly-supervised fashion.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to improvements in machine learning, and more particularly to automatically labeling datasets for autonomous driving models and networks. [Background technology]

[0002] Machine learning provides the foundation for the design, programming, and operation of autonomous vehicles. Autonomous and semi-autonomous vehicles may be trained according to environmental and situational data, enabling the vehicle to operate and navigate known and unknown trajectories. Sensors installed and configured on the ego-vehicle, i.e., the autonomous or semi-autonomous vehicle, provide environmental data to the machine learning system. While monocular cameras are a cost-effective approach compared to more complex imaging systems, such as LiDAR and stereo cameras, the sensor data from monocular cameras does not explicitly include depth information. Instead, the vehicle implements processing routines that extract depth information from the monocular images.

[0003] One of the challenges in machine learning is dataset labeling. For systems that rely on active learning to process the vast amounts of data collected, features in the data must be labeled. Labeling features in collected data allows the system to identify the same or similar features in later data acquisition. Traditionally, in computer vision and autonomous driving modeling, datasets are manually labeled by employees, contractors, or crowdsourced resources who view data, such as video frames, and assign labels to target features. Dataset labeling is a time-consuming and costly process that is out of proportion to the rate at which raw data is collected by the many sensors and onboard systems. Summary of the Invention

[0004] Aspects of the present disclosure provide systems and methods for managing multimodal datasets in an active learning pipeline. Models trained on datasets and already labeled may provide relevant prior information on actual labels for multimodal scenes. Additional aspects of the present disclosure provide a combination of inference results from several high-performing models or ensembles for a specific task to further improve the quality of such pseudo-labels. Aspects of the present disclosure may utilize appropriately versioned models (including model weights and the datasets on which they were trained) to perform inference on unlabeled data, thereby generating auto-labeled datasets. These auto-labeled datasets may then be used to train new models. The system may record that a new model was obtained using auto-labels, as well as which model and unlabeled data led to the auto-labels. The system may also support the ability to utilize sparsely labeled datasets. In this case, training may be performed in a semi-supervised and weakly supervised manner, since the auto-labels can be used to fill in the gaps.

[0005] According to one aspect, a method for managing datasets is disclosed. A first dataset may be received. The first dataset may include unlabeled data. The first dataset may be input to a trained network. The trained network may be trained with an existing model. The unlabeled data of the first dataset may be labeled using inference from the trained network to generate a labeled dataset. The untrained network may be trained using the labeled dataset to generate a new model.

[0006] According to another aspect, a system for managing datasets is disclosed. The system may include one or more processors and a memory communicatively coupled to the one or more processors. The memory may store an inference module including instructions that, when executed by the one or more processors, cause the one or more processors to generate a labeled dataset. The dataset may be generated by receiving a first dataset including unlabeled data and inputting the first dataset to a trained network. The trained network may be trained with an existing model. The unlabeled data of the first dataset may be labeled using inferences from the trained network to generate a labeled dataset. The untrained network may be trained using the labeled dataset.

[0007] According to another aspect, a non-transitory computer-readable medium for managing datasets is disclosed. The medium may include instructions, which when executed by one or more processors, cause the one or more processors to receive a first dataset, the first dataset including unlabeled data, and input the first dataset to a trained network. The trained network may be trained with an existing model. The unlabeled data of the first dataset may be labeled using inference from the trained network to generate a labeled dataset. The untrained network may be trained using the labeled dataset.

[0008] The foregoing has outlined rather broadly the features and technical advantages of the present disclosure in order that the detailed description that follows may be better understood. Additional features and advantages of the present disclosure are described below. Those skilled in the art will appreciate that this disclosure may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. Moreover, those skilled in the art will appreciate that such equivalent constructions do not depart from the teachings of the present disclosure as set forth in the appended claims. The novel features believed characteristic of the present disclosure, both as to its organization and method of operation, together with additional objects and advantages, will be better understood from the following description when considered in conjunction with the accompanying figures. It is to be expressly understood, however, that each of the figures is provided for the purpose of illustration and description only and is not intended to define the scope of the present disclosure. [Brief explanation of the drawings]

[0009] The features, nature and advantages of the present disclosure will become more apparent from the detailed description set forth below when taken in conjunction with the drawings in which like reference characters identify correspondingly throughout.

[0010] [Figure 1A] FIG. 1 illustrates an example of an autonomous actor according to aspects of the present disclosure. [Figure 1B] FIG. 1 illustrates an example of a labeled frame according to aspects of the present disclosure. [Figure 2] FIG. 1 illustrates an example of labeling frames by inference according to aspects of the present disclosure. [Figure 3] 1 is a flow diagram for training a model according to an aspect of the present disclosure. [Figure 4] 1 is a flow diagram for automatically labeling a dataset according to an aspect of the present disclosure. [Figure 5] FIG. 1 illustrates a method for labeling a dataset according to an aspect of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] The detailed description set forth below in connection with the accompanying drawings is intended to illustrate various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. The detailed description includes specific details for providing a general understanding of various concepts. However, it will be apparent to those skilled in the art that such concepts may be practiced without such specific details. In some instances, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0012] The behavior of autonomous and semi-autonomous agents may be controlled or adjusted based on detected objects in the agent's vicinity. For example, a route for an autonomous agent may be planned based on the location of other objects on the road. As another example, if a detected object is in the agent's path, the route may be adjusted to avoid a collision. In this disclosure, an agent refers to an autonomous or semi-autonomous agent.

[0013] An actor may detect objects from the output of various sensors. For example, a 2D camera may generate a 2D red-green-blue (RGB) image, and a light detection and ranging (LIDAR) sensor may generate a 3D point cloud that provides height and depth information. The sensor outputs may be combined into a multimodal frame, where a series of frames generates a video. An object detection model, such as a convolutional neural network (CNN), is trained to identify objects of interest in each frame. Each identified object may be labeled or annotated with a bounding box. For each frame in the series of frames, the model may output a classification label corresponding to each identified object and a bounding box corresponding to each object's location. The object detection model may be referred to as a model. An annotated frame may refer to a frame that includes the generated bounding boxes. The annotated frame may also include a classification label or a reference to a classification label.

[0014] To improve a model, it is desirable to label objects of interest that may be unknown to a particular model so that when the model is run again, it can more easily distinguish between such objects and objects that may have similar attributes. Automatically labeling objects of interest reduces the cost and time required to generate reliable data sets upon which subsequent systems and networks can be trained.

[0015] The video generated by the machine learning model may be high-resolution (e.g., high-quality) video. This may result in a large video file size. For example, the video file size may be more than twice the size of the object detection model output. As known to those skilled in the art, traditional wireless networks have limited bandwidth. Furthermore, traditional wireless networks have uneven coverage areas, resulting in inconsistent quality of service. Therefore, when an autonomous subject is parked in a location with a high-bandwidth data connection, such as a home garage, the subject may upload data to a remote device, such as a server. That is, the data may be transmitted when the subject establishes a high-bandwidth connection with the remote device. The data may include the video and the object detection model output. Furthermore, in some cases, the subject may also upload sensor data.

[0016] Data transmitted over a high-bandwidth connection may be used to analyze and update the model. Unidentified or misclassified objects may be appropriately labeled offline to update the dataset and retrain the model. In traditional systems, labeling unannotated data requires manual labeling by a human analyst. Over time, as more data is collected and labeled, the model may be retrained. If the retrained model is an improvement over the previous model, the retrained model is deployed. The time between training the initial model and retraining the model may be on the order of days or months.

[0017] According to aspects of the present disclosure, to alleviate the limitations of traditional manual data labeling, datasets for autonomous driving systems and multimodal scenes may be automatically labeled using previously trained models. A properly versioned model, including the model weights and knowledge of the datasets on which the model was previously trained, can be used to perform inference operations on unlabeled data to automatically label the dataset. The newly labeled dataset may then be used to train a new model with a sparse dataset in a semi-supervised or weakly supervised manner.

[0018] FIG. 1A illustrates an example of an operating subject 100 detecting objects 104, 116, 118 using sensors 106, 108, according to aspects of the present disclosure. As shown in FIG. 1A, the operating subject 100 may be traveling on a road 110. A first vehicle 104 may be in front of the operating subject 100, and a second vehicle 116 may be adjacent to the operating subject 100. Additionally, a cyclist 118 may be in a bicycle lane 102 adjacent to the operating subject 100. In this example, the operating subject 100's object detection system is in communication with a 2D camera 108, such as a 2D RGB camera, and a LIDAR sensor 106. The 2D camera 108 and LIDAR sensor 106 may be integral with the operating subject 100. Other sensors, such as radio detection and ranging (RADAR) and / or ultrasonic, are also contemplated. Additionally or alternatively, the operating subject 100 may include one or more additional 2D cameras and / or LIDAR sensors. For example, the additional sensors may be side-facing and / or rear-facing sensors.

[0019] In one configuration, the 2D camera 108 captures a 2D image including objects within a field of view 114 of the 2D camera 108. The LIDAR sensor 106 may generate one or more output streams. A first output stream may include a 3D cloud of points of objects within a first field of view, such as a 360° field of view 112 (e.g., a bird's-eye view). A second output stream may include a 3D cloud of points of objects within a second field of view, such as a forward-looking field of view 126. An object detection system may use the data streams of the LIDAR sensor 106 to detect objects within an environment.

[0020] The 2D image captured by the 2D camera 108 may include 2D images of the first vehicle 104 and the cyclist 118 because they are within the field of view 114 of the 2D camera 108. The object detection system of the actor 100 may extract features from objects of interest in the 2D image. For example, an artificial neural network, such as a convolutional neural network, of the object detection system may extract features of the first vehicle 104 and the cyclist 118. Based on the extracted features, the object detection system may classify the first vehicle 104 as a car and the cyclist 118 as a bicyclist. Furthermore, the object detection system may locate the first vehicle 104 and the cyclist 118 within the first frame 150, as shown in FIG. 1B .

[0021] 1B illustrates an example of a frame 150 generated by an object detection system according to aspects of the present disclosure. In this example, the frame 150 includes an object within the field of view 114 of the 2D camera 108. Specifically, the frame includes the first vehicle 104 and the cyclist 118 because both are within the field of view 114 of the 2D camera 108.

[0022] Based on the features extracted by the object detection model, the first vehicle 104 and the cyclist 118 may be labeled (e.g., "car" and "bicyclist"). As described herein, the labels may be artifacts from a previously trained model or the result of offline analysis of an inference module to automatically label objects as such.

[0023] The object detection system may also locate the first vehicle 104 and the cyclist 118 by annotating the location of each object with bounding boxes 152, 154 generated by the object detection model. As shown in FIG. 1B, a first bounding box 152 may be drawn around an area corresponding to the first vehicle 104, and a second bounding box 154 may be drawn around an area corresponding to the cyclist 118. Those skilled in the art will understand that the labels in FIG. 1B are for illustrative purposes and that aspects of the present disclosure are not limited to labeling objects according to the labels shown in FIG. 1B.

[0024] The first frame 150 in FIG. 1B may represent one frame of a series of frames (e.g., a video). An autonomous driving system may control the behavior of the operating entity 100 based on the objects detected in each frame. The video generated from the series of frames may be stored at the operating entity 100 for later analysis. Additionally, the sensor data and / or model output may also be stored at the operating entity 100 for later analysis. For example, the video, sensor data, and model output may be stored on a memory device of the operating entity. The stored video, sensor data, and / or model output may be transmitted to a remote device for further analysis as described herein, including using the dataset and multimodal scene information to generate additional inferences regarding labels and boundaries of previously unlabeled objects.

[0025] For example, a training system or pipeline may receive frame 150 of FIG. 1B and determine whether the model included unlabeled objects. Offline analysis may leverage other datasets that have objects similar to the unlabeled objects in the current dataset to infer labels for the unlabeled objects. The analysis may determine that the labels ("car" and "bicyclist") correspond to the first vehicle 104 and the cyclist 118 and automatically provide such labels for the objects in the scene.

[0026] FIG. 2 illustrates an example of automatic labeling of a frame 200 according to an embodiment of the present disclosure. As shown in FIG. 2 , an initial frame 200 may be processed by an actor-based inference module 212. The frame 200, acquired from an autonomous driving system or other source, may include previously unknown objects of interest, such as a vehicle 204 on a road 206 and a cyclist 218 on a bike lane 202 adjacent to the road 206. The frame 200 may also include a guardrail 208 and a tree 210. An object detection model may have previously identified the first vehicle 204 and the cyclist 218, or similar objects, as objects of interest during prior training. The inference model 212 may receive data related to such objects from a model output 214. The data from the model output 214 may be utilized to assist the inference module 212 in identifying the objects in the frame 200. To this end, the reasoning module 212 may output classification labels ("car" and "bicyclist") and bounding boxes 252, 254 corresponding to the vehicle 204 and the cyclist 218, respectively.

[0027] According to another aspect, the vehicle 204 and the cyclist 218 may have been previously labeled by the model during a previous training run, but other objects, such as the guardrail 208 or the tree 210, may not have been previously labeled. The reasoning module 212 may incorporate input data information from the model output 214 to assist the reasoning module in identifying and automatically labeling the guardrail 210 and the tree 210 without human interaction.

[0028] According to one aspect, a hash function using the unique identifiers of the object and the inference model may be used to determine a universally unique identifier (UUID). The inference module 212 may be considered task-specific. As such, the inference module 212 may have one or more associated ontologies, and the label / annotation type that the inference module 212 intends to generate may be known in advance.

[0029] According to one aspect, the reasoning module 212 may process the frame 200 according to one or more ontologies to make the automatic labeling function applicable to a wide range of applications. For example, the reasoning module may output a labeled dataset according to a first ontology 220 in which the vehicle 204 is labeled as an “automobile” 252 and the cyclist 218 is labeled as a “bicyclist” 254. According to another application model, the reasoning module 212 may analyze the same frame 200 to identify and label the vehicle 204 as an “auto” 252′ and the cyclist 218 as a “manual” 254′ according to a second ontology 230. Using a third ontology 240, the reasoning module 212 may identify the vehicle 204 as an “object” 252″ and the cyclist 218 as a “human” 254″. Those skilled in the art will recognize that other ontologies and potential labels may be considered within the scope and spirit of the present disclosure.

[0030] The inference module 212 of the present disclosure may use a deep learning architecture. The deep learning architecture may be embodied in a deep convolutional neural network (CNN). During training, the CNN may be presented with various viewpoints of various object categories. A network designer may want the CNN to output estimates of unseen objects and corresponding poses with high confidence. Before training, the output generated by the CNN is likely to be incorrect, so an error may be calculated between the actual output and the target output. The weights of the CNN may then be adjusted so that the CNN output more closely aligns with the target (e.g., ground truth data).

[0031] To adjust the weights, a learning function may be used to calculate a gradient vector for the weights. The gradient may indicate the amount by which a small adjustment to the weights would increase or decrease the error. In the top layer, the gradient may correspond directly to the value of the weights connecting activated neurons in the penultimate layer to neurons in the output layer. In lower layers, the gradient may depend on the value of the weights and the calculated error gradient of the upper layer. The weights may then be adjusted to reduce the error. This method of adjusting weights is sometimes called "backpropagation" because it involves a "backward pass" through the neural network.

[0032] In practice, the error gradient of the weights can be computed over a small number of examples, so that the computed gradient approximates the true error gradient. This approximation is sometimes called stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate of the entire system stops decreasing, or until the error rate reaches a target level.

[0033] 3 illustrates a flow diagram for training one or more machine learning models 300 according to one aspect of the present disclosure. In one configuration, an image (x) may be stored in a data source 302, such as a training server. The data source may store ground truth data samples (y*) corresponding to one or more objects in the image (x).

[0034] The machine learning model 300 may be initialized with a set of parameters (w). The parameters (w) may be used by layers of the machine learning model 300, such as layer 1, layer 2, and layer 3 of the machine learning model 300, to set weights and biases. Layer 3 may be a fully connected layer. During training, the machine learning model 300 receives an image (x) and labels detected objects.

[0035] The machine learning model 300 may output an estimated label (y) for one or more objects in each image (x). The estimated label (y) may be received by a loss function 308. The loss function 308 may compare the predicted label (y) to the ground truth data action (y*). The prediction error is the difference (e.g., loss) between the predicted label (y) and the ground truth data action (y*). The prediction error is output from the loss function 308 to the machine learning model 300. The error may be back-propagated through the machine learning model 500 to update parameters. Training may be performed during an offline phase of the machine learning model 300.

[0036] According to another aspect, additional images (z) are input into one or more machine learning models 300 with known data images (x), allowing the machine learning models 300 to automatically label objects from the images (z) using data from previous images to the models 300. Such newly labeled images and data may then be used in future and subsequent training operations on new datasets. The system may assign labels to new images (z) based on inferences made from the known images (x). In this manner, the machine learning models 300 may train themselves in a self-supervised or weakly supervised manner. Additionally, such implementations may eliminate the need for human annotators to manually label such offline datasets.

[0037] FIG. 4 illustrates an example hardware implementation of an autonomous driving system 400 according to an embodiment of the present disclosure. The autonomous driving system 400 may be a component of a vehicle, a robotic device, or another device. For example, as shown in FIG. 4, the autonomous driving system 400 is a component of an automobile 428. Of course, other devices, such as a bus, a boat, a drone, or a robot, are also contemplated as being targets for using the autonomous driving system 400, and the embodiment of the present disclosure is not limited to the autonomous driving system 400 being a component of an automobile 428.

[0038] According to one aspect, autonomous driving system 400 may be distributed across multiple local and remote components. For example, the components described herein may be components of vehicle 428, or more specifically, certain components may be remote from vehicle 428. As described herein, certain functions, data analysis, machine learning, model creation and modification, etc. may be completed on a remote server. The results of such analysis and processing may then be loaded into vehicle 428 using its various communication components. For simplicity of explanation, certain modules and components are shown as part of vehicle 428, but those skilled in the art will recognize that the modules and components may be remote or external to vehicle 428.

[0039] Autonomous driving system 400 may be implemented using a bus architecture, generally represented by bus 430. Bus 430 may include any number of interconnected buses and bridges, depending on the particular application and overall design constraints of autonomous driving system 400. Bus 430 couples together various circuits, including one or more processors and / or hardware modules, represented by processor 420, communications module 422, position module 418, sensor module 402, movement module 426, planning module 424, and computer-readable medium 414. Bus 430 may also couple various other circuits, such as timing sources, peripherals, voltage regulators, and power management circuits. Such circuits are well known in the art and will not be described further.

[0040] The autonomous driving system 400 includes a transceiver 416 coupled to a processor 420, a sensor module 402, an artificial neural network module 408, a communication module 422, a position module 418, a movement module 426, a planning module 424, and a computer-readable medium 414. The transceiver 416 is coupled to an antenna 434. The transceiver 416 communicates with various other devices via a transmission medium. For example, the transceiver 416 may receive commands via transmission from a user or a remote device. As another example, the transceiver 416 may transmit driving statistics and information from the artificial neural network module 408 to a server (not shown). According to one aspect, the artificial neural network 408 may be an offline system that communicates with input devices and other vehicle systems via a wireless network, as described herein. In such a configuration, the artificial neural network 408 may develop and train autonomous driving models offline for subsequent deployment or redeployment.

[0041] The autonomous driving system 400 includes a processor 420 coupled to a computer-readable medium 414. The processor 420 performs processes, including executing software stored on the computer-readable medium 414, that provide functionality according to the present disclosure. The software, when executed by the processor 420, causes the autonomous driving system 400 to perform various functions described for a particular device, such as the vehicle 428 or any of the modules 402, 408, 414, 416, 418, 420, 422, 424, and 426. The computer-readable medium 414 may also be used to store data manipulated by the processor 420 when executing the software.

[0042] The sensor module 402 may be used to obtain measurements via different sensors, such as a first sensor 406 and a second sensor 404. The first sensor 406 may be a visual sensor, such as a stereo camera or a red-green-blue (RGB) camera for capturing 2D images. The second sensor 404 may be a ranging sensor, such as a light detection and ranging (LIDAR) sensor or a radio detection and ranging (RADAR) sensor. Of course, aspects of the present disclosure are not limited to the aforementioned sensors, as other types of sensors, such as thermal, sonar, and / or laser, are also contemplated for either of the sensors 404, 406. The measurements of the first sensor 406 and the second sensor 404 may be processed by one or more of the processor 420, the sensor module 402, the artificial neural network module 408, the communication module 422, the position module 418, the movement module 426, and the planning module 424, in combination with the computer-readable medium 414 to implement the functionality described herein. In one configuration, data captured by the first sensor 406 and the second sensor 404 may be transmitted to an external device via the transceiver 416. The first sensor 406 and the second sensor 404 may be coupled to or in communication with the vehicle 428.

[0043] The location module 418 may be used to determine the location of the vehicle 428. For example, the location module 418 may use a global positioning system (GPS) to determine the location of the vehicle 428. The communication module 422 may be used to facilitate communication via the transceiver 416. For example, the communication module 422 may be configured to provide communication capabilities via different wireless protocols, such as WiFi, Long Term Evolution (LTE), 3G, etc. The communication module 422 may also be used to communicate with other components of the vehicle 428 that are not modules of the autonomous driving system 400.

[0044] The locomotion module 426 may be used to facilitate movement of the vehicle 428. As one example, the locomotion module 426 may control the movement of the wheels. As another example, the locomotion module 426 may be in communication with a power source of the vehicle 428, such as an engine or a battery. Of course, aspects of the present disclosure are not limited to providing locomotion via wheels, and other types of components for providing locomotion are contemplated, such as propellers, treads, fins, and / or jet engines.

[0045] The autonomous driving system 400 also includes a planning module 424 for planning a route or controlling the movement of the vehicle 428 via a movement module 426 based on the analysis performed by the artificial neural network 408. In one configuration, the planning module 424 overrides user input when the user input is expected (e.g., predicted) to cause a collision. The module may be a software module executing on the processor 420, a software module resident / stored on the computer-readable medium 414, one or more hardware modules coupled to the processor 420, or some combination thereof.

[0046] The artificial neural network 408 may be in communication with the sensor module 402, the transceiver 416, the processor 420, the communication module 422, the position module 418, the movement module 426, the planning module 424, and the computer-readable medium 414. In one configuration, the artificial neural network 408 receives sensor data from the sensor module 402. The sensor module 402 may receive sensor data from the first sensor 406 and the second sensor 404. According to aspects of the present disclosure, the sensor module 402 may filter the data, remove noise, encode the data, decode the data, merge the data, extract frames, or perform other functions. In an alternative configuration, the artificial neural network 408 may receive the sensor data directly from the first sensor 406 and the second sensor 404.

[0047] Designing a system to properly manage multimodal and multi-label datasets can be particularly challenging. For this reason, aspects of the present disclosure are designed and configured so that a dataset management system maintains and versions the dataset. According to one or more aspects, automatically labeling a multimodal dataset enables several possibilities. For example, a LiDAR and a camera may be synchronized to perform 2D detection on images and elevate such detection to 3D rendering or detection. Aspects of the present system also enable additional system-wide functionality, such as depth estimation from a monocular camera or from LiDAR input during training time.

[0048] As shown in FIG. 4 , the artificial neural network 408 may include a feature extractor 410, a classifier 412, and an inference module 432. The feature extractor 410, the classifier 412, and the inference module 432 may be components of the same or different artificial neural networks, such as a deep convolutional neural network (CNN). The artificial neural network 408 is not limited to a CNN and may be another type of artificial neural network, such as a support vector machine (SVM). The feature extractor 410 receives data streams from a first sensor 406 and a second sensor 404. The data streams may include a 2D RGB image from the first sensor 406 and LIDAR data points from the second sensor 404. Different spatial environments may be captured in each data stream. Furthermore, the LIDAR data points may be separated into multiple streams. For example, one LIDAR data stream may provide a bird's-eye view spatial representation, and another LIDAR data stream may provide a forward-looking spatial representation. The data streams may include multiple frames, such as image frames.

[0049] The feature extractor 410 extracts (e.g., identifies) areas of interest from each frame of the data stream. For example, the feature extractor 410 may be trained to extract features of objects of interest. As another example, the feature extractor 410 may be trained to extract features of different terrain features, such as roads, sidewalks, buildings, and backgrounds. That is, the feature extractor 410 identifies areas of interest based on the training. The artificial neural network 408 may include one or more feature extractors 440. For example, one feature extractor 410 may be configured to detect objects, and another feature extractor 410 may be configured to segment different elements of the data, such as roads, sidewalks, buildings, and backgrounds. In another example, each data stream output from the sensors 402, 404 may be received by a separate feature extractor 410.

[0050] The feature extractor 410 may encode each set of features in a multidimensional tensor. The classifier 412 classifies each set of features received from the feature extractor 410. The inference module 432 may classify (e.g., label) one or more objects of interest and confine each object of interest to a particular region using a bounding box, as described herein. The feature extractor 410, the classifier 412, and the inference module 432 may be referred to as an object detection model.

[0051] The classifier 412 / inference module 432 may output the labels and bounding boxes to one or more of the sensor module 402, the transceiver 416, the processor 420, the communication module 422, the location module 418, the movement module 426, the planning module 424, and the computer-readable medium 414. For example, the classified features may be output to the processor 420 to determine a course of action that may be taken based on the detected objects and their respective positions in relation to the vehicle 428. The action may be retrieved from the computer-readable medium 414 or the planning module 424. Further, based on the classified features, one or more of the modules 402, 408, 414, 416, 418, 420, 422, 424, 426 may determine or update an action for the vehicle 428. The planning module 424 may perform route planning, collision avoidance, or other planning functions based on the classified features.

[0052] Also, more specifically, in aspects of the present disclosure, the artificial neural network 408 may be a remote system or component that is used offline to train or update one or more models and automatically label datasets with unknown objects. Offline processing may be performed as described herein to update object detection models that may be downloaded to the vehicle 428 for online execution. As described herein, the artificial neural network 408 may also rely on previously properly trained models to identify and automatically label objects in new datasets. The system may track when new models are obtained using the automatic labels, as well as which models and unlabeled data led to the automatic labels.

[0053] FIG. 5 illustrates a method 500 for labeling a dataset according to one or more aspects of the present disclosure. The system may include an inference module, such as an artificial neural network component, configured to automatically infer and label previously unlabeled data by leveraging a suitably versioned model, including model weights and an identification of the dataset on which the model was trained, as described herein. According to one aspect, the system may receive a dataset, as shown in block 502. As described herein, the system, or certain components of the system, may be offline components. The dataset may be obtained from an online system collected from operational sensors on a vehicle or from another source. According to one aspect, the dataset may include multimodal data obtained from onboard sensors, such as, but not limited to, LiDAR, RADAR, and RGB cameras.

[0054] As shown in block 504, the system may identify unlabeled data in the dataset. According to one embodiment, the dataset may include both labeled and unlabeled data. The labeled data, according to one embodiment, may provide contextual information used to infer labels for the unlabeled data. As shown in block 506, the dataset may be input to a trained neural network. The trained neural network may have been previously trained with a model 508 that includes a second dataset that includes labeled data. According to one embodiment, the previously trained model may be a robust, appropriately versioned, and weighted model.

[0055] As shown in block 510, the system may use a previously trained model to infer automatic labels for unlabeled data. Leveraging a previously well-trained model, inference may be performed on the current dataset and unlabeled data to automatically generate labels to apply to such data. As described herein, data from multiple sensors may be synchronized to infer labels for multimodal scenes. As shown in block 512, additional networks may be trained using newly labeled datasets. The system may track that new models have been obtained using automatic labels, as well as which models and unlabeled data led to the automatic labels. The system may also be configured to utilize sparsely labeled datasets. In this case, automatic labels can be used to fill in the gaps, allowing for training in semi-supervised and weakly supervised fashions.

[0056] Based on the teachings, one skilled in the art should understand that the scope of the present disclosure is intended to cover any aspect of the present disclosure, whether implemented independently of or in combination with any other aspect of the present disclosure. For example, an apparatus may be implemented or a method may be practiced using any number of the described aspects. Furthermore, the scope of the present disclosure is intended to cover such apparatus or methods that are implemented using other structure, function, or structure and function in addition to or other than the various aspects of the present disclosure described. It should be understood that any aspect of the present disclosure may be embodied by one or more elements of a claim.

[0057] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0058] While particular aspects are described herein, many variations and permutations of such aspects are within the scope of the present disclosure. While certain benefits and advantages of the preferred aspects are mentioned, the scope of the present disclosure is not intended to be limited to particular benefits, applications, or purposes. Rather, aspects of the present disclosure are intended to be broadly applicable to different technologies, system configurations, networks, and protocols, some of which are illustrated by way of example in the drawings of the preferred aspects and in the following description. The detailed description and drawings are not limiting but merely illustrative of the present disclosure, the scope of which is defined by the appended claims and their equivalents.

[0059] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, examining, looking up (e.g., looking up in a table, database, or another data structure), ascertaining, and the like. Furthermore, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Furthermore, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0060] As used herein, a phrase referring to "at least one" of a list of items refers to any combination of such items, including single members. By way of example, "at least one of a, b, or c" is intended to cover a, b, c, ab, ac, bc, and abc.

[0061] The various illustrative logic blocks, modules, and circuits described in connection with this disclosure may be implemented or embodied using a processor specially configured to perform the functions discussed in this disclosure. The processor may be a neural network processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof, designed to perform the functions described herein. Alternatively, a processing system may include one or more neuromorphic processors for implementing the neuron models and models of neural systems described herein. The processor may be a microprocessor, controller, microcontroller, or state machine specially configured as described herein. Alternatively, a processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or other special configurations as described herein.

[0062] The steps of a method or algorithm described in connection with the present disclosure may be embodied directly in hardware, in a software module executed by a processor, or a combination of the two. The software module may reside in storage or machine-readable media such as random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, removable disk, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. A software module may comprise a single instruction, or many instructions, and may be distributed across several different code segments, distributed among different programs, and distributed across multiple storage media. A storage medium may be coupled to the processor such that the processor is capable of reading information from, and writing information to, the storage medium. Alternatively, the storage medium may be integral to the processor.

[0063] The methods disclosed herein include one or more steps or actions for achieving the described method. Method steps and / or actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be changed without departing from the scope of the claims.

[0064] The described functionality may be implemented in hardware, software, firmware, or any combination thereof. When implemented in hardware, an example hardware configuration may include a processing system within a device. The processing system may be implemented using a bus architecture. The bus may include any number of interconnecting buses and bridges, depending on the particular application and overall design constraints of the processing system. The bus may interconnect various circuits, including a processor, machine-readable media, and a bus interface. The bus interface may be used to, among other things, connect a network adapter to the processing system via the bus. The network adapter may be used to implement signal processing functions. In certain aspects, a user interface (e.g., keypad, display, mouse, joystick, etc.) may also be connected to the bus. The bus may also connect various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, etc., which are well known in the art and will not be described further.

[0065] The processor may be responsible for managing the bus and for processing, including executing software stored on a machine-readable medium. Software shall be taken to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0066] In a hardware implementation, the machine-readable medium may be part of a processing system separate from the processor. However, as one skilled in the art will readily appreciate, the machine-readable medium, or any portion thereof, may be external to the processing system. By way of example, the machine-readable medium may include a transmission line, a carrier wave modulated by data, and / or a computer product separate from the device, any of which may be accessed by the processor via a bus interface. Alternatively, or in addition, the machine-readable medium, or any portion thereof, may be integrated into the processor, such as may be with a cache and / or special register files. While various components discussed may be described as being in a particular location, such as a local component, they may also be organized in various ways, such as with particular components configured as part of a distributed computing system.

[0067] The machine-readable medium may include several software modules. The software modules may include a transmitting module and a receiving module. Each software module may reside on a single storage device or may be distributed across multiple storage devices. As an example, a software module may be loaded from a hard drive into RAM when a triggering event occurs. During execution of a software module, the processor may load some instructions into a cache to increase access speed. One or more cache lines may then be loaded into a special-purpose register file for execution by the processor. When referring to the functionality of a software module below, it will be understood that such functionality is performed by the processor when executing instructions from that software module. It will further be understood that aspects of the present disclosure provide improved functionality for a processor, computer, machine, or other system that implements such aspects.

[0068] If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium, which includes both computer storage media and communication media including any storage medium that facilitates transfer of a computer program from one place to another.

[0069] Furthermore, it should be understood that modules and / or other suitable means for implementing the methods and techniques described herein can be downloaded and / or otherwise obtained by a user terminal and / or base station, as appropriate. For example, such a device can be coupled to a server to facilitate the transfer of means for implementing the methods described herein. Alternatively, the various methods described herein can be provided via storage means, such that the user terminal and / or base station can obtain the various methods when coupling or providing storage means to the device. Furthermore, any other suitable technique for providing the methods and techniques described herein to a device can be utilized.

[0070] It is to be understood that the claims are not limited to the precise configuration and components described above. Various modifications, changes and variations may be made in the arrangement, operation and details of the methods and apparatus described above without departing from the scope of the claims. The invention disclosed in this specification includes the following aspects. [Aspect 1] 1. A method for managing a dataset, comprising: receiving a first dataset including unlabeled data; inputting the first dataset into a network trained with an existing model; labeling the unlabeled data of the first dataset using inferences from the trained network to generate a labeled dataset; and training an untrained network using the labeled dataset to generate a new model; A method comprising: [Aspect 2] 2. The method of embodiment 1, wherein the first data set includes data from two or more sensors. Aspect 3 3. The method of claim 2, wherein the two or more sensors include at least two of a LiDAR sensor, a RADAR sensor, an RGB camera, a monocular camera, and a stereo camera. Aspect 4 2. The method of claim 1, wherein the existing model comprises a versioned model. Aspect 5 5. The method of embodiment 4, wherein the versioned model comprises a weighted model. Aspect 6 5. The method of embodiment 4, wherein the versioned model includes an identification of a training dataset. Aspect 7 2. The method of embodiment 1, further comprising tracking the new model as auto-labeled. Aspect 8 2. The method of embodiment 1, further comprising tracking the existing model and first dataset. Aspect 9 2. The method of embodiment 1, wherein training the untrained network comprises semi-supervised training. Aspect 10 2. The method of embodiment 1, wherein labeling the unlabeled dataset comprises labeling the unlabeled dataset according to a first ontology. Aspect 11 11. The method of embodiment 10, wherein the unlabeled dataset is labeled according to a second ontology. Aspect 12 1. A system for managing a dataset, comprising: one or more processors; a memory communicatively coupled to the one or more processors, the memory comprising: an inference module including instructions that, when executed by the one or more processors, cause the one or more processors to: receiving a first dataset including unlabeled data; inputting the first dataset into a network trained with an existing model; labeling the unlabeled data of the first dataset using inferences from the trained network to generate a labeled dataset; and a memory storing an inference module that generates a labeled dataset by training an untrained network using the labeled dataset; and A system comprising: Aspect 13 13. The system of aspect 12, wherein the first data set includes data from two or more sensors. Aspect 14 The system of aspect 13, wherein the two or more sensors include at least two of a LiDAR sensor, a RADAR sensor, an RGB camera, a monocular camera, and a stereo camera. Aspect 15 13. The system of claim 12, wherein the existing model comprises a versioned model. Aspect 16 16. The system of embodiment 15, wherein the versioned model comprises a weighted model. Aspect 17 16. The system of embodiment 15, wherein the versioned model includes an identification of a training dataset. Aspect 18 13. The system of claim 12, wherein the new model is tracked as auto-labeled. Aspect 19 13. The method of embodiment 12, wherein the untrained network comprises semi-supervised training. Aspect 20 A non-transitory computer-readable medium managing a data set and containing instructions, comprising: The instructions, when executed by one or more processors, cause the one or more processors to: receiving a first dataset including unlabeled data; inputting the first data set into a network trained on an existing model; labeling the unlabeled data of the first dataset using inferences from the trained network to generate a labeled dataset; A non-transitory computer-readable medium for training an untrained network using the labeled dataset.

Claims

1. 1. A method of managing a dataset by a system, comprising: receiving a first data set; determining whether the received first data set includes unlabeled data; inputting the first data set into a network trained with an existing model; labeling the unlabeled data of the first dataset using inferences from the trained network to generate a labeled dataset; and training an untrained network using the labeled dataset to generate a new model; Including, Inferring labels for the unlabeled data includes leveraging other datasets having objects similar to the unlabeled objects in the received first dataset to infer labels for the unlabeled data.

2. The method of claim 1 , wherein the first data set includes data from two or more sensors.

3. The method of claim 2 , wherein the two or more sensors include at least two of a LiDAR sensor, a RADAR sensor, an RGB camera, a monocular camera, and a stereo camera.

4. The method of claim 1 , wherein the existing model comprises a versioned model.

5. The method of claim 4 , wherein the versioned model comprises a weighted model.

6. The method of claim 4 , wherein the versioned model includes an identification of a training data set.

7. The method of claim 1 , further comprising recording the new model as auto-labeled.

8. The method of claim 1 , further comprising recording the existing model and the first data set.

9. The method of claim 1 , wherein training the untrained network comprises semi-supervised training.

10. The method of claim 1 , wherein labeling the unlabeled dataset comprises labeling the unlabeled dataset according to a first ontology.

11. The method of claim 10 , wherein the unlabeled dataset is labeled according to a second ontology.

12. 1. A system for managing a dataset, comprising: one or more processors; a memory communicatively coupled to the one or more processors, the memory comprising: an inference module including instructions that, when executed by the one or more processors, cause the one or more processors to: receiving a first data set; determining whether the received first data set includes unlabeled data; inputting the first data set into a network trained with an existing model; labeling the unlabeled data of the first dataset using inferences from the trained network to generate a labeled dataset; a memory for storing an inference module that generates a new model by training an untrained network using the labeled dataset; and Equipped with Inferring a label for the unlabeled data includes leveraging other datasets having objects similar to the unlabeled object in the received first dataset to infer a label for the unlabeled data.

13. The system of claim 12 , wherein the first data set includes data from two or more sensors.

14. The system of claim 13 , wherein the two or more sensors include at least two of a LiDAR sensor, a RADAR sensor, an RGB camera, a monocular camera, and a stereo camera.

15. The system of claim 12 , wherein the existing model comprises a versioned model.

16. The system of claim 15 , wherein the versioned model comprises a weighted model.

17. The system of claim 15 , wherein the versioned model includes an identification of a training data set.

18. The system of claim 12 , wherein the new model is recorded as auto-labeled.

19. The system of claim 12 , wherein the untrained network comprises semi-supervised training.

20. A non-transitory computer-readable medium managing a data set and containing instructions, comprising: The instructions, when executed by one or more processors, cause the one or more processors to: receiving a first data set; determining whether the received first data set includes unlabeled data; inputting the first data set into a network trained on an existing model; labeling the unlabeled data of the first dataset using inferences from the trained network to generate a labeled dataset; training an untrained network using the labeled dataset; Inferring a label for the unlabeled data includes leveraging other datasets having objects similar to the unlabeled object in the received first dataset to infer a label for the unlabeled data.

Images