System and method for acquiring training data

Abstract

To provide systems and methods for machine learning for obtaining training data.SOLUTION: In deep learning systems used for applications such as autonomous driving, a training data generating system includes receiving sensor data and applying a neural network to the sensor data. A trigger classifier is applied to an intermediate result of the neural network to determine a classifier score for the sensor data. Based at least in part on the classifier score, determination is made whether to transmit at least a portion of the sensor data via a computer network. Upon positive determination, the sensor data is transmitted and used to generate training data.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This application incorporates by reference in its entirety the entire U.S. Provisional Patent Application No. 62 / 731,651, filed on September 14, 2018, entitled "NEURAL NETWORK TRAINING", which is incorporated by reference in its entirety herein.

[0002] Any and all applications identified in the application data sheet as having foreign or domestic priority claims filed with this application are incorporated herein by reference in their entirety under 37 C.F.R. § 1.57.

[0003] This disclosure relates to systems and methods for machine learning. More particularly, this disclosure relates to methods for generating training data.

Background Art

[0004] Deep learning systems used in applications such as autonomous driving are developed by training machine learning models. Typically, the performance of a deep learning system is at least partially limited by the quality of the training set used to train the model. Often, significant resources are invested in the collection, curation, and annotation of training data. The effort required to create a training set can be substantial and is often cumbersome. Furthermore, it is often difficult to collect data for specific use cases where the machine learning model needs improvement.

Brief Description of the Drawings

[0005] The following drawings and related descriptions are provided to illustrate embodiments of the present disclosure and do not limit the scope of the claims. Many aspects of the present disclosure and associated advantages will be more readily understood by referring to the following detailed description, which will be better understood when interpreted in conjunction with the accompanying drawings.

[0006] [Figure 1A] This is a schematic diagram showing a vehicle that travels on a road and detects tires placed on the road.

[0007] [Figure 1B] This is a block diagram showing one embodiment of a system for generating training data.

[0008] [Figure 2] This flowchart illustrates an embodiment of the process for applying a trigger classifier to the intermediate results of a machine learning model.

[0009] [Figure 3] This flowchart illustrates an embodiment of the process for creating a trigger classifier using the intermediate results of a machine learning model.

[0010] [Figure 4] This flowchart illustrates an embodiment of the process for identifying potential training data using a trigger classifier and transmitting sensor data.

[0011] [Figure 5] This flowchart illustrates an embodiment of the process for deploying training data from data corresponding to use cases identified by the trigger classifier.

[0012] [Figure 6] This flowchart illustrates an embodiment of the process for selecting a classifier on a vehicle and transmitting sensor data.

[0013] [Figure 7] This is a block diagram showing an embodiment of a deep learning system for identifying potential training data. [Modes for carrying out the invention]

[0014] This specification describes one or more innovations, which can be implemented in many ways, including as processes, as apparatus, as systems, as composites, as computer program products embodied on computer-readable storage media, and / or as processors, such as processors configured to execute instructions, stored on and / or provided by memory linked to a processor. In this specification, these embodiments, or any other form an innovation may take, may be referred to as techniques. Generally, the order of the steps in a disclosed process may be modified within the scope of the innovation. Unless otherwise specified, a processor or a component such as memory described as configured to perform a task may be implemented as a general-purpose component temporarily configured to perform that task at a given time, or as a specific component manufactured to perform that task. As used herein, the term “processor” refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0015] A detailed description of one or more embodiments of one or more innovations is provided below, along with accompanying diagrams illustrating the principles of the innovations. While the innovations are described in relation to these embodiments, they are not limited to any of these embodiments. The scope of the innovations is limited only by the claims, and the innovations include numerous alternative forms, modifications, and equivalent forms. Many specific details are given in the following description to provide a complete understanding of the innovations. These details are provided for illustrative purposes only, and the innovations may be practiced in accordance with the claims without some or all of these specific details. For clarity, and to avoid unnecessarily obscuring the innovations, known technical material in the art related to the innovations is not described in detail.

[0016] introduction This specification describes innovations that address at least the following technical problems: Effective machine learning techniques depend on the training dataset used to inform the underlying machine learning model. For example, a neural network may be trained using thousands, hundreds of thousands, or millions of examples. During training, these examples may be used to tune the neural network's parameters (e.g., weights, biases, etc.). Furthermore, these examples may be used to tune the neural network's hyperparameters (e.g., several layers). Thus, access to training data is a constraint on the use of such machine learning techniques.

[0017] As machine learning models become more complex, such as deeper neural networks, the need for larger training datasets increases accordingly. These deeper neural networks may require more training examples compared to shallower neural networks to ensure their generalizability. For example, a neural network may be trained to be highly accurate with respect to the training data, but it may not generalize well to unseen future examples. In this example, the neural network can benefit from the additional examples included in the training data.

[0018] It can be understood that obtaining training data can present a very significant technical hurdle. For example, a particular machine learning model may be used to classify features or objects contained in an image. In this example, the machine learning model may learn to recognize a first object (e.g., a car) from a second object (e.g., a stop sign). The effectiveness of these machine learning models can be limited by the number of examples of features or objects. For example, an entity may request a machine learning model to recognize a motorcycle being ridden on the road. In another example, the entity may request a machine learning model to recognize a motorcycle being carried behind a car. Without sufficient training examples of these examples, the machine learning model may not be accurate enough to be usable in its recognition. In general, an entity may require a significant amount of effort to have a person label the image to include a particular feature or object. For example, a person may have to manually re-examine the image and then assign labels to parts of the image to correspond to a particular feature or object.

[0019] One embodiment is a system and method that addresses this problem by rapidly generating training data. In one embodiment, the training data may include examples of any desired learnable features. With respect to computer vision, training data including examples of any desired objects or features in an image can be rapidly generated. These objects or features may typically represent “edge cases” that are difficult to identify. For example, training data may be generated that includes images of a complex scene requested by an entity. In this example, the entity may prefer to obtain an image depicting a motorcycle behind or in front of a vehicle (for example, a motorcycle being carried on the front rack of a public bus).

[0020] The entities described above may leverage a large number of vehicles (e.g., thousands, millions) traveling across various roads or other navigable areas of the world. These vehicles may include sensors (e.g., cameras) or may have access to them in other ways. As these vehicles move, they may capture sensor information. For example, sensor information may be captured during the normal operation of the vehicle. Sensor information may be used by the vehicle for certain autonomous driving functions, such as lane navigation. However, in one embodiment, the system includes circuitry and software that enables the vehicle to collect image features or examples of objects requested by the entities for use as training data for a machine learning system.

[0021] For example, a classifier (e.g., a small or shallow neural network, a support vector machine, etc.) can be uploaded to at least a part of a vehicle. The vehicle can acquire sensor information (e.g., images, videos) during normal operation, and the classifier can be configured to identify specific features or objects represented in the sensor information. Before being provided to the vehicle, these classifiers can be trained to classify images to include specific image features or objects. For example, a limited number of examples (e.g., hundreds, thousands, etc.) of specific image features or objects can be used to train these classifiers. As described, the classifier can then classify sensor data using information from an intermediate layer of a machine learning model executed on the vehicle. An exemplary machine learning model can include a convolutional network. This exemplary machine learning model can be used, at least in part, for the above-described autonomous driving function. Thus, the classifier can utilize an existing exemplary machine learning model.

[0022] A number of these classifiers can be uploaded to a computer system within the vehicle so that the classifier can be used to recognize specific image features or objects associated with the classifier. The captured images designated by the classifier as including the specific features or objects are then transmitted to a central server system and can be used as training data for a neural network system. Since the classifier can utilize an existing machine learning model already being executed by the vehicle in normal operation, the classifier can be efficient in terms of processing requirements. Further, there can be a number of vehicles operating in heterogeneous environments, increasing the possibility of obtaining examples that are difficult to find for certain specific features of "edge cases". In this way, an entity can quickly obtain sensor information (e.g., images) representing specific image features or objects of interest to the entity. For example, a classifier (e.g., a small or shallow neural network, a support vector machine, etc.) can be uploaded to at least a part of a vehicle. The vehicle can acquire sensor information (e.g., images, videos) during normal operation, and the classifier can be configured to identify specific features or objects represented in the sensor information. Before being provided to the vehicle, these classifiers can be trained to classify images to include specific image features or objects. For example, a limited number of examples (e.g., hundreds, thousands, etc.) of specific image features or objects can be used to train these classifiers. As described, the classifier can then classify sensor data using information from an intermediate layer of a machine learning model executed on the vehicle. An exemplary machine learning model can include a convolutional network. This exemplary machine learning model can be used, at least in part, for the above-described autonomous driving function. Thus, the classifier can utilize an existing exemplary machine learning model.

[0023] In this specification, the objects or features to be learned can represent any real-world objects, scenarios, features, etc. that can be captured within sensor data. Exemplary objects or features can include a tire in a road, an exit of a tunnel, a bike, a tree with branches extending into the road, a scene where a vehicle is oriented in a particular direction or performing a particular action or operation, etc. Further, mention is made here of identifying training data for a use case or purpose. Exemplary use cases or purposes can include identifying one or more objects, features, etc. Further, although this specification describes a vehicle that acquires sensor information such as an image, it can be understood that the features described herein can be widely applicable. For example, a classifier can be provided to a user device (e.g., a smartphone) and used to recognize specific image features or objects. As another example, a classifier can be used in an airplane, a drone, an autonomous vehicle, etc.

[0024] Generation of Training Data A neural network training technique for identifying additional training data related to a specific use case is disclosed. By identifying and collecting additional training data, particularly data for use cases that are difficult to analyze correctly, a deep learning system can be retrained to improve its performance. For example, a difficult use case can be identified and data can be collected based on that use case. Then, a new machine learning model with better performance than the old model can be trained using the newly collected data. In various embodiments, an existing machine learning model is utilized with a trigger classifier to identify relevant training data. Then, the relevant training data is returned for processing to create new training data. In some embodiments, an initial dataset representative of the target use case is created and used to create a trigger classifier.

[0025] For example, a deep learning system for autonomous driving may have difficulty analyzing and identifying tunnel exits. The training dataset is created using positive and negative examples of tunnel exits. In some embodiments, the trigger classifier is trained on the initial training dataset using the intermediate outputs of layers in an existing machine learning model. In some embodiments, that layer is an intermediate layer. For example, data from the training set is fed into an existing machine learning model, and the output of the second to last layer of the model is used as input to train the trigger classifier. In some embodiments, the trigger classifier is a support vector machine trained offline from the deep learning application to be deployed. Once trained, the trigger classifier can be installed or deployed to run alongside a deep learning system already in use within the vehicle's autonomous driving system. For example, the trigger classifier can be deployed via a wireless network to download and install the trigger classifier into the vehicle. The trigger classifier is applied to the intermediate outputs of the same layer in the deployed deep learning system to determine the classifier score. In some embodiments, the input to the trigger classifier is a convolutional neural network applied to sensor data captured by the autonomous vehicle, for example, image data captured by a camera on the vehicle. This is the intermediate output of a network (CNN) layer.

[0026] In some embodiments, a trigger classifier, implemented using a single support vector machine, a small neural network, or another suitable classifier, may be applied to the entire captured image and / or specific locations within the image. For example, the trigger classifier may be applied to any location or subset of locations within the image. The trigger classifier may be applied to spatially effectively scan the features of a neural network to identify small parts such as a shopping cart or an animal. Once applied, the trigger classifier determines a classifier score, and depending on the score, sensor data is identified and retained as potentially useful training data. As an example, the trigger classifier scores sensor data from a camera based on how likely the data is to represent a tunnel exit. Sensor data with a high score and a high probability of representing a tunnel exit is retained and flagged for use as training data. In some embodiments, trigger characteristics such as filters are applied to the trigger classifier to determine conditions that must be met to proceed with determining the classifier score, situations in which the classifier score exceeds a threshold, and / or conditions required to retain sensor data. For example, in some embodiments, sensor data is scored and collected at most once per interval, such as once every 30 minutes. In some embodiments, the classifier score must exceed a threshold for sensor data to be collected and retained. If the sensor data meets the set threshold, the sensor data is retained and used as potential new training data. In one embodiment, the sensor data is uploaded wirelessly to a server managing the training data system.

[0027] In some embodiments, additional metadata such as location, road type, vehicle type, whether the vehicle is left-hand drive or right-hand drive, time, classifier score, length of time since the last transmitted sensor data, and / or vehicle control parameters / operating conditions such as speed, acceleration, steering, braking, and steering angle are collected and retained along with the sensor data. In various embodiments, the data and metadata are sent to a computer data server and used to create new training datasets to improve the application of the deep learning system for a specific use case. For example, retained sensor data associated with an identified tunnel exit is identified by a trigger classifier and used to create additional training data for identifying tunnel exits.

[0028] In some embodiments, after uploading, the sensor data is reviewed and annotated to create a new training dataset used to improve the vehicle's autonomous driving capabilities. For example, data may be annotated as positive samples of tunnel exits and used to supplement the original training dataset, which may contain more use cases. The new machine learning model is trained using the newly curated dataset to improve the autonomous vehicle's neural network and is then deployed to the vehicle as an update to the autonomous vehicle system. The newly deployed machine learning model has an improved ability to detect specific use cases (e.g., tunnel exits) that are targeted by trigger classifiers. As an example, the improved model will have improved accuracy and performance in identifying tunnel exits. Further examples of use cases include trigger classifiers trained to identify specific objects (e.g., shopping carts, animals, etc.), road conditions, weather, driving patterns, hazards, etc.

[0029] In various embodiments, the trigger classifier does not require updating the vehicle's core software, such as components of a deep learning system used for autonomous driving. New and updated trigger classifiers can be developed and deployed to a fleet of vehicles. Linked to and associated with the vehicle's existing neural network software, these new and updated trigger classifiers can be enforced on vehicles much more frequently, and with little to no impact on core vehicle functionality such as driving, safety systems, and navigation. For example, a trigger classifier can be trained to identify cobblestone roads, deployed to a fleet of vehicles, and begin collecting images and associated data of cobblestone roads within minutes. Using the disclosed techniques, the speed at which relevant training data can be collected for specific use cases is greatly improved with little to no impact on ongoing vehicle operation or the driver or passengers of the vehicle. New trigger classifiers can be deployed without a long and cumbersome installation process. This process can be performed remotely and dynamically, for example, using over-the-air updates, without requiring the vehicle to be transported to a service location. After such updates, the trigger classifier can begin scanning captured images to find images that satisfy trigger conditions, and then upload those images as future training data.

[0030] In some embodiments, sensor data is transmitted and received by different devices. For example, a vehicle with autonomous driving technology, which includes sensors that collect relevant information about its surroundings, receives sensor data from those sensors. In some embodiments, the vehicle is equipped with sensors such as cameras, ultrasonic sensors, radar sensors, LiDAR, and / or other suitable sensors to capture data relevant to autonomous driving. In some embodiments, neural networks are applied to the sensor data. For example, a convolutional neural network (CNN) is applied to received sensor data, such as an image of the road in front of the vehicle. The CNN may be used to identify objects in the captured sensor data, and the result of applying the neural network is used to control the vehicle. As an example, road lane lines are identified and used to keep the vehicle between the identified lane lines.

[0031] In some embodiments, a trigger classifier is applied to the intermediate outputs of a neural network to determine a classifier score for sensor data. For example, the intermediate output of a layer is fed to the trigger classifier that determines the classifier score for sensor data. In some embodiments, the neural network includes multiple intermediate layers, and it is configurable that a specific intermediate output (and corresponding layer) should receive input to the trigger classifier from it. For example, the trigger classifier may be configured to receive the output of the second to last, third to last, fourth to last, and so on. In some embodiments, the intermediate output is the output from any of the intermediate layers of the neural network. In some embodiments, the intermediate output may be the output of the first layer of the neural network. In some embodiments, a decision is made, at least in part, based on the classifier score, to send at least a portion of the sensor data over the computer network. For example, the decision is made based on whether the classifier score exceeds a threshold required to retain the sensor data and send the data for further use. In some embodiments, the decision is made based on the classifier score and whether additional trigger classifier conditions are met. Examples of required conditions may be used to filter the captured sensor data based on the vehicle's location, the time the vehicle has been driving, the vehicle type, and whether the autonomous driving function has recently been deactivated. In various embodiments, sensor data that meets the required conditions and score thresholds is transmitted to a computer server via a computer network, such as WiFi or a cellular network, for further processing. In some embodiments, the data is processed to create new or additional training datasets. In various embodiments, the training data includes both training data and validation data.

[0032] Example of a block diagram Figure 1A is a schematic diagram showing a vehicle moving along a road and collecting training data from its surroundings. In the exemplary block diagram, vehicle 102 is traveling on the road. Vehicle 102 may include sensors such as cameras and radar so that the sensors capture information about vehicle 102 in a sensor volume 104. An exemplary sensor 107 is shown in Figure 1A. For example, vehicle 102 may acquire images of its surroundings. These acquired images may then be analyzed in an attempt to understand the surroundings. For example, the images may be analyzed to classify the objects represented in the images. In this example, the images may be analyzed to identify other vehicles, road markings, trees or other plants, road obstacles, pedestrians, signs, etc. Vehicle 102 may leverage machine learning techniques to analyze the sensor information, as will be described in more detail below. For example, one or more convolutional neural networks may be used to classify the objects contained in the exemplary sensor volume 104. With respect to Figures 1B and 7, an exemplary description of a deep learning system 700 that may be used by vehicle 102 is included below.

[0033] While the above machine learning techniques may be used to analyze sensor information, it should be understood that certain real-world objects or scenarios may be difficult for vehicle 102 to accurately understand or classify. For example, a tire 106 is positioned and indicated on the road that vehicle 102 is traveling on. Being able to recognize this tire 106 can improve the safety and performance of vehicle 102. As an example, if tire 106 is in vehicle 102's path, vehicle 102 may perform an autonomous driving technique to navigate around tire 106. Furthermore, even if the tire is not in vehicle 102's path, recognizing tire 102 can still affect vehicle 102's autonomous driving. For example, another vehicle might suddenly enter vehicle 102's lane to avoid tire 106. Thus, in this example, being able to identify tire 106 can inform vehicle 102 of its anticipated future movements (e.g., preemptively slowing down as another vehicle approaches tire 106).

[0034] Thus, accurately identifying the tire 106 as being contained in the sensor volume 104 can be beneficial to the vehicle 102. However, as mentioned above, being able to identify the tire 106 may require a considerable amount of training data. The training data may include images of numerous tires in all configurations on various roads. The training data can be enhanced by including images of different tires on different roads. Furthermore, the training data can be enhanced by images of different tires on different roads in different driving environments. For example, having images of tires partially embedded in snow on different roads may be advantageous. As another example, having images of deflated tires embedded in dusty roads may be advantageous. Gaining access to such images may present a significant technical challenge.

[0035] As described, one or more classifiers may be trained to recognize tires. For example, classifiers may be trained using a limited set of training examples. These classifiers may then be provided to the vehicle 102 via over-the-air (OTA) updates. For example, OTA updates may be received wirelessly through the vehicle 102 (e.g., via Wi-Fi, via cellular signals such as an LTE network, etc.). The classifiers may then analyze the sensor information acquired by the vehicle 102. If the classifiers detect that a tire is depicted in the sensor information (e.g., an image), the vehicle 102 may transmit the sensor information to an external system for processing. This external system may aggregate such received sensor information to create a training dataset of tires. As described, these training datasets are This can then be used to train a complex machine learning model (e.g., a convolutional neural network) that runs on the vehicle 102. In this way, the capabilities of the machine learning model, and therefore the vehicle 102, can be enhanced to perform autonomous driving tasks.

[0036] Figure 1B is a block diagram illustrating the generation of training data. In this illustrated example, sensor data 108 is received by the vehicle 102. The sensor data 108 may include one or more images or videos depicting the tire 106 shown in Figure 1. This sensor data 108 can be provided to a deep learning system 700 comprising one or more processors, which is included in the vehicle 102. An example of an embodiment of the deep learning system 700 is shown in Figure 1B.

[0037] As shown, the deep learning system 700 may use exemplary machine learning techniques, such as a convolutional neural network, to analyze the received sensor data 108. As illustrated in Figure 2, the sensor data 108 may be preprocessed (e.g., normalized and filtered). It can be understood that the convolutional neural network may include a number of convolutional layers. These convolutional layers may have convolutional filters applied to them so that an output volume is created. In some embodiments, one or more fully connected or dense layers may be used as the final layer to classify features or objects contained in the sensor data 108. As an example, one or more softmax layers or independent logistic classifiers may be used to classify features or objects. In this way, the deep learning system 700 may identify real-world objects, scenarios, etc., contained in the sensor volume 104 around the vehicle 102. Based on the identification of these real-world objects, scenarios, etc., the vehicle 102 may perform an autonomous driving task. Thus, the vehicle 102 may implement a convolutional neural network in its typical operation.

[0038] The deep learning system 700 includes one or more classifiers. For example, classifiers A-N 110A-110N are shown in Figure 1B. These classifiers 110A-110N may have been received via OTA updates to the vehicle 102 (e.g., periodic updates provided to the vehicle). Prior to receiving classifiers 110A-110N, the entity may have trained them to identify each feature or object represented in the sensor data. For example, classifier A 110A may have been trained to identify a snowscape. As another example, classifier N 110N may have been trained to identify tires, motorcycles on the road, etc. The entity may have trained classifiers 110A-110N using limited training data. For example, classifier N 110N may have been trained using 100, 500, and 1000 examples of tires on the road, or a specific type of tire on a specific type of road.

[0039] As shown, classifiers 110A–110N may use information obtained from an intermediate layer of an exemplary machine learning model (e.g., a convolutional neural network). For example, feature 112 may be obtained from an intermediate layer of a convolutional neural network. Since a convolutional neural network can be trained to classify or otherwise identify features or objects in sensor data, the classifiers may leverage this existing capability. As an example, a convolutional neural network may learn to apply convolutional filters to learn features that represent real-world objects. The convolutional neural network may then classify the features to correspond to a particular category or class of real-world objects.

[0040] Therefore, when training classifiers 110A to 110N, they are convolutional. A neural network can be trained using information obtained from one of its intermediate layers. For example, classifier 110N can be trained using a limited training dataset of images depicting tires. In this example, images can be provided to an exemplary convolutional neural network. At a specific intermediate layer of the convolutional neural network, features 112 can be provided to classifier 110N. Classifier 110N can then be trained to assign high classifier scores to images depicting tires. Classifier 110N can be trained to assign low classifier scores to images that do not depict tires, in a discretionary selection. In this way, classifier 110N can leverage the convolutional neural network, which can be used in the typical operation of vehicle 102, as described above.

[0041] As shown in Figure 1B, classifiers 110A to 110N receive features 112 from one of the intermediate layers of the convolutional neural network. Selectively, classifiers 110A to 110N may use features from different intermediate layers. For example, classifier N 110N may use features from a first layer (e.g., layers 4 and 5), while classifier A 110N may use features from a second layer (e.g., layers 6 and 7). During training, a specific layer from which each classifier should receive features may be identified. For example, a specific layer may be identified based on the accuracy of the corresponding training classifier on a validation dataset.

[0042] With respect to the tire 106 shown in Figure 1A, one of the classifiers 110A to 110N can be trained to identify the tire. For example, classifier N 110N can be trained to identify the tire. In this example, classifier N 110N can assign a classifier score to the sensor data 108. In the example shown, classifier N 110N assigns a classifier score greater than the threshold (e.g., 0.5, 0.7, etc.). Thus, the vehicle 102 can transmit the sensor data 108 to an external system (e.g., a training data generation system 120). For example, the vehicle can transmit the sensor data 108 via a network (e.g., the Internet) via Wi-Fi, cellular services, etc.

[0043] Thus, the external system 120 can receive sensor data 108 from a large number of vehicles. For example, the external system 120 can receive images depicting tires from vehicles that happen to pass near the tires during normal operation. Advantageously, these tires may be of different types, may be deflated or deteriorated, may be represented in different road conditions, may be partially blocked, etc. Classifiers 110A-110N may, as an example, use classifier scores that trigger the transmission of a large number of sensor data 108 to the external system 120. For example, some of the images transmitted to system 120 may not contain tires. Thus, in some embodiments, entities may be quickly reviewed and discarded from some of the images. The remaining images may be aggregated into a large training dataset and used to update a machine learning model running on the vehicle. For example, a convolutional neural network may be trained to identify tires. Arbitrarily, bounding boxes or other label information may be assigned to images included in the aggregated training dataset.

[0044] In some embodiments, vehicle 102 may have more classifiers than are currently running. For example, vehicle 102 may have 50, 75, or 100 classifiers. However, during the operation of vehicle 102, vehicle 102 may run 20, 30, 40, or more classifiers. For example, vehicle 102 may determine a classifier score for each of a subset of all classifiers stored by vehicle 102. At will, each classifier may run for a certain period of time before being replaced by another classifier.

[0045] Furthermore, vehicle 102 may execute certain classifiers in response to one or more triggers. For example, vehicle 102 may receive information that identifies a location or approximate location known to have a particular real-world object, feature, or represent a particular scenario. For instance, vehicle 102 may access map information that identifies a tunnel exit in a particular area. In this example, vehicle 102 may ensure that a classifier associated with identifying the tunnel exit is executed when vehicle 102 is approaching the tunnel exit.

[0046] As another example, the external system 120 may selectively receive location information along with the received sensor data. In this way, the external system 120 can identify that a threshold number of vehicles have transmitted sensor data based on the same classifier for a particular real-world area. As an example, the external system 120 may identify that a particular on-ramp has an obstacle on the road. As another example, the external system 120 may identify that a particular on-ramp has a particular type of obstacle on the road. The system 120 can then transmit information to a portion of the vehicles to perform the same classifier when they are near that particular real-world area. In this way, the system 120 can ensure that it can acquire a larger amount of training data based on this same sensor.

[0047] Furthermore, system 120 may instruct vehicles to transmit sensor data even if the classifier described above does not assign a classifier score above a threshold. For example, system 120 may receive sensor data from a threshold number of vehicles close to a real-world location. In this example, system 120 may instruct any vehicles within a threshold distance of the real-world location to transmit sensor data (e.g., images) even if their classifiers do not generate a classifier score above a threshold. Since a classifier can be trained on a training set of a limited number of examples (e.g., 100, 1000 as described above), depending on the angle of a particular vehicle to an object, the classifier of that particular vehicle may not identify the object. However, the sensor data can be useful for generating a robust training set about the object. For example, an object may be partially visible in an image acquired by a particular vehicle and therefore can be useful in a large training set for identifying the object. In this way, the external system 120 may override the classifier and cause a particular vehicle to transmit sensor data.

[0048] It should be understood that in all situations where the external system 120 receives location information or any identifying information, the information may be anonymized. Furthermore, such methods may require the consent of a positive user (e.g., opt-in).

[0049] Example flowchart Figure 2 is a flowchart illustrating an embodiment of the process for applying a trigger classifier to the intermediate output of a machine learning model. In some embodiments, the process in Figure 2 is used to collect and retain sensor data captured by sensors for a machine learning model for autonomous driving, satisfying a specific use case. For example, a specific use case may be associated with the identification of a specific feature, object, scenario, etc. In some embodiments, the process in Figure 2 is performed on a vehicle capable of autonomous driving, whether or not autonomous driving control is enabled. For example, sensor data may be collected immediately after autonomous driving is deactivated or when the vehicle is being driven by a human driver. In some embodiments, the method described by Figure 2 can be applied to other deep learning systems outside the context of autonomous driving to improve the training dataset, particularly for use cases that are difficult to analyze. In various embodiments, the trigger classifier is designed for the intermediate output of the machine learning layers and for the use case. It is trained using the training data provided.

[0050] In some embodiments, multiple triggers and / or multiple classifiers may be used together to identify sensor data for multiple use cases. For example, one trigger may be used to identify tunnels, another for manholes, another for road junctions, and so on. In some embodiments, functional components of the trigger classifier for determining the classifier score and / or applying required conditions are shared among different triggers. In some embodiments, each trigger is specified using a weighted vector, an optional bias, and one or more threshold values ​​for comparing classifier scores. In some embodiments, additional required conditions such as time, vehicle location, and road type are specified for a particular trigger. For example, a trigger may require that tunnel sensor data be captured only at dawn and dusk. As another example, and to help reduce duplicate data, a trigger may require that sensor data be captured only at least every 30 minutes and only after the vehicle has been driven for at least 20 minutes. In various embodiments, trigger thresholds and required conditions are characteristics specified for the trigger classifier.

[0051] In 201, sensor data is received. For example, a vehicle equipped with sensors captures sensor data and provides that sensor data to a neural network running on the vehicle. In some embodiments, the sensor data may be visual data, ultrasonic data, LiDAR data, or other suitable sensor data. For example, images are captured from a high dynamic range forward-facing camera. As another example, ultrasonic data is captured from a lateral ultrasonic sensor. In some embodiments, the vehicle is equipped with multiple sensors for capturing data. For example, in some embodiments, eight surround cameras are mounted on the vehicle to provide a 360-degree view around the vehicle at a range of up to 250 meters. In some embodiments, the camera sensors include a wide-angle forward camera, a narrow-angle forward camera, a rearview camera, a forward-facing side camera, and / or a rear-facing side camera. In some embodiments, ultrasonic and / or radar sensors are used to capture surrounding details. For example, twelve ultrasonic sensors may be mounted on the vehicle to detect both hard and soft objects. In some embodiments, forward-facing radar is used to capture data of the surrounding environment. In various embodiments, radar sensors can capture surrounding details despite heavy rain, fog, dust, and other vehicles. Various sensors are used to capture the environment around the vehicle, and the captured images are provided for deep learning analysis.

[0052] In 203, the sensor data is preprocessed. In some embodiments, one or more preprocessing passes may be performed on the sensor data. For example, the data may be preprocessed to remove noise in order to correct alignment problems and / or blurring. In some embodiments, one or more different filtering passes may be performed on the data. For example, a high-pass filter may be applied to the data and a low-pass filter may be applied to the data in order to separate different components of the sensor data. In various embodiments, the preprocessing steps performed in 203 are optional and / or may be incorporated into a neural network.

[0053] In step 205, deep learning analysis of the sensor data is initiated. In some embodiments, the deep learning analysis is performed on sensor data that has been selectively preprocessed in step 203. In various embodiments, the deep learning analysis is performed using a neural network such as a convolutional neural network (CNN). In various embodiments, the machine learning model is trained offline. It is installed in the vehicle to perform inference on sensor data. For example, the model may be trained to identify road lane lines, obstacles, pedestrians, moving vehicles, parked vehicles, drivable spaces, etc., as needed. In various embodiments, the neural network includes multiple layers, each containing one or more hidden layers.

[0054] In 207, potential training data is identified. For example, sensor data that can be used to train a machine learning model is identified from sensor data analyzed using deep learning analysis. In some embodiments, the identified training data is data associated with a specific use case. For example, a possible use case may involve identifying curved roads, on-ramps, off-ramps, tunnel entrances, tunnel exits, road obstacles, road junctions, road lane lines or markers, drivable spaces, road signs, sign content (e.g., words, numbers, symbols, etc.), and / or other features for autonomous driving as needed. In various embodiments, the use cases depicted in the sensor data are identified by using the intermediate outputs of layers in a neural network used for deep learning analysis, and a trigger classifier. For example, the trigger classifier uses the outputs of the intermediate layers of the neural network to determine a classifier score. Classifier scores that exceed a threshold and pass the required conditions specified with the trigger are identified as potential training data. In various embodiments, the threshold is used to identify positive examples of a use case. For example, a higher classified score indicates that the sensor data is more likely to represent a use case. In some embodiments, the classifier score is a number between a negative and a positive number. Scores closer to a positive number are more likely to represent the target use case. In various embodiments, conditions specified by additional filters, such as time, vehicle type, and location, are used to identify the sensor data for the target use case.

[0055] In 209, identified sensor data is transmitted. For example, sensor data identified in 207 is sent to a computer server for further processing. In some embodiments, further processing includes creating a training set using the identified sensor data. In various embodiments, the sensor data is transmitted wirelessly from the vehicle to a data center, for example, via WiFi or a cellular connection. In some embodiments, metadata is transmitted along with the sensor data. For example, metadata may include vehicle control and / or operating parameters such as speed, acceleration, braking, whether autonomous driving was enabled, steering angle, classifier score, time, timestamp, location, vehicle type, etc. Further metadata may include the time since the last sensor data was transmitted, vehicle type, weather conditions, road conditions, etc.

[0056] In 211, post-processing of the data is performed. In some embodiments, different post-processing techniques are used to improve quality and / or reduce the amount of data required to represent the data. In some embodiments, the output of the deep learning analysis is merged with the results of deep learning applied to other sensors. In some embodiments, post-processing is used to facilitate analysis performed on different sensor data. The processed data may be used to control the vehicle. Additional information related to the data may also be processed in 211. For example, information such as the settings of the autonomous driving system, including which autonomous driving functions are enabled, may be combined with the deep learning analysis. Other information may include vehicle operation and / or control parameters, and / or environmental data such as maps, terrain, and / or GPS data. In some embodiments, post-processing may include combining the results of deep learning analysis performed on data from other sensors to create a unified representation of the vehicle's surrounding environment. In some embodiments, the post-processing step in 211 is an optional step.

[0057] In 213, the results of the deep learning analysis are provided to the vehicle control. For example, the results are used by a vehicle control module to control the vehicle for autonomous driving. In some embodiments, the vehicle control can adjust the vehicle's speed and / or steering. In various embodiments, the vehicle control may be disabled, but the intermediate results of the deep learning analysis in 205 are used to identify training data in 207 and transmit the identified sensor data in 209. In this way, the deep learning analysis can be used to identify and retain appropriate training data even when the vehicle is not under the control of the autonomous driving system. In various embodiments, when the autonomous driving system is active, the sensor data is identified and retained.

[0058] Figure 3 is a flowchart illustrating an embodiment of the process for creating a trigger classifier using the intermediate results of a machine learning model. In some embodiments, the process in Figure 3 is used to train a trigger classifier to identify and retain relevant sensor data for a specific use case. For example, sensor data processed by a deep learning system during its normal use includes a subset of data useful as training data. The trigger classifier can be trained to identify use cases such as tunnel entrances, tunnel exits, road junctions, curves, on-ramps, and other suitable features useful for autonomous driving, using the intermediate results of a deep learning system for autonomous driving. By utilizing the intermediate results of a deep learning system with the trigger classifier, the efficiency of identification and collection is significantly improved. In various embodiments, the trained trigger classifier is installed with trigger characteristics in a deployed deep learning system to collect and retain potential training data for the relevant use cases. The trigger classifier is a support vector machine, although other suitable classifiers may be used in some embodiments. For example, in some embodiments, the trigger classifier is a neural network and may include one or more hidden layers. In some embodiments, the deployed deep learning system utilizes the process shown in Figure 2.

[0059] In 301, training data is prepared. For example, positive and negative examples of a specific use case are prepared as training data. As an example, positive and negative examples of tunnel exits are collected and annotated. The curated and annotated dataset is used to create the training set. In some embodiments, annotation includes labeling the data and may be performed by a human curator. In some embodiments, the data format is compatible with the machine learning model used in the deployed deep learning application. In various embodiments, the training data includes validation data to test the accuracy of the trained model.

[0060] In 303, deep learning analysis is applied to training data. For example, an existing machine learning model is used to initiate the deep learning process. In some embodiments, the deep learning model is a neural network, such as a convolutional neural network (CNN) having multiple layers. In some embodiments, the CNN may include three or more hidden layers. An example of deep learning analysis includes a neural network for autonomous driving. In various embodiments, the deep learning analysis is applied to the training data prepared in 301 to generate the results of the hidden layers.

[0061] In 305, the trigger classifier is trained. In some embodiments, the trigger The trigger classifier is a support vector machine or a small neural network. In various embodiments, the input to the trigger classifier is the output of the first layer or one of the hidden layers of the machine learning model of the deep learning system. The specific layer used for input may be configurable. For example, the outputs up to the first layer, such as the second to last layer, the third to last layer, the fourth to last layer, etc., may be used as input to train the trigger classifier. In various embodiments, annotated results of training data, along with raw data (such as image data), are used to train the trigger classifier. By using positive and negative examples, the trigger classifier is trained to identify the likelihood that the input (e.g., input from sensor data) matches a specific use case, such as a tunnel exit. In some embodiments, the results of the trained trigger classifier are validated using a validation dataset created in 301. In some embodiments, the trigger classifier is trained using an offline neural network that matches the neural network deployed in the vehicle.

[0062] In some embodiments, the output of the neural network is a feature vector that identifies features of the input data (such as a RAW image). Features may include the number of vehicles, the number of signs, the number of lanes, etc., within the RAW data. The intermediate outputs of layers, for example, layers processed before the final layer, contain semantic information about the RAW input data. In some embodiments, the intermediate outputs of layers may be represented in vector form, and the vectors have more elements than the vector output of the final layer. For example, the final output of a neural network may be a 32-element vector, while the output of the second-to-last layer may be a 64-element vector. In various embodiments, the outputs of the first and intermediate layers of the neural network (e.g., a 64-element vector) contain more semantic information associated with the RAW input data than the output of the final layer of the neural network (e.g., a 32-element vector), and are therefore used to train a trigger classifier. In some embodiments, specific layers selected for training the trigger classifier may be selected dynamically. For example, a particular intermediate layer (e.g., an earlier layer) may be selected based on an improvement in the accuracy of that particular layer compared to another layer (e.g., a layer closer to the final layer). In some embodiments, specific layers are selected based on the efficiency with which they are used. For example, if the results obtained using the layers meet accuracy requirements, layers with smaller output vectors may be selected.

[0063] In some embodiments, inputs from different hidden layers are used to train two or more trigger classifiers, and the differently trained classifiers are compared to each other. A balance between accuracy and performance is used to determine which of the multiple classifiers should be used. For example, for some use cases, the output of an earlier hidden layer is needed, while for others, the output of a later hidden layer is sufficient. The optimal hidden layer output can be determined by comparing multiple trained trigger classifiers. In various embodiments, which layer of the neural network should receive the intermediate results is dynamically selected as part of the trigger classifier training process.

[0064] In some embodiments, the trained classifier may be specified by a vector and a bias factor. For example, the trained classifier may be a vector of weights offset by a bias factor to determine the classifier score. In some embodiments, the number of elements in the vector is the same as the number of elements in the output of the hidden layer used and the number of elements in the input for training the classifier. For example, if the output of the hidden layer used to train the classifier has 1024 elements, then the input data used to train the trigger classifier has 1024 elements, and the resulting trigger classifier can be represented as a weighted vector and bias of 1024. In this case, bias is optional and can be taken into account by the elements of the weighted vector.

[0065] In 307, trigger characteristics are determined for the classifier trained in 305. For example, a threshold may be determined to be compared to the classifier score determined by the trained trigger classifier. For example, a classifier score above the threshold indicates that the RAW input associated with the score is likely to be a positive example of the target use case. For example, a trigger classifier trained to identify tunnel exits determines a classifier score. Using a threshold of 0.5, a classifier score of 0.7 indicates that the data is likely to represent a tunnel exit. In some embodiments, a score of -1.0 is a negative example and a score of 1.0 is a positive example. Classifier scores lie between -1.0 and 1.0, indicating the likelihood that the RAW input is a positive or negative example of the target use case.

[0066] In some embodiments, trigger characteristics include necessary conditions such as trigger filters. A trigger filter is a filter used to restrict the retention of sensor data to the described conditions. For example, sensor data may be triggered for retention based on the location associated with the data. Other examples include the length of time since the last sensor data triggered and positive identification, the length of time since the drive started, time, location, road type, etc. In various embodiments, one or more trigger characteristics may be specified to restrict the conditions used by the trigger classifier to collect and retain sensor data.

[0067] In 309, the trigger classifier and trigger characteristics are deployed. For example, the trigger classifier and characteristics used to trigger the classifier to hold sensor data are installed together with the deep learning system. For example, the trigger classifier and characteristics may be packaged as a small binary that is transmitted wirelessly to the vehicle. In some embodiments, the packaged trigger classifier and characteristics are transmitted as a wireless update using wireless technology such as WiFi or a cellular network connection. Upon receipt in the vehicle, the trigger classifier and characteristics are installed as part of the autonomous driving system. In some embodiments, only the trigger classifier is installed. In some embodiments, the trigger classifier and a deep learning model for autonomous driving are installed together. In various embodiments, the machine learning model of the autonomous driving system uses trigger classification

[0068] Figure 4 is a flowchart illustrating an embodiment of the process for identifying potential training data using a trigger classifier. In some embodiments, the trigger classifier is run in conjunction with a deep learning system. For example, a deep learning system using a machine learning model that matches the one used to train the trigger classifier is used with the trigger classifier as part of an autonomous driving system. The trigger classifier analyzes the sensor data, which has been at least partially analyzed by the deep learning system, to identify whether the sensor data meets a specific use case that ensures the sensor data is retained. The sensor data is then sent to a computer server and may be used to create training data for a revised machine learning model that has improved performance in identifying the specific use case. Examples of use cases include identifying on-ramps, tunnel exits, road obstacles, road junctions, specific vehicle types, etc. In some embodiments, trigger parameters are used to set conditions under which the trigger classifier identifies the relevant results. In some embodiments, one or more trigger classifiers and parameters are used to identify one or more different use cases. In some embodiments, the process in Figure 4 is performed in Figures 205, 207, 209, 211, and / or 213. In this scenario, the trigger classifier used in the process shown in Figure 4 is trained using the process shown in Figure 3.

[0069] In 401, a deep learning analysis is initiated. For example, a deep learning analysis of an autonomous driving system is initiated with sensor data captured by sensors mounted on the vehicle. In some embodiments, the initiated deep learning analysis includes preprocessing the sensor data. In various embodiments, the deep learning analysis utilizes a trained machine learning model having multiple layers, including one or more hidden layers. In some embodiments, the outputs of the first layer and any hidden layers are considered intermediate outputs. In various embodiments, intermediate outputs are the outputs of the layers of the machine learning model other than the final output (e.g., the output of the final layer of the model).

[0070] In 403, inference is completed using one layer of deep learning analysis. For example, a neural network includes multiple layers, including hidden layers followed by a final layer. The output of each layer (e.g., an intermediate result) is fed as input to the next layer. In some embodiments, the outputs of the first layer and each hidden layer are considered intermediate results. In various embodiments, the decision result of the output of a single layer is a vector that can be used as input to the next layer. In some embodiments, the input to the first layer of the neural network is sensor data, such as image data. In some embodiments, the neural network is a convolutional neural network.

[0071] In 405, it is determined whether the output of the layer analysis performed in 403 is the result of the final layer of the neural network. If the output is not the result of the final layer, for example, if the output is an intermediate result, the process continues to 409. If the output is the result of the final layer of the neural network, the inference performed using the machine learning model is complete, and the process continues to 407. In some embodiments, the output in 405 provided to 407 is a feature vector.

[0072] In 407, the results of deep learning analysis performed on sensor data are provided to vehicle control. In some embodiments, the results are post-processed. For example, the results of one or more different neural networks for inputs from one or more different sensors may be combined. In some embodiments, vehicle control is performed using a vehicle control module to control the vehicle's behavior. For example, vehicle control may change the vehicle's speed, steering, acceleration, braking, etc., for autonomous driving. In some embodiments, vehicle control may enable or disable turn signals, brake lights, headlights, and / or manipulate other vehicle controls / signals, including network control such as sending network messages over a wireless network such as WiFi or a cellular network. In various embodiments, for example, when the autonomous driving function is disabled, vehicle control may not be enabled to actively control the vehicle. For example, even when the autonomous driving system is not actively controlling the vehicle, deep learning analysis in 401 and 403 is performed to provide results as input to a trigger classifier to identify potential training data.

[0073] In step 409, a determination is made as to whether the neural network layers and trigger conditions are appropriate for applying the trigger classifier. For example, the trigger characteristics indicate the conditions required for applying the trigger classifier. Examples of conditions include whether the time elapsed since the last capture exceeds a minimum time, whether the minimum driving time has elapsed, and whether the time is within a specific range. Examples of different times may include dawn, dusk, daytime, and nighttime. Further conditional requirements include location, weather, road conditions, road type, vehicle type, Triggers may be based on the disengagement of the autonomous driving function, steering angle (e.g., exceeding a steering angle threshold), changes in acceleration, application of brakes, or other appropriate features. Examples of different weather conditions may include snow, hail, sleet, rain, heavy rain, overcast, sunny, cloudy, fog, etc. Different conditions may be specified by the trigger characteristics. In some embodiments, different use cases may utilize different trigger characteristics and intermediate results from different layers of the neural network. For example, some use cases may be more efficient and may use intermediate results from later layers of the neural network to produce higher quality results. Other use cases may require earlier intermediate results to identify useful examples of sensor data that satisfy the use case. In some cases, the trigger characteristics used to specify the conditions under which the trigger classifier applies may be nested using multiple conditional checks and / or logical operators such as AND and OR operators.

[0074] In 411, the trigger classifier score is determined. For example, the trigger classifier score is determined by applying the trigger classifier to the intermediate results of a neural network. In some embodiments, the trigger classifier application utilizes weighted vectors and optional biases to determine the classifier score associated with the sensor data. In some embodiments, the trigger classifier is a support vector machine or a neural network. In some embodiments, the performance of the trigger classifier is improved by running the classifier on a customized artificial intelligence (AI) processor. For example, the AI ​​processor can perform dot product operations on two vectors in very few cycles and / or perform multiple dot products with limited wasted cycles. In some embodiments, the determined classifier score is a floating-point number representing the likelihood that the sensor data is a positive (or negative) example of the target use case. For example, a specific range, such as between -1 and +1, may be used to represent the likelihood that the sensor data is a negative or positive example of the target use case.

[0075] In step 413, a determination is made as to whether the classifier score exceeds a threshold and whether the necessary trigger conditions are met. For example, in some embodiments, the classifier score is compared to a threshold. If the classifier score exceeds the threshold, processing continues to step 415. If the classifier score does not exceed the threshold, processing continues to step 403. In some embodiments, additional trigger requirements may be applied after the classifier score has been determined. For example, the determined classifier score may be compared to a previously determined classifier score within a particular time frame. As another example, the determined classifier score may be compared to a previously determined score from the same location. As yet another example, sensor data may be required to satisfy both time and location conditions. For example, only sensor data with the highest score from the same location within the last 10 minutes may be retained as potential data. In various embodiments, the conditions may include trigger characteristics that act as a filter for either transmitting or not transmitting sensor data. In some embodiments, the conditions in step 413 are optional, and only the classifier score is compared to a threshold.

[0076] In some embodiments, separate thresholds exist for both positive and negative examples. For example, thresholds of +0.5 and -0.5 may be used to identify positive and negative sensor data as potential training data. A classifier score between +0.5 and 1.0 is used to identify positive examples, and a classifier score between -1.0 and -0.5 is used to identify negative examples. In some embodiments, only positive examples are retained for transmission.

[0077] In 415, the identified sensor data is transmitted. For example, the identified sensor data may be transmitted to a computer server (e.g., a training data generation system 120) where it can be used to create training data. In various embodiments, the training data includes a training dataset and a validation dataset. The data set includes metadata. In some embodiments, the transmitted sensor data includes metadata. Examples of metadata may include the time of the data, a timestamp, road conditions, weather conditions, location, vehicle type, whether the vehicle is left-hand drive or right-hand drive, classifier score, use case, neural network identifier, trigger classifier identifier, firmware version associated with the autonomous driving system, or other appropriate metadata associated with the sensor data and / or vehicle. In some embodiments, the time may indicate a period such as twilight, dawn, night, daylight, full moon, or solar eclipse. For example, a neural network and / or trigger classifier identifier may be transmitted to identify a specific trained machine learning model used to train the trigger classifier and to determine the classifier score. In some embodiments, the sensor data and / or metadata are first compressed before transmission. In some embodiments, the sensor data is transmitted in batches to transfer the sensor data more efficiently. For example, compression of multiple images of the sensor data is performed, and a set of sensor data is transmitted together.

[0078] Figure 5 is a flowchart illustrating an embodiment of the process for creating training data from data corresponding to use cases identified by a trigger classifier. For example, received sensor data is processed to create training data for training a machine learning model. In some embodiments, the sensor data corresponds to driving data captured via an autonomous driving system utilizing a trigger classifier. In some embodiments, the sensor data is received using the process in Figure 4 by a trigger classifier trained using the process in Figure 3. In some embodiments, the sensor data corresponds to sensor data captured based on specific use cases, such as the identification of road junctions, on-ramps, off-ramps, and tunnel entrances. In some embodiments, the received sensor data corresponds only to positive examples of a use case. In some embodiments, the sensor data includes both positive and negative examples. In various embodiments, the sensor data includes metadata such as classifier scores, location, time, or other appropriate metadata.

[0079] In 501, sensor data that satisfies the trigger condition is received. For example, sensor data corresponding to a specific target use case may be received and used as potential training data. In various embodiments, the sensor data is in a format that the machine learning model uses as input. For example, the sensor data may be raw image data or processed image data. In some embodiments, the data is data captured from an ultrasonic sensor, radar, LiDAR sensor, or other suitable technology. In various embodiments, the trigger condition is specified using a trigger classifier and trigger characteristics, as described with respect to Figures 2 to 4.

[0080] In 503, the sensor data is converted into training data. For example, the sensor data received in 501 includes data identified as potentially useful training data. In some embodiments, the received sensor data is compressed and then decompressed to improve efficiency for transmitting data from a vehicle at a distance. In some embodiments, the data is reviewed to determine whether the sensor data accurately represents a target use case. For example, a target use case is reviewed to identify examples of tunnel exits in order to determine whether the raw sensor data is indeed data from a tunnel exit. In some embodiments, a highly accurate machine learning model is used to verify whether the sensor data represents a target use case. In some embodiments, a human reviews and verifies whether the sensor data represents a target use case. In some embodiments, useful data for training is annotated. For example, data may be marked as either a positive or negative example. In some embodiments, the data is annotated for a target object. These can be annotated and labeled. For example, depending on the target use case, lane markers, signs, traffic lights, etc., may be annotated. In various embodiments, annotations may be used for training and / or validating trained machine learning models.

[0081] In 505, the training data transformed in 503 is prepared as training and validation datasets. In various embodiments, the sensor data transformed in 503 is prepared as a training dataset and a validation dataset for validating the machine learning model. In some embodiments, the training data from 503 is merged with an existing training dataset. For example, an existing training dataset applicable to most use cases is merged with newly transformed training data to improve coverage for a specific use case. The newly transformed training data is useful in improving the accuracy of the model when identifying a specific use case. In some embodiments, some portions of the existing training data are discarded and / or replaced with new training data.

[0082] In 507, the machine learning model is trained. For example, the machine learning model is trained using the data prepared in 505. In some embodiments, the model is a neural network, such as a convolutional neural network (CNN). In various embodiments, the model includes multiple hidden layers. In some embodiments, the neural network may include multiple layers, including multiple convolutional and pooling layers. In some embodiments, the trained model is validated using a validation dataset created from received sensor data.

[0083] In 509, the trained machine learning model is deployed. For example, the trained machine learning model is installed in the vehicle as an update to the autonomous learning system. For example, a wireless update may be used to install the new model. In some embodiments, the update is a firmware update transmitted using a wireless network such as WiFi or a cellular network. In some embodiments, the new model is used to train a new trigger classifier. In various embodiments, the existing trigger classifier based on the old model expires, and a new trigger classifier based on the newly trained model is deployed. In some embodiments, the new machine learning model is installed when the vehicle is inspected.

[0084] Figure 6 is a flowchart illustrating an embodiment of the process for triggering classifier selection on a vehicle. This process can be performed arbitrarily and selectively by a vehicle, such as a vehicle with one or more processors. For example, a vehicle may have a large number of classifiers. In this example, the vehicle may run a subset of classifiers to conserve processing resources. For example, the vehicle may determine classifier scores for only a subset. As illustrated in Figure 1B, the vehicle may periodically refresh the subset (e.g., select a new classifier after a threshold time). In some embodiments, the vehicle may receive information from an external system (e.g., system 120) that identifies that the vehicle will run one or more specific classifiers.

[0085] In block 601, the vehicle runs the classifier. As described above, the vehicle can acquire sensor data and determine a classifier score based on the sensor data.

[0086] In block 603, the vehicle receives a trigger to select a new classifier. The vehicle receives at least a Global Navigation Satellite System (GNSS) receiver. The vehicle can monitor its position. In some embodiments, the vehicle may be able to access map information. The map information may identify certain features or use cases for which it may be advantageous to acquire training data. For example, the map information may identify tunnel exits. For another example, the map information may identify partially blocked or hidden side roads. For yet another example, the map information may identify the location of a particular style or form of bike lane (e.g., raised or offset bike lanes). The vehicle may determine when it is approaching a particular feature or use case (e.g., at a threshold distance). The vehicle may then acquire information to identify a new classifier associated with the particular feature or use case. This new classifier may then be run by the vehicle to determine a classifier score for the received sensor data.

[0087] Furthermore, the vehicle may transmit location information to an external system. The external system may then transmit information to the vehicle regarding one or more new classifiers that the vehicle should run. For example, the external system may transmit a unique identifier associated with each classifier. As illustrated in Figure 1B, the external system may have received information from the same classifier running on at least a certain number of vehicles (e.g., 1, 3, 10, 20). These vehicles may have been located within a threshold distance (e.g., radius) of each other so that the external system could determine the presence of features or use cases close to their locations. Thus, the external system may instruct the vehicle to run the same classifier if it is within a threshold distance of location. In this way, the external system can obtain sensor data associated with this classifier.

[0088] In block 605, the vehicle runs a new classifier. As described herein, the new classifier may obtain information from the intermediate layers of a machine learning model (e.g., a convolutional neural network). Then, in block 607, the vehicle determines a classifier score. Then, in block 609, the vehicle transmits sensor data (e.g., images) based on a classifier score that exceeds a threshold. As described above, the sensor data may be transmitted along with metadata.

[0089] Figure 7 is a block diagram illustrating an embodiment of a deep learning system for identifying potential training data. For example, the block diagram includes different components of a deep learning system connected to a trigger classifier for autonomous driving, where a subset of sensor data captured for autonomous driving is identified as potential training data. In some embodiments, the deep learning system may passively analyze the sensor data, and the intermediate outputs of the layers of the deep learning system are used as inputs to the trigger classifier. In some embodiments, the deep learning system actively analyzes and controls the vehicle's behavior, while also identifying and retaining potentially useful sensor data for creating additional training data. In some embodiments, the autonomous driving system is used for the operation of the vehicle in autonomous driving or driver assistance. In various embodiments, the processes in Figures 2–6 utilize the deep learning system and / or components of the system, such as those described in Figure 7.

[0090] In the example shown, the deep learning system 700 is a deep learning network including a sensor 701, an image preprocessor 703, a deep learning network 705, an artificial intelligence (AI) processor 707, a vehicle control module 709, a network interface 711, and a trigger classifier module 713. In various embodiments, different components are connected in a communicative manner. For example, sensor data from sensor 701 is fed to the image preprocessor 703. The processed sensor data from the image preprocessor 703 is fed to the deep learning network 705 running on the AI ​​processor 707. The output of the learning network 705 is supplied to the vehicle control module 709. Intermediate results of the deep learning network 705, running on the AI ​​processor 707, are supplied to the trigger classifier module 713. Sensor data that triggers a hold for transmission by the trigger classifier module 713 is transmitted via the network interface 711. In some embodiments, the trigger classifier module 713 runs on the AI ​​processor 707. In various embodiments, the network interface 711 is used to communicate with a remote server, make phone calls, send and / or receive text messages, transmit sensor data identified by the trigger classifier module 713, etc., based on the autonomous operation of the vehicle and / or the results of the trigger classifier module 713. In some embodiments, the deep learning system 700 may include more or fewer components as needed. For example, in some embodiments, an image preprocessor 703 is an optional component. As another example, in some embodiments, a post-processing component (not shown) is used to perform post-processing on the output of the deep learning network 705 before the output is provided to the vehicle control module 709.

[0091] In some embodiments, the sensor 701 includes one or more sensors. In various embodiments, the sensor 701 may be mounted on the vehicle at different locations and / or in one or more different orientations. For example, the sensor 701 may be mounted on the front, side, rear, and / or roof of the vehicle in orientations such as forward, backward, and sideways. In some embodiments, the sensor 701 may be an image sensor, such as a high dynamic range camera. In some embodiments, the sensor 701 includes non-visual sensors. In some embodiments, the sensor 701 includes, among other things, radar, LiDAR, and / or ultrasonic sensors. In some embodiments, the sensor 701 is not mounted on a vehicle having a vehicle control module 709. For example, the sensor 701 may be mounted on an adjacent vehicle and / or on a road or in the environment and may be included as part of a deep learning system for capturing sensor data.

[0092] In some embodiments, the image preprocessor 703 is used to preprocess sensor data from the sensor 701. For example, the image preprocessor 703 may be used to preprocess the sensor data, divide the sensor data into one or more components, and / or postprocess one or more components. In some embodiments, the image preprocessor 703 is a graphics processing unit (GPU), a central processing unit (CPU), an image signal processor, or a dedicated image processor. In various embodiments, the image preprocessor 703 is a tone mapper processor for processing high dynamic range data. In some embodiments, the image preprocessor 703 is implemented as part of an artificial intelligence (AI) processor 707. For example, the image preprocessor 703 may be a component of the AI ​​processor 707.

[0093] In some embodiments, the deep learning network 705 is a deep learning network for performing autonomous vehicle control. For example, the deep learning network 705 may be an artificial neural network, such as a convolutional neural network (CNN), trained using sensor data, and its output is provided to the vehicle control module 709. In some embodiments, a replica of the neural network of the deep learning network 705 is used to create a trigger classifier for the trigger classifier module 713.

[0094] In some embodiments, the artificial intelligence (AI) processor 707 is a hardware processor for running a deep learning network 705 and / or a trigger classifier module 713. In some embodiments, the AI ​​processor 707 is a hardware processor for running sensor data It is a specialized AI processor for performing inference using a convolutional neural network (CNN) on data. In some embodiments, the AI ​​processor 707 is optimized for the bit depth of the sensor data. In some embodiments, the AI ​​processor 707 is optimized for deep learning operations such as neural network operations, including convolution, dot product, vector operations, and / or matrix operations, among other things. In some embodiments, the AI ​​processor 707 is implemented using a graphics processing unit (GPU). In various embodiments, the AI ​​processor 707 is coupled to memory configured to provide the AI ​​processor with instructions that, when executed, cause the AI ​​processor to perform deep learning analysis on received input sensor data and determine machine learning results used to operate the vehicle at least partially autonomously. In some embodiments, the AI ​​processor 707 is configured to output intermediate results from one or more layers of the deep learning network 705 to a trigger classifier module 713 to determine a classifier score.

[0095] In some embodiments, the vehicle control module 709 is used to process the output of the artificial intelligence (AI) processor 707 and convert the output into vehicle control operations. In some embodiments, the vehicle control module 709 is used to control the vehicle for autonomous driving. In some embodiments, the vehicle control module 709 can adjust the vehicle's speed and / or steering. For example, the vehicle control module 709 may be used to control the vehicle by braking, steering, lane changes, acceleration, and merging into another lane. In some embodiments, the vehicle control module 709 is used to control vehicle lighting such as brake lights, turn signals, and headlights. In some embodiments, the vehicle control module 709 is used to control vehicle audio conditions such as the vehicle's sound system, which plays audio alerts, enables the microphone, enables the horn, etc. In some embodiments, the vehicle control module 709 is used to control notification systems, including warning systems that notify the driver and / or passengers of driving events such as a potential collision or approaching an intended destination. In some embodiments, the vehicle control module 709 is used to adjust sensors such as the vehicle's sensors 701. For example, the vehicle control module 709 may be used to modify the parameters of one or more sensors, such as changing orientation, changing output resolution and / or format type, increasing or decreasing capture rate, adjusting the captured dynamic range, adjusting camera focus, and enabling and / or disabling sensors. In some embodiments, the vehicle control module 709 may be used to modify the parameters of the image preprocessor 703, such as changing the filter frequency range, adjusting feature and / or edge detection parameters, and adjusting channels and bit depth. In various embodiments, the vehicle control module 709 is used to implement control of the vehicle in autonomous driving and / or driver assistance.

[0096] In some embodiments, the network interface 711 is a communication interface for transmitting and / or receiving data, including voice data. In various embodiments, the network interface 711 includes a cellular or wireless interface for interfacing with a remote server, such as for connecting and initiating voice calls, transmitting and / or receiving text messages, transmitting sensor data, and receiving updates to an autonomous driving system, including trigger classifiers and characteristics. For example, the network interface 711 may be used to receive updates to instructions and / or operating parameters for a sensor 701, an image preprocessor 703, a deep learning network 705, an AI processor 707, a vehicle control module 709, and / or a trigger classifier module 713. For example, the machine learning model of the deep learning network 705 may be updated using the network interface 711. As another example... The network interface 711 may be used to update the firmware of the sensor 701 and / or the operating parameters of the image preprocessor 703, such as image processing parameters.

[0097] In some embodiments, the network interface 711 is used to transmit sensor data identified by the trigger classifier module 713. For example, sensor data corresponding to a specific use case identified by the trigger classifier and satisfying the conditions of the associated trigger characteristics is transmitted via the network interface 711 to a computer server, such as a remote computer server. In some embodiments, the trigger classifier and trigger characteristics are updated via the network interface 711. The updated trigger classifier and trigger characteristics are installed in the trigger classifier module 713 and used to identify and retain sensor data corresponding to a specific use case.

[0098] In some embodiments, the network interface 711 is used to make emergency contact with emergency services in the event of an accident or near-accident. For example, in the event of a collision, the network interface 711 may be used to contact emergency services for assistance and to notify emergency services of the vehicle's location and collision details. In various embodiments, the network interface 711 is used to perform autonomous driving functions, such as accessing calendar information to look up and / or update the location of a destination and / or the estimated time of arrival.

[0099] In some embodiments, the trigger classifier module 713 is used to identify and retain sensor data corresponding to a specific use case. For example, the trigger classifier module 713 determines a classifier score for data captured by one or more sensors of sensor 701. The classifier score may be compared to a threshold, retained, and transmitted to a remote computer server via the network interface 711. In some embodiments, the trigger classifier module 713 utilizes trigger characteristics to determine whether appropriate conditions are met in order to determine the classifier score and / or retain sensor data that satisfies the classifier score threshold. In some embodiments, the trigger classifier module is a support vector machine that receives the intermediate output of a deep learning network 705 as input representing the sensor data of sensor 701. In some embodiments, the trigger classifier module 713 is configured to receive the intermediate results of one or more layers of the deep learning network 705. The output of a particular layer may depend on the trigger classifier and / or trigger characteristics. For example, some use cases may use earlier intermediate results, while others may use later intermediate results. In some embodiments, the AI ​​processor 707 may be used to perform processing by the trigger classifier module 713. In various embodiments, the sensor data identified by the trigger classifier module 713 is used to create a new training dataset for identifying a specific use case.

[0100] Various aspects, embodiments, configurations, or features of the described embodiments can be used separately or in any combination. Various aspects of the described embodiments can be implemented by software, hardware, or a combination of hardware and software. The described embodiments can also be embodied as computer-readable code on a computer-readable medium for controlling manufacturing operations, or as computer-readable code on a computer-readable medium for controlling a manufacturing line. A computer-readable medium is any data storage device capable of storing data that can then be read by a computer system. Examples of computer-readable mediums include read-only memory, random-access memory, CD-ROMs, HDDs, DVDs, magnetic tapes, and optical data storage devices. Computer-readable mediums can also be computer-readable. The code reading process can also be distributed across networked computer systems so that it is stored and executed in a distributed manner.

[0101] The foregoing description uses a specific nomenclature to provide a complete understanding of the embodiments described, for illustrative purposes. However, it will be apparent to those skilled in the art that specific details are not required to practice the embodiments described. Thus, the foregoing description of specific embodiments is presented for illustrative and explanatory purposes only. They are not intended to be exhaustive or to limit the embodiments described to the disclosed as-is form. Given the above teachings, it will be apparent to those skilled in the art that many modifications and variations are possible.

[0102] It will also be understood that each of the processes, methods, and algorithms described herein and / or shown in the figures may be embodied in code modules executed by one or more physical computing systems, hardware computer processors, application-specific circuitry, and / or electronic hardware configured to execute specific and particular computer instructions, and thereby fully or partially automated. For example, a computing system may include a general-purpose computer (e.g., a server) or a dedicated computer, dedicated circuitry, etc., programmed with specific computer instructions. The code modules may be compiled and linked into an executable program and installed in a dynamic-link library, or they may be written in an interpreted programming language. In some embodiments, specific operations and methods may be executed by circuitry specific to a given function.

[0103] Furthermore, certain functional embodiments of the present disclosure are so mathematically, computationally, or technically complex that, for example, application-specific hardware or one or more physical computing devices (utilizing appropriate dedicated executable instructions) may be required to perform the functionality, for example, due to the amount or complexity of the computations involved, or to provide results substantially in real time. For example, video may contain many frames, each having millions of pixels, and uniquely programmed computer hardware is required to process the video data to provide a desired image processing task or application in a commercially reasonable time.

[0104] Code modules or any type of data may be stored on any type of non-temporary computer-readable medium, including physical computer memory such as hard drives, solid-state memory, random-access memory (RAM), read-only memory (ROM), optical discs, volatile or non-volatile memory, combinations thereof, and / or similar. In some embodiments, the non-temporary computer-readable medium may be part of one or more of local processing and data modules, remote processing modules, and remote data repositories. Methods and modules (or data) may also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) over a variety of computer-readable transmission media, including wireless-based and wired / cable-based media, and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple individual digital packets or frames). The results of the disclosed process or process steps may be stored permanently or otherwise in any type of non-temporary tangible computer memory or communicated via computer-readable transmission media.

[0105] The processes, blocks, states, steps, or functionalities in the flowcharts described herein and / or shown in the accompanying diagrams are code modules containing one or more executable instructions for performing a particular function (e.g., logical or arithmetic) or step in a process. , segments, or parts of code should be understood as potentially representing a segment or part of code. Various processes, blocks, states, steps, or functionalities may be combined, rearranged, added, deleted, modified, or otherwise altered from the exemplary examples provided herein. In some embodiments, additional or different computing systems or code modules may perform some or all of the functionalities described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps, or states associated therewith may be performed in any other appropriate sequence, e.g., serial, parallel, or any other manner. Tasks or events may be added to or deleted from the disclosed exemplary embodiments. Furthermore, the separation of various system components in the embodiments described herein is for illustrative purposes only and should not be understood as requiring such separation in all embodiments. The program components, methods, and systems described may generally be integrated together in a single computer product or packaged in multiple computer products.

[0106] In the aforementioned specification, one or more innovations have been described with reference to their specific embodiments. However, it will be clear that various modifications and changes can be made to them without deviating from the broader spirit and scope of the innovations. Accordingly, this specification and the drawings should be considered illustrative rather than restrictive.

[0107] In fact, it will be understood that each of the systems and methods disclosed herein has several innovative aspects, and that not just one aspect of them is required or embodied in the desirable attributes disclosed herein. The various features and processes described above may be used independently of each other or combined in various ways. All possible combinations and partial combinations are intended to be included within the scope of this disclosure.

[0108] Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately or in any suitable partial combination in multiple embodiments. Furthermore, features may be described as playing a role in a particular combination, and even if initially claimed so, one or more features from the claimed combination may be removed from the combination, and the claimed combination may be subject to a partial combination or a variation of a partial combination. Not a single feature or group of features is required or essential in every embodiment.

[0109] In particular, conditional language used herein, such as “can,” “can,” “could,” “might,” “may,” “may,” “may,” “may,” “may,” “e.g.,” and similar expressions, is generally intended to convey that a particular embodiment includes certain features, elements, and / or steps, while other embodiments do not, unless otherwise specified or understood in the context in which they are used. Thus, such conditional language is not generally intended to imply that features, elements, and / or steps are required in any way in one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps are included in or should be performed in any particular embodiment, with or without input or prompting by the author. The terms "comprising," "including," and "having," and similar terms are synonymous and are used comprehensively in an open-ended style, encompassing additional elements, characteristics, or effects. This does not exclude actions, etc. Furthermore, the term "or" is used in its inclusive (not exclusive) sense; for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. Moreover, the articles "a," "an," and "the" used in this application and the attached claims should be interpreted as meaning "one or more" or "at least one" unless otherwise specified. Similarly, while actions may be depicted in a particular order in the drawings, it should be recognized that such actions do not need to be performed in a specific order or sequential order shown, or that not all illustrated actions need to be performed, in order to achieve the desired result. Furthermore, the drawings may schematically depict another exemplary process in the form of a flowchart. However, other actions not depicted may be incorporated into the schematicly shown exemplary methods and processes. For example, one or more additional actions may be performed before, after, simultaneously with, or between any of the shown actions. Furthermore, in other embodiments, actions may be rearranged or reordered. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged in multiple software products. Further embodiments are within the scope of the following claims. In some cases, the actions described in the claims may be performed in a different order and still achieve the desired results.

[0110] Accordingly, the claims are not intended to be limited to the embodiments shown herein, but should be granted the broadest scope consistent with the disclosure, the principles, and the novel features disclosed herein.

Claims

1. It is a method, The system determines a trigger classifier to be executed by one or more processors of the vehicle, wherein the trigger classifier is configured to receive intermediate results from a machine learning model and generate a classifier score indicating the likelihood that one or more objects are represented in the sensor data processed by the machine learning model. The system provides the trigger classifier and the conditions for the one or more processors of the vehicle to identify the sensor data. A method comprising the step of causing the system to cause one or more processors of the vehicle to hold the sensor data identified based on the trigger classifier and the conditions.

2. The system further includes the step of providing a probability threshold for one or more objects to be represented in the sensor data processed by the machine learning model, in order to cause one or more processors of the vehicle to identify the sensor data, The step of causing one or more processors of the vehicle to hold the sensor data that satisfies the conditions is: The method according to claim 1, comprising causing one or more processors of the vehicle to hold the probability threshold and the sensor data that satisfies the conditions.

3. The step of determining the trigger classifier executed by one or more processors of the vehicle is: The method according to claim 1, comprising determining that the trigger classifier is configured to output a possibility for at least one object related to a use case.

4. The step of determining the trigger classifier is: The system determines the intermediate layer of the machine learning model, which is configured to generate the intermediate result from among multiple layers of the machine learning model. The method according to claim 1, comprising determining that the trigger classifier, which is performed by one or more processors of the vehicle, is configured to receive the intermediate results from the intermediate layers of the machine learning model.

5. The step of providing the conditions in order to cause one or more processors of the vehicle to identify the sensor data that satisfies the conditions is: The method according to claim 1, comprising providing the conditions for specifying one or more of the following: the position of the vehicle when the sensor data is generated, the amount of time the vehicle has been traveling when the sensor data is generated, the type of vehicle, and whether the autonomous driving function is enabled when the sensor data is generated.

6. The method according to claim 1, further comprising the step of causing one or more processors of the vehicle to refrain from retaining the sensor data that does not satisfy the conditions, by the system.

7. The method according to claim 1, further comprising the step of obtaining sensor data satisfying the conditions from one or more processors of the vehicle, based on the system providing the trigger classifier and the conditions to one or more processors of the vehicle.

8. It is a system, It comprises one or more processors, and the one or more processors are Determine the trigger classifier to be run by one or more processors in the vehicle. The trigger classifier is configured to receive intermediate results from a machine learning model and generate a classifier score indicating the likelihood that one or more objects are represented in the sensor data processed by the machine learning model. The trigger classifier and the conditions for causing one or more processors of the vehicle to identify the sensor data are provided. A system configured to cause one or more processors of the vehicle to hold the trigger classifier and the sensor data identified based on the conditions.

9. The aforementioned one or more processors are In order to cause one or more processors of the vehicle to identify the sensor data, the one or more objects are configured to provide probability thresholds represented in the sensor data processed by the machine learning model, The one or more processors of the vehicle are configured to hold the sensor data that satisfies the conditions, The system according to claim 8, wherein one or more processors of the vehicle are configured to hold the probability threshold and the sensor data that satisfies the conditions.

10. The one or more processors configured to determine the trigger classifier executed by the one or more processors of the vehicle, The system according to claim 8, wherein the trigger classifier is configured to determine that it is configured to output a possibility for at least one object related to a use case.

11. The one or more processors configured to determine the trigger classifier, The intermediate layer of the machine learning model is determined from among the multiple layers of the machine learning model, which is configured to generate the intermediate result. The system according to claim 8, wherein the trigger classifier, which is executed by one or more processors of the vehicle, is configured to determine that it is configured to receive the intermediate results from the intermediate layers of the machine learning model.

12. The one or more processors configured to provide the conditions for causing the one or more processors of the vehicle to identify the sensor data that satisfies the conditions, The system according to claim 8, configured to provide one or more of the following conditions: the position of the vehicle when the sensor data is generated, the amount of time the vehicle has been traveling when the sensor data is generated, the type of vehicle, and whether the autonomous driving function is enabled when the sensor data is generated.

13. The system according to claim 8, wherein the one or more processors are configured to cause the one or more processors of the vehicle to refrain from retaining the sensor data that does not satisfy the conditions.

14. The system according to claim 8, wherein the one or more processors are configured to acquire the sensor data satisfying the conditions from the one or more processors of the vehicle, based on providing the trigger classifier and the conditions to the one or more processors of the vehicle.

15. A non-temporary computer-readable storage medium having computer instructions stored in the non-temporary computer-readable storage medium, wherein when the computer instructions are executed by one or more processors, the one or more processors Determine the trigger classifier to be executed by one or more processors in the vehicle. The trigger classifier is configured to receive intermediate results from a machine learning model and generate a classifier score indicating the likelihood that one or more objects are represented in the sensor data processed by the machine learning model. The trigger classifier and the conditions for the one or more processors of the vehicle to identify the sensor data are provided. A non-temporary computer-readable storage medium that causes one or more processors of the vehicle to hold the sensor data identified based on the trigger classifier and the conditions.

16. The computer instruction is transmitted to one or more processors. To cause one or more processors of the vehicle to identify the sensor data, the one or more objects are provided with probability thresholds that are represented in the sensor data processed by the machine learning model. The instruction to cause one or more processors to hold the sensor data satisfying the conditions in the one or more processors of the vehicle is to cause one or more processors to hold the sensor data satisfying the conditions in the one or more processors of the vehicle. The non-temporary computer-readable storage medium according to claim 15, which causes one or more processors of the vehicle to hold the sensor data that satisfies the probability threshold and the conditions.

17. The computer instruction causing one or more processors to determine the trigger classifier executed by the one or more processors of the vehicle is, The non-temporary computer-readable storage medium according to claim 15, wherein the trigger classifier is configured to output possibilities for at least one object related to a use case.

18. The computer instruction that causes one or more processors to determine the trigger classifier is, The intermediate layer of the machine learning model is determined from among multiple layers of the machine learning model, configured to generate the intermediate result. A non-temporary computer-readable storage medium according to claim 15, wherein the trigger classifier, which is executed by one or more processors of the vehicle, is configured to receive the intermediate results from the intermediate layer of the machine learning model.

19. The computer instruction that causes one or more processors to provide the conditions in order to cause the one or more processors of the vehicle to identify the sensor data that satisfies the conditions is, A non-temporary computer-readable storage medium according to claim 15, which provides the conditions for specifying one or more of the following: the position of the vehicle when the sensor data was generated, the amount of time the vehicle was traveling when the sensor data was generated, the type of vehicle, and whether the autonomous driving function was enabled when the sensor data was generated.

20. The computer instruction is transmitted to one or more processors. The non-temporary computer-readable storage medium according to claim 15, which causes one or more processors of the vehicle to refrain from retaining the sensor data that does not satisfy the conditions.

Images