Systems and methods for intelligent selection of data for building machine learning models

Abstract

Systems and methods for selecting data for training a machine learning model using active learning are disclosed. These methods include: receiving a plurality of unlabeled sensor data logs corresponding to an environment surrounding an autonomous vehicle; identifying one or more trends associated with a training data set including a plurality of labeled data logs; determining a function for assigning an importance score to each of the plurality of unlabeled sensor data logs; using the one or more trends; using a function for assigning an importance score to each of the plurality of unlabeled sensor data logs; selecting a subset of the plurality of sensor data logs having an importance score greater than a threshold; and using the subset of the plurality of sensor data logs to further train a machine learning model trained using the training data set to generate an updated model.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent document claims priority to U.S. patent application No. 17 / 101,633, filed on November 23, 2020, which is incorporated herein by reference in its entirety. Background Art

[0003] For the operation of an autonomous vehicle, it is important to be able to predict or forecast, including correctly detecting, the trajectory of objects in the environment of the autonomous vehicle. Accurate prediction of object behavior enables the autonomous vehicle to implement anticipatory planning and control rather than reacting to its environment. This results in more natural driving behavior of the autonomous vehicle and improved comfort and safety for its passengers.

[0004] Autonomous vehicles collect a large amount of data from many sensors in order to perform object detection and object behavior prediction. It is important to obtain real-world data about vehicle events that can be used to develop, evaluate and / or test systems for vehicle control. However, real-world vehicle events of interest are rare and difficult to separate from continuously collected real-world data because the total real-world data set from driving sessions usually contains redundant and / or a large amount of irrelevant or less relevant information related to vehicle operation.

[0005] Typically, passive learning methods accept randomly selected training data. However, such random selection of training data requires expensive labeling, which may not improve the training of the machine learning model (for example, when the randomly selected training data does not include useful information). In active learning, "useful" data is selected for subsequent training of the machine learning model, rather than passively accepting randomly selected data. Compared to passive learning, active learning can significantly reduce the amount of data required to train, test, and / or validate machine learning models, while achieving similar or better object detection (including classification as well as bounding box regression) accuracy as passive learning. However, current active learning methods do not take into account information about the environment of the autonomous vehicle in which the autonomous vehicle operates and / or will operate during the selection of data samples.

[0006] This document describes methods and systems directed to solving the above-referenced problems and / or other problems. Summary of the invention

[0007] In various embodiments, systems and methods for training a machine learning model using active learning are disclosed. The system may include a processor and a non-transitory computer-readable medium including programming instructions that can be executed by the processor to perform the method of the present disclosure. The computer-readable medium and programming instructions may optionally be included in a computer program product.

[0008] The method includes receiving a plurality of unlabeled sensor data logs corresponding to the surrounding environment of the autonomous vehicle from a plurality of sensors. The method then includes identifying one or more trends associated with a training data set (and / or other data sets such as a validation data set / test data set) including a plurality of labeled data logs, and using the one or more trends to determine a function for assigning an importance score to each of the plurality of unlabeled sensor data logs. The training data set is used to train a machine learning model. The method then includes using a function for assigning an importance score to each of the plurality of unlabeled sensor data logs, selecting a subset of the plurality of sensor data logs having an importance score greater than a threshold, and using a subset of the plurality of sensor data logs to further train the machine learning model trained using the training data set to generate an updated model.

[0009] In some embodiments, the method may also include identifying bounding boxes and associated tags within each of the plurality of sensor data logs (e.g., a machine learning model trained using a training data set), and determining a confidence score for the machine learning model trained using the training data set for identifying the associated tags. The system may then use the confidence score to determine the function. The machine learning model may be an object detection model. Optionally, the confidence score may be determined based on the following example features associated with the machine learning model trained using the training data set: actual accuracy, expected accuracy, false positive rate, false negative rate; convergence, output, statistical fit, identifying a problem to be solved, and / or training status.

[0010] Alternatively and / or in addition, the method may further include receiving metadata associated with each of a plurality of labeled data logs in the training data set, and using the metadata to identify one or more trends. The metadata may include, for example, a capture time of the data log, a capture location of the data log, information about environmental conditions at the capture location or time of the data log, or information about one or more events at the capture location or time of the data log.

[0011] Optionally, one or more trends can be used to identify one or more features that are underrepresented in a plurality of labeled data logs of a training data set. Examples of such features can include, for example, tag category counts, tag counts by time; tag counts by location; event category counts; event category counts by time, event category counts by location, instance counts by environmental conditions, or machine learning model uncertainty.

[0012] In certain other embodiments, the method may further include receiving metadata associated with the sensor data log. Examples of such metadata may include, but are not limited to, the capture time of the data log, the capture location of the data log, information related to the environmental conditions at the capture location or time of the data log, or information related to one or more events at the capture location or time of the data log. Optionally, the method may further include using the function to assign the importance score to each of the multiple unlabeled sensor data logs by assigning the score to one or more of the multiple characteristics of the data log. The multiple characteristics may include, but are not limited to, tag category counts, tag counts by time; tag counts by location; event category counts; event category counts by time, event category counts by location, instance counts by environmental conditions, or machine learning model uncertainty.

[0013] Optionally, the system may discard one or more sensor data logs among the plurality of sensor data logs having an importance score less than a threshold. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A flow diagram of an example method for intelligently selecting data to train a machine learning model according to an embodiment is illustrated.

[0015] Figure 2A illustrates an example data log including unlabeled images, and Figure 2B The diagram includes detected bounding boxes and corresponding categories. Figure 2A of the image.

[0016] Figure 3A-3C The distribution / trend of examples within the training dataset is illustrated.

[0017] Figure 4A illustrates an example data log including unlabeled images, and Figure 4B The diagram includes detected bounding boxes and corresponding categories. Figure 4A of the image.

[0018] Figure 5 is a block diagram illustrating an exemplary autonomous vehicle system.

[0019] Figure 6 An example vehicle controller system is illustrated.

[0020] Figure 7 is a block diagram illustrating various elements of possible electronic systems, subsystems, controllers, and / or other components of an AV and / or external electronic devices. DETAILED DESCRIPTION

[0021] As used herein, the singular forms "a", "an", and "the" include plural references unless the context clearly dictates otherwise. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. As used herein, the term "including" means "including but not limited to". Definitions of additional terms relevant to this document are included at the end of the Detailed Description.

[0022] Various algorithms for control and navigation of autonomous vehicles, such as object detection algorithms for detecting objects in images, use machine learning models that are built using labeled data (e.g., training data, test data, validation data, etc.). Although training such models requires a large amount of training data (i.e., labeled images), it is not feasible to use all or most of the data collected by autonomous vehicles due to processing, cost, and timing constraints. In addition, one of the biggest training obstacles for autonomous vehicles is recognizing edge cases, such as unexpected roadblocks, foreign animals, unusual road signs, and trucks towing trailers of four-wheeled vehicles (which object detection models can distinguish as two or even three separate vehicles). While training a machine learning model to recognize common objects and scenes is relatively straightforward, teaching it to recognize unusual conditions and objects on the road can be complex and requires sufficient labeled data to consider these edge cases.

[0023] However, it is often expensive to select high-quality labeled training examples (and / or validation / test examples) from the virtually unlimited sensor data collected by the vehicle that is available to the machine learning algorithm. In active learning, "useful" data (e.g., mispredicted labels, uncertain prediction labels, edge case labels, diverse data, etc.) are selected for subsequent construction (i.e., training, validation, and testing) of the machine learning model, rather than passively accepting randomly selected data. However, current active learning methods do not consider information about the environment in which the autonomous vehicle is operating and / or will operate (e.g., location, time of day, etc.), information about already existing training data and labeled (and / or validation / test data), information about various events (e.g., takeover, lane change, truck towing vehicle, etc.) during the selection of data samples.

[0024] Various aspects of the present disclosure improve the construction of machine learning models - including training, testing and / or validation - by using a data (or information) driven data selection model to intelligently select useful and / or important data samples for annotation and / or for training, testing and / or validation of the machine learning model.

[0025] Reference now Figure 1 , a flowchart illustrates an example method for selecting relevant data for use in building a machine learning model. The method can be performed in real time or near real time (e.g., when recording image data), after a delay (e.g., using previously stored sensor data), or at any suitable time. The method can be performed a predetermined number of times in a driving session, repeatedly at a predetermined frequency in a driving session, or at any suitable time. Multiple instances of the method can be performed simultaneously for multiple simultaneous driving sessions (e.g., for different vehicles, via different vehicle systems). However, any suitable number of method instances can be performed at any suitable time. It should be noted that although the method describes the selection of training data, the present disclosure is not so limited, and similar principles can be used for the selection of test data, validation data, etc.

[0026] At 102, the system may receive raw data logs collected (e.g., recorded, obtained, etc.) by sensors mounted on a vehicle during operation (e.g., driving). The received data logs are not labeled. Examples of such raw data logs may include, but are not limited to, point clouds, 2D images, 3D images, audio logs, etc. The data logs may include, but are not limited to, point clouds, 2D images, 3D images, audio logs, etc. The data logs may include a single data point (e.g., a single image, a single point cloud frame), a data sequence (e.g., a sequence of images, a video, a burst of point cloud frames, etc.), and / or any other suitable data sample. A single "log" of sensor data refers to data collected by a sensor over a continuous time period "t". The data log may include sensor measurement data in a frame or scans over time, such as periodically, intermittently, in response to a sensing event, etc. The data log data may be recorded constantly (e.g., continuously), periodically, randomly, upon occurrence of a triggering event (e.g., a driving event, a data value or signal exceeding a threshold, etc.), and / or using any other suitable time characteristics.

[0027] In various embodiments, each received data log may include metadata, such as, but not limited to, a timestamp indicating the time of collection of the data log, location information and / or map information corresponding to the location from which the data log was collected, information related to environmental conditions at the time and location when the data log was collected (e.g., rainy, cloudy, sunny, etc.), information related to one or more events at the time and location when the data log was collected (e.g., typically high traffic times, near critical events, collision events, takeover events, etc.), etc. Optionally, the system can receive metadata about the data log from other data stores (e.g., weather databases, traffic databases, satellite image databases, etc.), associating the data log with appropriate metadata. For example, the system can receive location data corresponding to the data log based on the route history of the vehicle used to collect the data log and the timestamp of the data log. The system can then query various databases to receive weather information, traffic information, event information, etc. related to the data log.

[0028] At 104, the system may detect and mark one or more objects and / or events in each received raw data log (e.g., point cloud, camera image, etc.). For example, an example object detection model (e.g., convolutional neural network (CNN), mask R-CNN, etc.) may be used to detect and mark one or more objects and / or events in each received raw data log. The object detection model may output an image or point cloud including a bounding box around the detected object and / or a mark of the object. The bounding box indicates an area of ​​the unlabeled data log that is believed to contain the identified object. Thus, the bounding box may be assigned a mark indicating the identified object that is believed to be contained within the bounding box. Each detected object may be represented by its location (center of mass or center point), boundary, mark category, or even size. In an embodiment, the object detector may output an image or point cloud including a bounding box around the detected object and / or a mark of the detected object.

[0029] The system can also identify a confidence score associated with each bounding box and / or identified tag, the confidence score indicating the degree of likelihood that the identified object exists in a portion of the unlabeled raw data enclosed by the bounding box or the degree of confidence that the detected object is correctly labeled. Specifically, the object detection model generates a confidence score for each bounding box, which indicates the likelihood that the tag assigned to the bounding box is correct (i.e., the object associated with the tag is depicted in the bounding box). The confidence score can be generated based on the classification accuracy of the object detection model for the identified object / tag. Optionally, the value of the confidence score can be about 0.0 to about 1.0. For example, Figure 2AThe illustrated data log is an example of an image 200 that includes an object 202 that is an orange scooter. When analyzing the image 200, the object detection model generates a bounding box 204 around the orange scooter and labels it as "construction" based on the orange color with a confidence score of 0.6 ( Figure 2B ). A bounding box containing a car may have a bounding box score of 1.00 (or even higher), indicating that the car was detected and correctly labeled as a “car” with approximately 100% accuracy.

[0030] Optionally, the system can also identify one or more attributes of (multiple) machine learning models (e.g., object detection models) to be trained using the data log, and use the identified attributes to determine a confidence score for the machine learning model associated with predicting each of the plurality of labels. Examples of the attributes can include, but are not limited to, actual and / or expected accuracy, false positive rate, false negative rate, convergence, statistical fit, output of the machine learning model, problem solved using the machine learning model, training status of the machine learning model, etc.

[0031] At 106, the system can identify distributions and / or trends related to data logs (e.g., occurrences, percentages of objects, events, and / or instances within the data logs) within an existing training dataset (including labeled data). Such identification can include analysis of various tags in the training dataset and metadata associated with the data logs in the training dataset (e.g., location of collection, time of collection, events, etc.). Example data distributions are as follows: Figure 3A-3C shown.

[0032] In some embodiments, the system can use distributions and / or trends to identify object categories (i.e., tags), objects, events, and / or instances that are missing or inadequately represented in the training data. For example, some types of objects (e.g., construction vehicles, trailers, etc.) may be missing and / or inadequately represented in the training data set; data logs associated with events (e.g., vehicles not stopping at red lights, lane change events, takeover events, etc.) may be missing and / or inadequately represented in the training data set); data logs collected at certain times and / or locations (e.g., at night, during morning rush hour, in parking lots, on rainy days, etc.) may be missing and / or inadequately represented in the training data set, etc. Figure 3A An exemplary labeled class count distribution of a training dataset is illustrated and can be used to identify labeled classes that are underrepresented in the training dataset (e.g., Figure 3A 301-related data log in the construction). Similarly, Figure 3BThe label density of the training dataset by time of day and location is illustrated, and can be analyzed to identify that data logs between 11:30 am and 12:30 pm are under-represented in the training dataset.

[0033] In other examples, certain types of objects, events, and / or instances that need to be included in and / or over-represented in existing training data and can also be recognized by the system using similar principles. Figure 3C The distribution of the training data set (i.e., event distribution) of the number of takeovers per 100,000 vehicle miles by time of day is shown. Figure 3C From the distribution of , we can infer that there are more takeover events during peak hours and more takeover events need to be included in the training dataset in order to improve the performance of the machine learning model during peak hours.

[0034] At 108, the system may assign an importance score to each of the received data logs (from step 102) based on the confidence score of step 104 and the existing training data set trend / distribution of step 106. The importance score of the data log indicates whether the data log will be useful for training the machine learning model. A data log is "useful" if, for example, the data log can be used for initial training of the machine learning model, reducing training time, improving the accuracy of the machine learning model, reducing overfitting (i.e., the log data has reduced redundancy), etc. The data log may be useful for correction learning and / or positive reinforcement learning. For example, events or objects detected within the data log that indicate incorrect detection results may be marked so that they can be used to adjust the model for better future performance (i.e., correction learning in which the model learns from errors). Events indicating correct detection results may also need to be marked so that they can be used to enhance the model instead of correction (i.e., positive reinforcement learning).

[0035] To assign importance scores to data logs, the system can first define an importance function that can be used to analyze various features of the data logs in order to determine the importance score of the data log with respect to training a machine learning model. The importance function combines the confidence score used to predict tags in the data log using the machine learning model with the trend / distribution of the existing training data set and assigns an importance score that indicates whether the data log will improve the training of the machine learning model.

[0036] Examples of features may include, but are not limited to, tag category counts, tag counts by time and location, event category counts, event category counts by time and location, instance counts by environmental conditions (e.g., rain, poor light, occlusion, etc.), machine learning model uncertainty, consistency of data logs with various events (e.g., near / near collision events during takeover, etc.), etc. The defined function may assign a percentile score or rank to each feature of the data log, and combine feature scores for determining an importance score for the data log. For example, an importance function may compare a confidence score with a (multiple) threshold to determine whether the confidence score needs to be improved for a certain tag category, and may assign a higher importance to a data log with a low confidence score for object detection. Similarly, an importance function may assign a higher importance to data logs that include category tags, event tags, and environmental factors that are not fully represented in existing data sets.

[0037] At 110, the system may determine whether the importance score of the data log is greater than a threshold. If the importance score of the data log is greater than (or equal to) the threshold (110: yes), the system may use the data log to further annotate (e.g., manually label) and / or build (i.e., train, test, validate, update, etc.) the machine learning model (112). If the importance score of the data log is less than the threshold (110: no), the system may not use the data log to update / train the machine learning model (114). Optionally, the system may discard data logs whose importance scores are less than the threshold.

[0038] For example, references include Figure 2A and Figure 2B For the data log of the image shown in , the importance function can assign a high importance score to the image (i.e., greater than a threshold) for two reasons: (1) the construction category (labeled category count) is important because the construction label is underrepresented in the existing training dataset ( Figure 3A (2) the object detection model is uncertain about the classification of the orange scooter (a low confidence score of 0.6 for classifying it as construction). Therefore, the importance function assigns a high score to the data log for both the feature label class count and the confidence score. This is because if the training dataset includes more construction labels and more objects that look like construction labels but are not actually construction labels (i.e., an orange scooter that is a false positive construction label), the detection algorithm will be less likely to mistake orange objects for construction cones in the future.

[0039] exist Figure 4A and Figure 4B In another example shown, Figure 4AIllustrated is an example of an image 400 where the data log includes an object 202 that is the base of a utility pole. In analyzing the image 400, the object detection model generates a bounding box 404 around the base of the utility pole and labels it as "construction" (based on the shape of the base being similar to a construction cone) ( Figure 4B ), which has a confidence score of 0.6. The importance function can assign a high importance score (i.e., greater than a threshold) to an image for two reasons: (1) the construction category (labeled category count) is important because construction labels are underrepresented in existing training datasets ( Figure 3A (2) the object detection model is uncertain about the classification of the pole base (a low confidence score of 0.6 for classifying it as construction). Therefore, the importance function assigns a high score to the data log for both the feature marker class count and the confidence score. This is because if the training dataset includes more construction markers and more objects that look like construction markers but are not (i.e., pole bases that are false positive construction markers), the detection algorithm will be less likely to mistake conical structures such as pole bases for construction cones in the future.

[0040] Thousands of data logs with labels and confidence scores can be collected from the perception module in the autonomous vehicle and stored in a database. Since there is too much sensor data to be stored in the actual system, the active learning system disclosed in this article helps reduce the amount of data logs stored in the database by identifying the error annotations (i.e., labels) that need to be manually labeled in the database and are important for building a machine learning model. This balances the quantity and quality of the data logs in the database to optimize system performance and cost. In addition, the impact of different types of detection and labeling errors (e.g., lost objects, inaccurate positions) on performance can be analyzed. The machine learning model is trained and / or constructed by adjusting the operating parameters of the machine learning model until the desired training output is achieved. The determination of whether the training output is desired can be completed manually or automatically by comparing the training output with the known characteristics of the training data. When the training output of the machine learning model is within a predetermined error threshold from the known characteristics of the training data, the machine learning model is considered to be trained or constructed. Examples of machine learning models may include, but are not limited to, tracking models, shape estimation models, object detection models, posture estimation models, classification and labeling models, segmentation models, motion planning models, prediction models, prediction models, etc.

[0041] Although the above description involves selecting raw or unlabeled images for subsequent labeling and for training machine learning models that perform object detection, the present disclosure is not so limited. In particular, the methods and systems of the present disclosure can be used to select any kind of raw data for training many different kinds of models, such as but not limited to tracking models, motion planning models, forecasting models, prediction models, and the like. For example, considering a prediction model for predicting whether a pedestrian will cross a zebra crossing based on, for example, the pedestrian's proximity to the zebra crossing, body orientation, facial features, and the like, the systems and methods of the present disclosure can be used to identify an importance function for selecting data for building a prediction model. In such examples, the importance function can be based on features, such as statistics about the data (e.g., where people tend to run red lights because the zebra crossing is too far away), the time of day when people use or do not use the zebra crossing, the impact of traffic conditions on the zebra crossing, and the like; in order to select useful data for building a model.

[0042] It should be noted that the methods for intelligent selection of data logs described in this disclosure may be performed onboard an autonomous vehicle, at a remote computing device that is not part of the autonomous vehicle, or a combination thereof.

[0043] Figure 5 5 is a block diagram illustrating an exemplary system 500 including an autonomous vehicle 501 communicating with one or more data stores 502 and / or one or more servers 503 via a network 510. Although one autonomous vehicle is shown, multiple autonomous vehicles may be communicatively coupled to each other and / or to data stores 502 and / or servers 503 via network 510. Network 510 may be any type of network, such as a local area network (LAN), a wide area network (WAN) such as the Internet, a cellular network, a satellite network, or a combination thereof, and may be wired or wireless. Data store(s) 502 may be any kind of data store, such as, but not limited to, map data store(s), historical information data store(s), traffic information data store(s), user information data store(s), point of interest data store(s), or any other type of content data store(s). Server(s) 503 may be any kind of server or server cluster, such as, but not limited to, a web or cloud server, an application server, a backend server, or a combination thereof.

[0044] like Figure 5 As shown in FIG. 5 , autonomous vehicle 501 may include sensor system 511, onboard computing device 512, communication interface 514, and user interface 515. Autonomous vehicle 501 may also include certain components contained in the vehicle (e.g., Figure 5), such as an engine, wheels, a steering wheel, a transmission, etc., which can be controlled by the onboard computing device 512 using various communication signals and / or commands (such as acceleration signals or commands, deceleration signals or commands, steering signals or commands, braking signals or commands, etc.).

[0045] Sensor system 511 may include one or more sensors coupled to and / or included within autonomous vehicle 501 and configured to sense or measure properties of the environment of the autonomous vehicle. Examples of such sensors include, but are not limited to, LIDAR systems, radio detection and ranging (RADAR) systems, laser detection and ranging (LIDAR) systems, sound navigation and ranging (SONAR) systems, one or more cameras (e.g., visible spectrum cameras, infrared cameras, etc.), temperature sensors, position sensors (e.g., global positioning systems (GPS), etc.), position sensors, fuel sensors, motion sensors (e.g., inertial measurement units (IMU), etc.), humidity sensors, occupancy sensors, etc. Sensor data may include information describing the location of objects within the surrounding environment of autonomous vehicle 501, information about the environment itself, information about the movement of autonomous vehicle 501, information about the route of autonomous vehicle, etc. At least some of the sensors may collect data related to the surface as autonomous vehicle 501 travels on it.

[0046] The LIDAR system may include a sensor configured to use light to sense or detect an object in the environment in which the autonomous vehicle 501 is located. Generally, a LIDAR system is a device that combines optical remote sensing technology, which can measure the distance to a target or other properties of the target by illuminating the target (e.g., the ground) with light. As an example, the LIDAR system may include a laser source and / or a laser scanner configured to emit laser pulses and a detector configured to receive the reflection of the laser pulses. For example, the LIDAR system may include a laser rangefinder reflected by a rotating mirror, and scan the laser around the digitized scene in one, two or more dimensions, thereby collecting distance measurements at specified angular intervals. For example, the LIDAR system may be configured to emit laser pulses as a beam, and scan the beam to generate a two-dimensional or three-dimensional range matrix. In an example, the LIDAR system may include components such as a light (e.g., laser) source, a scanner and optical devices, a photodetector and a receiver electronic device, and a positioning and navigation system. In an example, the LIDAR system may be configured to use ultraviolet (UV), visible or infrared light to image an object, and may be used with a variety of targets including non-metallic objects. In one example, a narrow laser beam can be used to map physical features of an object with high resolution. A wide divergent laser beam can be used with a single pulse and / or to illuminate the field of view point by point.

[0047] Figure 6 The diagram shows the Figure 5 601 of autonomous vehicle 501. Vehicle 601 may include an engine or motor 602 and various sensors for measuring various parameters of the vehicle and / or its environment. Operating parameter sensors common to both types of vehicles include, for example: a position sensor 636, such as an accelerometer, gyroscope, and / or inertial measurement unit; a speed sensor 638; and an odometer sensor 640. Vehicle 601 may also have a clock 642, which the system architecture uses to determine the vehicle time during operation. Clock 642 may be encoded into the vehicle onboard computing device 612, which may be a separate device, or multiple clocks may be available.

[0048] The vehicle 601 may also include various sensors that operate to collect information about the environment in which the vehicle is traveling. These sensors may include, for example: positioning sensors 660, such as a GPS device; object detection sensors, such as one or more cameras 662; LiDAR sensor systems 664; and / or radar and / or sonar systems 666. The sensors may also include environmental sensors 668, such as precipitation sensors and / or ambient temperature sensors. Object detection sensors may enable the vehicle 601 to detect objects in any direction within a given distance or range of the vehicle 601, while environmental sensors collect data about environmental conditions within the vehicle's travel area. The system architecture will also include one or more cameras 662 for capturing images of the environment.

[0049] During operation, information is communicated from the sensors to the onboard computing device 612. The onboard computing device 612 analyzes the data captured by the sensors and optionally controls the operation of the vehicle based on the results of the analysis. For example, the onboard computing device 612 can control braking via a brake controller 622; direction via a steering controller 624; speed and acceleration via a throttle controller 626 (in a gas-powered vehicle) or a motor speed controller 628 (such as a current level controller in an electric vehicle); a differential gear controller 630 (in a vehicle with a transmission); and / or other controllers, such as an auxiliary device controller 614.

[0050] Geolocation information may be transmitted from the location sensor 620 to the onboard computing device 612, which may then access a map of the environment corresponding to the location information to determine known fixed features of the environment, such as streets, buildings, stop signs, and / or stop / go signs. Captured images from the camera 662 and / or object detection information captured from sensors such as the LiDAR system 664 are transmitted from these sensors to the onboard computing device 612. The object detection information and / or captured images may be processed by the onboard computing device 612 to detect objects approaching the vehicle 601. Additionally or alternatively, the vehicle 601 may send any data to the remote server system 503 ( Figure 5 ) for processing. Any known or to-be-known technique for object detection based on sensor data and / or captured images may be used in the embodiments disclosed in this document.

[0051] The onboard computing device 612 may obtain, retrieve, and / or create map data that provides detailed information about the surrounding environment of the autonomous vehicle 601. The onboard computing device 612 may also determine the position, orientation, attitude, etc. (positioning) of the AV in the environment based on, for example, three-dimensional position data (e.g., data from GPS), three-dimensional orientation data, predicted position, etc. For example, the onboard computing device 612 may receive GPS data to determine the latitude, longitude, and / or altitude position of the AV. Other position sensors or systems, such as laser-based positioning systems, inertial assisted GPS, or camera-based positioning, may also be used to identify the location of the vehicle. The location of the vehicle may include an absolute geographic location, such as latitude, longitude, and altitude, as well as relative position information, such as relative to the location of other cars in its immediate surroundings, which may generally be determined with less noise than the absolute geographic location. The map data may provide information regarding: the identification and location of different roads, road segments, lane segments, buildings, or other items; the location, boundaries, and directions of traffic lanes (e.g., the location and directions of parking lanes, turn lanes, bicycle lanes, or other lanes within a particular road) and metadata associated with traffic lanes; traffic control data (e.g., the location and instructions of signs, traffic lights, or other traffic control devices); and / or any other map data that provides information to assist onboard computing device 612 in analyzing the surroundings of autonomous vehicle 601. The map data may also include information and / or rules for determining the right of way of objects and / or vehicles in a conflict area or space.

[0052] In some embodiments, the map data may also include reference path information corresponding to common patterns of vehicle travel along one or more lanes, such that the movement of an object is constrained to a reference path (e.g., a location within a lane of traffic on which the object typically travels). Such a reference path may be predefined, such as a centerline of a lane of traffic. Alternatively, a reference path may be generated based on historical observations of vehicles or other objects over a period of time (e.g., reference paths for straight driving, lane merging, turning, etc.).

[0053] In certain embodiments, the in-vehicle computing device 612 may also include and / or may receive information regarding the user's itinerary or route, real-time traffic information regarding the route, and the like.

[0054] The onboard computing device 612 may include and / or communicate with a route controller 631 that generates a navigation route for an autonomous vehicle from a starting location to a destination location. The route controller 631 may access a map data store to identify possible routes and road segments that a vehicle may travel to reach a destination location from a starting location. The route controller 631 may score possible routes and identify a preferred route to a destination. For example, the route controller 631 may generate a navigation route that minimizes the Euclidean distance or other cost function traveled during the route, and may also access traffic information and / or estimates that may affect the amount of time it will take to travel on a particular route. Depending on the implementation, the route controller 631 may use various route selection methods, such as a Dijkstra algorithm, a Bellman-Ford algorithm, or other algorithms to generate one or more routes. The route controller 631 may also use traffic information to generate a navigation route that reflects the expected conditions of the route (e.g., the current day of the week or the current time, etc.), so that the route generated for travel during peak hours may be different from the route generated for late night travel. The route controller 631 may also generate more than one navigation route to a destination, and send more than one of these navigation routes to the user for the user to select from various possible routes.

[0055] In various embodiments, the onboard computing device 612 may determine the perception information of the surrounding environment of the autonomous vehicle 601. Based on the sensor data provided by one or more sensors and the obtained positioning information, the onboard computing device 612 may determine the perception information of the surrounding environment of the autonomous vehicle 601. The perception information may represent what an ordinary driver would perceive in the surrounding environment of the vehicle. The perception data may include information related to one or more objects in the environment of the autonomous vehicle 601. For example, the onboard computing device 612 may process the sensor data (e.g., LIDAR or RADAR data, camera images, etc.) to identify objects and / or features in the environment of the autonomous vehicle 601. Objects may include traffic signals, road boundaries, other vehicles, pedestrians, and / or obstacles, etc. The onboard computing device 612 may determine the perception using any object recognition or detection algorithm, video tracking algorithm, and computer vision algorithm (e.g., iteratively tracking objects frame by frame over multiple time periods) known now or later. In an embodiment, the onboard computing device 512 may include an object detector that detects and marks objects.

[0056] In some embodiments, the onboard computing device 612 may also determine the current state of the object for one or more identified objects in the environment. The state information may include, but is not limited to, for each object: current location; current speed and / or acceleration, current heading; current pose; current shape, size, or footprint; type (e.g., vehicle vs. pedestrian vs. bicycle vs. static object or obstacle); and / or other state information. As discussed in more detail below, the onboard computing device 612 may also identify the lane that is being occupied by the object at any given time.

[0057] The onboard computing device 612 may perform one or more prediction and / or forecasting operations. For example, the onboard computing device 612 may predict the future location, trajectory, and / or motion of one or more objects. For example, the onboard computing device 612 may predict the future location, trajectory, and / or motion of an object based at least in part on perception information (e.g., state data for each object including an estimated shape and pose determined as discussed below), positioning information, sensor data, and / or any other data describing the past and / or current state of the object, the autonomous vehicle 601, the surrounding environment, and / or their (multiple) relationships. For example, if the object is a vehicle and the current driving environment includes an intersection, the onboard computing device 612 may predict whether the object will likely move straight ahead or turn. If the perception data indicates that there is no traffic light at the intersection, the onboard computing device 612 may also predict whether the vehicle may have to stop completely before entering the intersection.

[0058] The prediction and / or forecasting operations of the onboard computing device 612 can also be used to reduce the number of data samples (e.g., images) required to perform de novo object recognition by tracking previously detected and / or classified objects between frames in a sequence of image frames. In some examples, object tracking can be performed via point tracking, such as by a deterministic method (e.g., using parameter constraints based on the object class of the object) or a statistical method (e.g., Kalman filtering). In some other examples, object tracking can be performed via core filtering and core tracking, such as using a template-based method or a multi-view appearance method. Optionally, object tracking can be performed via silhouette tracking, such as using shape matching, edge matching, and / or contour tracking. However, object tracking and trajectory prediction and / or determination can be determined using motion analysis, or object tracking and trajectory prediction and / or determination can be appropriately performed via any suitable method or technique.

[0059] In various embodiments, onboard computing device 612 can determine a motion plan for the autonomous vehicle. For example, onboard computing device 612 can determine a motion plan for the autonomous vehicle based on the perception data and / or the prediction data. Specifically, given predictions and other perception data about the future locations of neighboring objects, onboard computing device 612 can determine a motion plan for autonomous vehicle 601 that optimally navigates the autonomous vehicle relative to objects at the future locations of the autonomous vehicle.

[0060] In one or more embodiments, the onboard computing device 612 can receive predictions and make decisions about how to handle objects in the environment of the autonomous vehicle 601. For example, for a specific object (e.g., a vehicle with a given speed, direction, steering angle, etc.), the onboard computing device 612 decides whether to overtake, yield, stop, and / or pass based on, for example, traffic conditions, map data, the state of the autonomous vehicle, etc. In addition, the onboard computing device 612 also plans the path of the autonomous vehicle 601 on a given route and driving parameters (e.g., distance, speed, and / or steering angle). That is, for a given object, the onboard computing device 612 decides what to do with the object and determines how to do it. For example, for a given object, the onboard computing device 612 can decide to pass the object and can determine whether to pass on the left or right side of the object (including motion parameters such as speed). The onboard computing device 612 can also evaluate the risk of collision between the detected object and the autonomous vehicle 601. If the risk exceeds an acceptable threshold, it can be determined whether the collision can be avoided if the autonomous vehicle follows a defined vehicle trajectory and / or implements one or more dynamically generated emergency maneuvers executed in a predefined time period (e.g., N milliseconds). If the collision can be avoided, the onboard computing device 612 may execute one or more control instructions to perform a cautious maneuver (e.g., moderate deceleration, acceleration, lane change, or swerve). Conversely, if the collision cannot be avoided, the onboard computing device 612 may execute one or more control instructions to perform an emergency maneuver (e.g., braking and / or changing the direction of travel).

[0061] For example, if onboard computing device 612 determines that an object may violate a right-of-way agreement with the autonomous vehicle based on the object trajectory forecast, onboard computing device 612 may determine a motion plan for the autonomous vehicle that avoids collision with such object in the space of conflict.

[0062] As described above, planning and control data for movement of the autonomous vehicle are generated for execution. The onboard computing device 612 can control braking, for example, via a brake controller; control direction via a steering controller; control speed and acceleration via a throttle controller (in a gas-powered vehicle) or a motor speed controller (such as a current level controller in an electric vehicle), a differential gear controller (in a vehicle with a transmission), and / or other controllers.

[0063] In various embodiments discussed in this document, the description may state that a vehicle or a controller included in a vehicle (e.g., in an onboard computing system) may implement programming instructions that cause the vehicle and / or the controller to make a decision and use the decision to control the operation of the vehicle. However, the embodiments are not limited to this arrangement, as in various embodiments, analysis, decision making, and / or operational control may be handled in whole or in part by other computing devices that are in electronic communication with the vehicle's onboard computing device and / or the vehicle control system. Examples of such other computing devices include electronic devices associated with a person riding in the vehicle (such as a smart phone), and a remote server that is in electronic communication with the vehicle via a wireless communication network. The processor of any such device may perform the operations discussed below.

[0064] Return to reference Figure 5 , the communication interface 514 can be configured to allow communication between the autonomous vehicle 501 and external systems, such as external devices, sensors, other vehicles, servers, data stores, databases, etc. The communication interface 514 can utilize any now or later known protocol, protection scheme, encoding, format, packaging, etc., such as but not limited to Wi-Fi, infrared link, Bluetooth, etc. The user interface system 515 can be part of the peripherals implemented in the vehicle 501, including, for example, a keyboard, a touch screen display device, a microphone and a speaker, etc.

[0065] Figure 7 Depicted are examples of internal hardware that may be included in any electronic component of the system, such as an internal processing system of an AV, an external monitoring and reporting system, or a remote server. Electrical bus 700 serves as an information highway for interconnecting the other illustrated components of the hardware. Processor 705 is the central processing device of the system, which is configured to perform the computational and logical operations required by programming instructions. As used in this document and the claims, the terms "processor" and "processing device" may refer to a single processor or any number of processors in a collection of processors that collectively perform a set of operations, such as a central processing unit (CPU), a graphics processing unit (GPU), a remote server, or a combination of these. Read-only memory (ROM), random access memory (RAM), flash memory, hard disk drives, and other devices capable of storing electronic data constitute examples of memory devices 725. Memory devices may include a single device or a collection of devices, and data and / or instructions are stored across a single device or a collection of devices. Various embodiments of the present disclosure may include a computer-readable medium containing programming instructions that are configured to cause one or more processors and / or computing devices to perform the functions described in the context of the previous figures.

[0066] The optional display interface 730 can allow information from the bus 700 to be displayed in a visual, graphical or alphanumeric format on a display device 735, such as on an in-dash display system of a vehicle. An audio interface and audio output (such as a speaker) can also be provided. Communication with external devices can be carried out using various communication devices 740, such as wireless antennas, radio frequency identification (RFID) tags and / or short-range or near-field communication transceivers, each of which can be optionally communicatively connected to other components of the device via one or more communication systems. (Multiple) communication devices 740 can be configured to be communicatively connected to a communication network, such as the Internet, a local area network, or a cellular telephone data network.

[0067] The hardware may also include user interface sensors 745 that allow data to be received from input devices 750 such as a keyboard or keypad, joystick, touch screen, touchpad, remote control, pointing device, and / or microphone. Digital image frames may also be received from a camera 720 that may capture video and / or still images. The system may also receive data from motion and / or position sensors 770 such as accelerometers, gyroscopes, or inertial measurement units. The system may also receive data from a LiDAR system 760 such as described earlier in this document.

[0068] The features and functions disclosed above and alternatives may be combined into many other different systems or applications. The various components may be implemented in hardware or software or embedded software. Those skilled in the art may make various currently unforeseen or unanticipated substitutions, modifications, variations or improvements, each of which is also intended to be covered by the disclosed embodiments.

[0069] Terms related to the disclosure provided above include:

[0070] An “automated device” or “robotic device” means an electronic device that includes a processor, programming instructions, and one or more physical hardware components that, in response to commands from the processor, can move with minimal or no human intervention. Through such movement, the robotic device can perform one or more automated functions or sets of functions. Examples of such operations, functions, or tasks may include, but are not limited to, operating wheels or propellers to achieve driving, flight, or other transportation actions, operating a robotic lift for loading, unloading, medical-related procedures, construction-related processes, and the like. Example automated devices may include, but are not limited to, autonomous vehicles, drones, and other autonomous robotic devices.

[0071] The term "vehicle" refers to any mobile form of transport capable of carrying one or more human occupants and / or cargo and powered by any form of energy. The term "vehicle" includes, but is not limited to, cars, trucks, vans, trains, autonomous vehicles, aircraft, drones, etc. An "autonomous vehicle" is a vehicle with a processor, programming instructions, and a drivetrain assembly that can be controlled by the processor without the need for a human operator. An autonomous vehicle can be fully autonomous because it does not require a human operator for most or all driving conditions and functions, or it can be semi-autonomous because an operator may be required under certain conditions or for certain operations, or an operator can override the autonomous system of the vehicle and can take control of the vehicle. Autonomous vehicles also include vehicles in which autonomous systems enhance human operation of the vehicle, such as vehicles with driver-assisted steering, speed control, braking, parking and other systems.

[0072] The term "machine learning model" or "model" refers to a collection of algorithmic routines and parameters that can predict (multiple) outputs of a real-world process (e.g., prediction of an object trajectory, diagnosis or treatment of a patient, appropriate recommendations based on a user search query, etc.) based on a set of input features without being explicitly programmed. The structure of the software routine (e.g., the number of subroutines and the relationships between them) and / or the values ​​of the parameters can be determined during a training process that can use actual results of the real-world process being modeled. Such systems or models are understood to be necessarily rooted in computer technology, and, indeed, could not be implemented or even exist without computing technology. Although machine learning systems utilize various types of statistical analysis, machine learning systems are distinguished from statistical analysis by their ability to learn without explicit programming and being rooted in computer technology.

[0073] A typical machine learning pipeline may include building a machine learning model from a sample dataset (referred to as a “training set”), evaluating the model against one or more additional sample datasets (referred to as a “validation set” and / or a “test set”) to decide whether to keep the model and to benchmark how good the model is, and using the model in “production” to make predictions or decisions on live input data captured by an application service.

[0074] The term "bounding box" refers to a rectangular box that represents the location of an object. A bounding box can be represented in the data by the x and y coordinates [x max ,y max ] and the x and y coordinates [x min ,y min] It can be calculated as the smallest rectangle that contains all the points of the object, optionally plus additional space to allow for error tolerance. The points of the object can be those detected by one or more sensors, such as pixels of an image captured by a camera, or points of a point cloud captured by a LiDAR sensor.

[0075] An “electronic device” or “computing device” refers to a device that includes a processor and memory. Each device may have its own processor and / or memory, or the processor and / or memory may be shared with other devices, such as in a virtual machine or container arrangement. The memory will contain or receive programming instructions that, when executed by the processor, cause the electronic device to perform one or more operations in accordance with the programming instructions.

[0076] The terms "memory," "memory device," "data storage," "data storage facility," and the like each refer to a non-transitory device on which computer-readable data, programming instructions, or both are stored. Unless specifically stated otherwise, the terms "memory," "memory device," "data storage," "data storage facility," and the like are intended to encompass single device embodiments, embodiments in which multiple memory devices together or collectively store a collection of data or instructions, and individual sectors within such devices.

[0077] The terms "processor" and "processing device" refer to a hardware component of an electronic device that is configured to execute programmed instructions. Unless otherwise specifically stated, the singular term "processor" or "processing device" is intended to include both single processing device embodiments and embodiments in which multiple processing devices together or collectively perform processing.

[0078] In this document, the terms "communication link" and "communication path" mean a wired or wireless path via which a first device sends communication signals to and / or receives communication signals from one or more other devices. Devices are "communicatively connected" or "communicatively coupled" if they are capable of sending and / or receiving data via a communication link. "Electronic communication" refers to the transmission of data via one or more signals between two or more electronic devices, whether over a wired or wireless network, and whether directly or indirectly via one or more intermediate devices.

[0079] In this document, when relative order terms such as "first" and "second" are used to modify nouns, such usage is intended only to distinguish one item from another, and is not intended to require a sequential order unless specifically stated.

[0080] In addition, terms of relative position, such as "vertical" and "horizontal", or "front" and "rear", when used, are intended to be relative to each other and are not necessarily absolute, and refer only to one possible position of the device associated with these terms based on the orientation of the device. When the terms "front", "rear" and "side" are used herein to refer to areas of a vehicle, they refer to areas of the vehicle relative to the default driving area of ​​the vehicle. For example, the "front" of a car is the area closer to the vehicle's headlights than the vehicle's taillights, and the "rear" of a car is the area closer to the vehicle's taillights than the vehicle's headlights. In addition, the terms "front" and "rear" are not necessarily limited to areas facing forward or rearward, but also include side areas that are closer to the front than the rear, and vice versa. The "side" of a vehicle is intended to mean a side-facing portion located between the front-most and rear-most of the vehicle.

Claims

1. A method for selecting data for building a machine learning model using active learning, the method comprising: receiving a plurality of unlabeled sensor data logs corresponding to a surrounding environment of an autonomous vehicle from a plurality of sensors; identifying one or more trends associated with a training data set comprising a plurality of labeled data logs, the training data set being used to train a machine learning model; determining a function for assigning a significance score to each of the plurality of unlabeled sensor data logs using the one or more trends; assigning the importance score to each of the plurality of unlabeled sensor data logs using the function; selecting a subset of the plurality of sensor data logs having a significance score greater than a threshold; as well as The machine learning model trained using the training data set is further trained using the subset of the plurality of sensor data logs to generate an updated model.

2. The method according to claim 1, further comprising: utilizing the machine learning model trained using the training data set, identifying a bounding box and an associated label within each of the plurality of sensor data logs; determining a confidence score for the machine learning model trained using the training data set for identifying the associated tags; as well as The confidence score is used in determining the function.

3. The method according to claim 2, wherein: The machine learning model is an object detection model.

4. The method according to claim 2, wherein: The confidence score is determined based on at least one of the following associated with the machine learning model trained using the training data set: actual accuracy; expected accuracy; false positive rate; false negative rate; convergence; output; statistical fit; identification of a problem to be solved; or training status.

5. The method according to claim 2, further comprising: receiving metadata associated with each of the plurality of labeled data logs in the training data set; as well as The one or more trends are identified using the metadata.

6. The method according to claim 5, wherein: The metadata includes at least one of: a capture time of the data log; a capture location of the data log; information related to environmental conditions at the capture location or time of the data log; or information related to one or more events at the capture location or time of the data log.

7. The method according to claim 2, wherein: The one or more trends are used to identify one or more underrepresented characteristics in the plurality of labeled data logs of the training data set.

8. The method according to claim 7, wherein: The one or more characteristics are selected from at least one of: tag category count; tag count by time; tag count by location; event category count; event category count by time; event category count by location; instance count by environmental condition; or machine learning model uncertainty.

9. The method of claim 2, further comprising receiving metadata associated with the sensor data log, the metadata comprising at least one of: a capture time of the data log; a capture location of the data log; information related to environmental conditions at the capture location or time of the data log; or information related to one or more events at the capture location or time of the data log.

10. The method according to claim 9 also includes using the function to assign the importance score to each of the multiple unlabeled sensor data logs by assigning scores to one or more of multiple characteristics of the data logs, the multiple characteristics including at least one of the following: tag category count; tag count by time; tag count by location; event category count; event category count by time; event category count by location; instance count by environmental condition; or machine learning model uncertainty. 11 . The method of claim 2 , further comprising discarding one or more sensor data logs among the plurality of sensor data logs having an importance score less than the threshold.

12. A system for selecting data for training a machine learning model using active learning, the system comprising: processor; as well as A non-transitory computer readable medium comprising one or more programming instructions that, when executed by the processor, cause the processor to: receiving a plurality of unlabeled sensor data logs corresponding to a surrounding environment of the autonomous vehicle from a plurality of sensors of the autonomous vehicle; identifying one or more trends associated with a training data set comprising a plurality of labeled data logs, the training data set being used to train a machine learning model; determining a function for assigning a significance score to each of the plurality of unlabeled sensor data logs using the one or more trends; assigning the importance score to each of the plurality of unlabeled sensor data logs using the function; selecting a subset of the plurality of sensor data logs having a significance score greater than a threshold; as well as The machine learning model trained using the training data set is further trained using the subset of the plurality of sensor data logs to generate an updated model.

13. The system of claim 12, further comprising programming instructions that, when executed by the processor, cause the processor to: utilizing the machine learning model trained using the training data set, identifying a bounding box and an associated label within each of the plurality of sensor data logs; determining a confidence score for the machine learning model trained using the training data set for identifying the associated tags; as well as The confidence score is used in determining the function.

14. The system according to claim 13, wherein: The machine learning model is an object detection model.

15. The system of claim 13, wherein: The confidence score is determined based on at least one of the following associated with the machine learning model trained using the training data set: actual accuracy; expected accuracy; false positive rate; false negative rate; convergence; output; statistical fit; identification of a problem to be solved; or training status.

16. The system of claim 13, further comprising programming instructions that, when executed by the processor, cause the processor to: receiving metadata associated with each of the plurality of labeled data logs in the training data set; and The one or more trends are identified using the metadata.

17. The system of claim 16, wherein: The metadata includes at least one of: a capture time of the data log; a capture location of the data log; information related to environmental conditions at the capture location or time of the data log; or information related to one or more events at the capture location or time of the data log.

18. The system of claim 13, wherein: The one or more trends are used to identify one or more underrepresented characteristics in the plurality of labeled data logs of the training data set.

19. The system of claim 18, wherein: The one or more characteristics are selected from at least one of: tag category count; tag count by time; tag count by location; event category count; event category count by time; event category count by location; instance count by environmental condition; or machine learning model uncertainty.

20. The system of claim 12, further comprising programming instructions that, when executed by the processor, cause the processor to: receive metadata associated with the sensor data log, the metadata comprising at least one of: a capture time of the data log; a capture location of the data log; information related to environmental conditions at the capture location or time of the data log; or information related to one or more events at the capture location or time of the data log.

21. The system of claim 20, further comprising programming instructions that, when executed by the processor, cause the processor to perform the following operations: use the function to assign the importance score to each of the plurality of unlabeled sensor data logs by assigning scores to one or more of a plurality of characteristics of the data logs, the plurality of characteristics comprising at least one of: tag category count; tag count by time; tag count by location; event category count; event category count by time; event category count by location; instance count by environmental condition; or machine learning model uncertainty.

22. The system of claim 12, further comprising programming instructions that, when executed by the processor, cause the processor to discard one or more sensor data logs of the plurality of sensor data logs having a significance score less than the threshold.

Images