Efficient active learning

Abstract

The present invention provides a method for assisted training of a machine learning model for computer-assisted vehicle control, the method comprising: in an iterative manner - sampling one or more candidate data points from a set of unlabeled data points; - calculating an acquisition function for the candidate data points; - removing a first data point from the set of unlabeled data points if a first condition is met for a first data point in the set of unlabeled data points, relating to a first value of the acquisition function for the first data point; - obtaining a label for a second data point in the set of candidate data points and adding the second data point to the set of labeled data points with the label if a second condition is met for a second value of the acquisition function for the second data point; and - training a machine learning model using the set of labeled data points.

Description

Technical Field

[0001] The present invention relates to a method and a machine learning device for assisted training of a machine learning model for computer-aided vehicle control.

[0002] The present invention also relates to a computer-readable storage medium storing program code, the program code including instructions for performing such a method.

Background Art

[0003] Deep convolutional neural networks (CNNs) have come to outperform classical image processing algorithms. However, such models require an enormous amount of training samples. In current literature, there are several approaches to address this problem. One of these approaches is based on the hypothesis that only a very small percentage of training samples have important information for our machine learning model. Since taking in new unlabeled images is much less costly than manual annotation, it is clear that selecting the most informative inputs for the neural network if labeled can reduce the number of samples required. This process is called active learning (AL).

[0004] One of the important problems when using AL for deep learning applications is that training deep neural networks is computationally expensive, and due to the complexity of the mapping being learned, classical active learning techniques will always fail.

Summary of the Invention

Means for Solving the Problems

[0005] The object of the present invention is to provide a method for assisted training of a machine learning model for computer-assisted vehicle control, and a machine learning device configured to perform the method for assisted training of a machine learning model for computer-assisted vehicle control, which overcome one or more of the aforementioned problems of the prior art.

[0006] A first aspect of the present invention is a method for assisted training of a machine learning model for computer-assisted vehicle control, the method comprising an iterative method, - Sampling one or more candidate data points from a set of unlabeled data points, - Calculate the acquisition function for candidate data points, - For a first data point in the set of unlabeled data points, if a first condition related to a first value of the acquisition function for the first data point is met, remove the first data point from the set of unlabeled data points, - For a second data point in the set of candidate data points, if a second condition related to the second value of the acquisition function for the second data point is met, obtain the label for the second data point and add the second data point along with the label to the set of labeled data points. - Training a machine learning model using a set of labeled data points and This provides a method that includes [something].

[0007] The method of the first embodiment has the advantage that the set of unlabeled data points is reduced by detecting one or more first data points that satisfy a first condition (which indicate low importance to training) and removing these data points, thereby reducing the computational complexity of training. In particular, this method may be implemented so that these removed data points are completely removed, i.e., no memory space and / or computation time is required to process these data points. In other embodiments, the removed data points are still stored in some memory, i.e., they are not completely deleted but are ignored, which reduces computation time.

[0008] Preferably, the first condition is defined such that it is satisfied when the first data point is unlikely to be relevant to training the machine learning model, and / or the second condition is defined such that it is satisfied when the second data point is important to training the machine learning model.

[0009] The first embodiment of the method is particularly advantageous when working with large datasets where the amount of available data is greater than what the machine learning model can process sensingly. For example, data points may include images, videos, radar, LiDAR, and / or additional sensor data. In particular, this sensor data may be acquired from an entire moving vehicle or a platoon of moving vehicles. It should be understood that a platoon of moving vehicles can acquire terabytes or even pallet-bits of input data in a short period of time. In such scenarios, the first embodiment of the method is particularly beneficial because it can iteratively remove data points that do not need to be labeled for training, thereby reducing the amount of input data that needs to be stored and labeled to a more manageable size.

[0010] Annotation in the first embodiment of the method may be performed by a human user. Thus, the method can be user-assisted. However, assistance can also be machine-assisted, for example, by some automated annotation mechanism, which is also "expensive," and it may still be preferable that machine annotation is not required to be performed for each data point in the dataset. In further embodiments, annotation may be performed by both a human annotator and a machine, i.e., a combination of the two methods may be used.

[0011] The method of the first embodiment may be performed on a dataset of fully available unlabeled data points at the start, or it may be performed by iteratively adding data points to the input data. In particular, new unlabeled data points may be added as soon as they are created, for example, as soon as they are acquired by one or more data acquisition devices, such as a convoy of vehicles equipped with sensors and using those sensors to acquire training data.

[0012] Please understand that the initial sampling step may be omitted if the input data is not yet available or if there are only a few data points and sampling is not required.

[0013] In a first implementation of the method according to the first aspect, the first condition is satisfied if the first value of the acquisition function for the first data point is ranked among a first predetermined number of minimum acquisition values.

[0014] In this implementation, the first condition (which results in the removal of the first data point from the set of unlabeled data points) is met when the acquisition function (which can be thought of as an indication of the importance of a given data point being sampled) is determined to be low enough that it is clearly irrelevant to learning. Even if the value of the acquisition function for this first data point increases in future iterations, it is still very unlikely to reach a value high enough to be selected for (expensive) annotation. Therefore, the first condition is met, and the data point is removed from the set of unlabeled data points (and will never appear for annotation).

[0015] The first predetermined number can be predetermined based on a preference for how many data points should be removed during iterations. A high value for the first predetermined number leads to the removal of many data points from the dataset. This has the effect of significantly reducing the size of the dataset, which can reduce the computational burden, but may lead to low-quality training results. On the other hand, a low value for the first predetermined number leads to the removal of only a few data points from the dataset. Therefore, the size of the dataset may increase, and the computational burden may increase. In practice, the first predetermined number can be chosen as an appropriate compromise.

[0016] In a further implementation of the method according to the first embodiment, the first condition is satisfied when the first value of the acquisition function is lower than a predetermined first threshold.

[0017] In a further implementation of the method according to the first embodiment, the second condition is satisfied if the second value of the acquisition function for the second data point is among a second predetermined number of maximum acquisition values.

[0018] In a further implementation, the second condition is met if the second value of the acquisition function for the second data point is higher than a predetermined second threshold.

[0019] In a further implementation of the method according to the first embodiment, sampling is performed such that in one iteration, a predetermined share of data points from a set of unlabeled data points is sampled as candidate data points, preferably the predetermined share being between 3% and 30%, particularly between 5% and 20%.

[0020] In a further implementation of the method according to the first embodiment, sampling is performed such that the higher the confidence value of the acquisition function for a given data point from a set of unlabeled data points, the more likely the given data point is to be sampled as a candidate data point. Thus, sampling does not completely ignore data points with low confidence values, but rather gives greater attention to data points with high confidence values.

[0021] In a further implementation of the method according to the first embodiment, sampling includes a step of determining a sampling probability based on the sampling temperature, preferably the sampling temperature being optimized in a hyperparameter optimization process. It should be understood that both high and low sampling temperatures may have specific advantages. The hyperparameter optimization process has proven to be ideal for determining the optimal value in a given application scenario.

[0022] In a further implementation of the method according to the first aspect, the machine learning model includes a neural network, in particular a convolutional neural network.

[0023] In a further implementation of the method according to the first aspect, the machine learning model is trained to enhance the operation of the vehicle. For the operation of the vehicle, a large amount of training data is often available. Vehicle sensors usually include one or more cameras, or a set of one or more cameras, which can acquire a large amount of data in a short time. For example, a video stream from one camera may involve recording exceeding 1GB per hour of video. Therefore, the method of the first aspect that can lead to a reduction in the size of the training data set can be employed to efficiently reduce the amount of training data and / or to reduce the number of (e.g., manual) annotations involved in labeling the training data.

[0024] In a further implementation of the method according to the first aspect, the vehicle is a fully or partially autonomous vehicle. For autonomous vehicles, it is particularly important that they are trained in many different scenarios. Therefore, the method of the first aspect is particularly important in that it helps to concentrate (expensive and / or timely) annotations (e.g., manual annotations) on important data points for the learning of the machine learning model.

[0025] In a further implementation of the method according to the first aspect, the machine learning model is trained to detect objects in 3D data and / or to predict trajectories in 3D data. Preferably, the 3D data includes a point cloud. Further, the training can be with the goal of performing 2D detection, segmentation, object detection in 2D or 3D, and / or any type of detection for automotive purposes in general.

[0026] In a further implementation of the method according to the first aspect, the data points include sensor data, particularly sensor data from a camera or a set of cameras, RADAR and / or LIDAR, or any type of sensor data used for training purposes.

[0027] In a further implementation of the method according to the first aspect, obtaining a label for a second data point includes outputting a data point acquisition instruction to the user and obtaining a label from the user. Thus, the method of the first aspect can be used to directly instruct the user to label the most important data points.

[0028] A second aspect of the present invention refers to a machine learning device configured to perform one of the methods or implementations of the first aspect. The machine learning device can be implemented on a desktop computer or a server, such as a data center.

[0029] A further aspect of the present invention is a computer-readable storage medium storing program code, the program code including instructions to perform one of the methods or implementations of the first aspect when executed by a processor.

[0030] To more clearly explain the technical features of the embodiments of the present invention, the accompanying drawings provided for explaining the embodiments are briefly introduced below. The accompanying drawings in the following description are only some embodiments of the present invention, and changes to these embodiments are possible without departing from the scope of the present invention defined in the claims.

Brief Description of the Drawings

[0031] [Figure 1] It is a flowchart showing a conventional method of active learning. [Figure 2] It is a flowchart showing the method according to the first embodiment of the present invention. [Figure 3] It is a flowchart showing the steps of the method in FIG. 2 in more detail. [Figure 4] It is a flowchart showing the method according to the second embodiment of the present invention. [Figure 5] It is a diagram showing the results achieved using the embodiments of the present invention. [Modes for carrying out the invention]

[0032] The description herein merely illustrates the methods of implementing the present invention, and the scope of the invention is not limited thereto. Any modifications or substitutions can be readily made by those skilled in the art. Accordingly, the scope of protection of the present invention should be subject to the scope of protection of the appended claims.

[0033] Figure 1 shows a flowchart of a conventional method of active learning. The method begins in step 5, where recorded samples are received. In the following step 10, the acquisition function value is updated for the newly received sample. In step 15, a check is performed to see if the acquisition function value for the new sample is among the top B samples. In other words, this step ranks the acquisition function values ​​of the samples in the dataset. If the new sample is among the top B samples, the sample is annotated and added to the training set (which includes the annotated data) in step 20.

[0034] The network is retrained in step 25. The retrained network may produce different results and may derive different acquisition functions. Therefore, the method iteratively continues step 10, updating the values ​​of the acquisition function for the (unlabeled) samples and determining the top B samples again. This active learning method can yield very good results. In particular, it is possible to iteratively prompt the user to label the data points that are most important to label. However, problems can arise if the original dataset is too large and updating the values ​​of the acquisition function and determining the top B samples in each iteration is computationally excessive.

[0035] Training deep neural networks requires large amounts of accurately annotated training data. In the labeling process, selecting the most informative training samples can reduce the labeling cost. This process is called active learning. However, this process can be computationally expensive. Embodiments of the present invention provide solutions that enable active learning using less computation and less disk space while maintaining, or in some cases exceeding, the accuracy of state-of-the-art active learning strategies. The proposed method can be used, for example, to improve automated annotation throughput related to autonomous driving, reduce annotation costs, and reduce space usage (irrelevant recordings can be removed).

[0036] Classical active learning methods typically select and label one sample at a time. However, training a deep neural network with one more sample requires significantly more computation, but may not result in a noticeable change in the model's accuracy. A solution to this problem is batch-mode AL, where instead of selecting only one sample at a time, a batch of samples is selected to annotate the network with the union of already labeled and newly labeled samples, and then retrain it. The fact that multiple samples can be selected at once makes it easy to see that the selected samples can become redundant in a sense, thus leading to a suboptimal solution. In recent years, various deep active learning strategies have been developed to address this problem. Deep active learning strategies can be divided into three groups: acquisition function-based, diversity-based, and hybrid strategies.

[0037] Acquisition function-based methods assign a scalar to each unlabeled sample, allowing the samples to be sorted by their values. These methods typically rely on Bayesian inference, which is not trivial for such models. Several techniques address this problem, such as Bayes-By-Backprop, ensemble learning, and Monte Carlo dropout. When using MC dropout for classification, the mean of the output distribution can be used as the true distribution, its entropy can be calculated, and a value can be given for sorting the samples. In regression, the deviation of the output distribution can be measured, and this value is used for sorting. These methods can also be extended to dense prediction tasks, including semantic segmentation or object detection. However, there is a mode collapse in acquisition function-based active learning, which means that these methods tend to create datasets with non-uniformly distributed label spaces. Also, selecting redundant data points can be a problem for these methods, and performance degrades as the batch size of newly labeled data points increases.

[0038] Diversity-based methods attempt to find inputs that represent diversity in an unlabeled pool. The most popular method is called CoreSet, which attempts to find a batch of samples such that, when added to the training set, the distance between an image in the unlabeled pool and its nearest image in the training set is minimized. This problem is NP-hard, and therefore approximations are necessary when selecting the desired batch.

[0039] Hybrid methods typically involve evaluating an acquisition function on unlabeled data points, sorting them, and then feeding the top N data points into a diversity sampling method, such as K-means or weighted K-means. N is usually the number of points to label, with a batch size of 10 times K.

[0040] One application of this invention is the proposal of a more sophisticated algorithm. This leads to increased computational needs and therefore slower active learning cycles. For example, evaluating MC dropout typically means running a forward pass of a given sample (with various dropout settings) 50 or more times to obtain a good estimate. For many unlabeled samples, this process alone can be enormously time-consuming. If a large amount of unlabeled data is recorded daily, pre-filtering these records can be very useful. This is beneficial not only because these samples may not be used for subsequent manual labeling and training, but also because they may still need to go through various transformations and processes that require time and computation. Even after transformation, unlabeled training samples can be removed using a sophisticated algorithm based on active learning.

[0041] A preferred embodiment is based on the hypothesis that the network becomes confident only once for each different input. This fact is highly unlikely to change through the active learning cycle.

[0042] One key concept of the embodiment is as follows: Given a pool of labeled data points and a pool of unlabeled data points, first, a neural network is trained, and then the acquisition function for all unlabeled samples is evaluated. Next, the best number of B from the data points to be labeled are selected and added to the labeled pool, and the network is retrained. Deep active learning is performed by repeatedly iterating these steps. According to the present invention, after the first training, evaluation, labeling, and retraining steps, the output of the acquisition function for each unlabeled data point is saved, and the dataset is subsampled into a so-called candidate pool based on these values. Sampling is performed in a manner in which the more confident the network is in a given sample, the less likely it is to be in the candidate pool. After the second training, the acquisition function is evaluated only for samples in the candidate pool, and updates are performed only on the values ​​of the acquisition function for these samples.

[0043] Subsequently, the partially updated value of the acquisition function is used to resample the candidate pool. Since the size of the candidate pool is appropriately chosen, this means that the time used to evaluate the acquisition function can be controlled in a non-naive manner at every step. The sampling effect not only saves time but also compensates for cases where a suboptimal point might be selected due to sampling. Sampling significantly reduces the probability of a severely redundant batch or an unbalanced label space.

[0044] The proposed algorithm was evaluated on two widely used AL benchmark datasets, MNIST and CIFAR10. A random but balanced set of samples was used to begin, and the size of the labeled dataset was gradually increased. The same starting samples, network architecture, training parameters, and acquisition function were used to evaluate this method. The only difference between the experiments was the size of the candidate pool, which varied between 2% and 100% of the original size of the unlabeled pool, compared to the conventional method using active learning 100%. The results showed that using the proposed algorithm not only resulted in faster iterations but also faster convergence of accuracy (as a function of the number of labeled samples). The experiments also showed that this algorithm can be used with various acquisition functions.

[0045] The input data provided to the method may include sensor data from an autonomous vehicle. Tasks for which this method is trained may include 3D object detection. Alternatively, it may be a set of tasks including, for example, point cloud segmentation, BEV segmentation, and future trajectory estimation.

[0046] A preferred embodiment may include several adjustable parameters, the optimal parameter values ​​of which may be determined, for example, using hyperparameter optimization.

[0047] In particular, the two adjustable parameters are the size of the candidate set, i.e., the number of candidate data points sampled, and the function that generates the probability distribution from the value of the acquisition function (AF). Number of sampled candidate data points The number of sample candidate data points can be considered an interpolation factor between conventional AL (Active Learning) and the proposed efficient AL solution. If the sample candidate data points include the entire unlabeled dataset, the algorithm is the same as conventional AL. The fewer the sampled candidate data points, the fewer samples with a computed acquisition function are evaluated. However, if the number of sampled candidate data points is extremely small, the algorithm will obviously fail to converge because the acquisition function values ​​are essentially not updated for most of the unlabeled dataset. Therefore, the number of sampled candidate data points is a function of available computational resources. If sufficient computational power is available, it is appropriate to choose a number of sampled candidate data points that is 15-30% of the size of the unlabeled pool; if computational power is more limited, 5-10% can be chosen. A number of sampled candidate data points exceeding 50% does not provide very good performance and requires more computation. The problem with choosing this value is that (experimental results suggest) sparser sampling may be advantageous in terms of performance. Therefore, determining this value is based on empirical experiments and available computational resources. Function to create a distribution from AL values Creating categorical distributions from unnormalized scalars is a frequently used method in ML and is applied to proposed algorithms for sampling unlabeled data. One of the most common methods is to use the Softmax function. However, if different AF values ​​have different ranges, a Softmax temperature T must be adopted to adjust the output distribution. In extreme cases, this would involve either selecting the top C samples (T=∞) or sampling the unlabeled dataset uniformly (T=0). For entropy as the acquisition function w, the observation is made that a basic T=1 is a good choice (and therefore requires less adjustment), but different T values ​​or different normalization functions (e.g., simply dividing by the sum of AF values) can also be used. Determining this function is based solely on empirical experimentation.

[0048] Other parts of the algorithm are based on conventional AL methods. For example, there is the option of performing Bayesian inference using Monte Carlo / Ensemble / Bayesian CNN. This can also be a design choice, but it is not within the scope of algorithmic novelty. However, it is beneficial to use MC dropout because an efficient method has been designed, as this is the most computationally efficient version of Bayesian inference.

[0049] Figure 2 is a further schematic flowchart of the method according to the present invention.

[0050] The method includes the first step 205, which involves receiving the recorded data as unlabeled data.

[0051] In step 210, the neural network is trained. This training may be based on labeled data and unlabeled data received in the previous step.

[0052] Next, in step 215, AF is calculated for all unlabeled data points. Then, in step 220, the top x% of unlabeled data points are annotated based on the acquisition function values ​​of the unlabeled data points.

[0053] In step 225, the iteration is executed N times until the termination condition is met in step 230.

[0054] Figure 3 shows the iterative steps of the method in Figure 2 in more detail.

[0055] In the first step, 305, the network is retrained. This involves training the neural network using labeled samples. (This step might use unlabeled data points if, for example, a semi-supervised learning method is used, but this is irrelevant in the case of active learning.)

[0056] In step 310, Sample Candidate Datapoints (C) are a set of candidate datapoints sampled from unlabeled datapoints (UL) based on the acquisition function value (AFV). In this step, the acquisition function value (AFV) is normalized to represent a probability distribution, and a subset of it is sampled (this is the so-called candidate set). The desired size of the candidate set, or in other words, the number of sampled candidate datapoints, is described earlier in this specification.

[0057] In step 315, the AFV is calculated and updated for C. The acquisition function (which can be any function) must be evaluated, and its corresponding sample values ​​must be updated.

[0058] In step 320, the lower y percent of the UL are removed based on their most recent AFV value. Preferably, this simply involves removing selected data points from that volume. The value of y can be chosen based on various motives, one of which is the number of unlabeled samples recorded each week. There is a finite amount of capacity to label (and train the network), which obviously means that if the dataset grows by 1% each week, then 0.5% of the entire dataset can also be removed. The number of samples to be remembered will still converge to infinity, but at a much slower rate.

[0059] Another reason to increase y might be that the network's desired performance approaches the defined goal. The better the performance achieved on a given task, the less likely new samples are to be useful, and therefore more unlabeled samples can be removed. However, the most influential reason might be the ability to label samples. If we can only afford to label a very small percentage of incoming samples, then obviously more unlabeled samples can be removed.

[0060] Additionally, you can define a constant to measure how many unlabeled samples you want to retain (for example, 100,000 unlabeled samples). The number of dropped samples can then be automatically calculated using not only the number of labeled samples but also the newly arriving unlabeled samples.

[0061] In step 325, the top x% of C are selected and labeled. This percentage may be based on the frequency at which the network should be trained, and this is a function of computational power (and possibly also a function of the economic budget available for labeling, along with how long samples are labeled, which is also based on the labeling budget).

[0062] In step 330, the labeled C is inserted into the labeled dataset L. This labels the selected samples and adds them to the labeled dataset.

[0063] The values ​​of the acquisition function are not updated for samples that were not selected for the candidate set in a given active learning iteration. This means that in the next active learning iteration, these values ​​will not be updated, but sampling will still be based on these values.

[0064] The acquisition function can be any function that may (but is not always required) use a trained (Bayesian) neural network and samples as input.

[0065] Figure 4 is a further flowchart of another method according to an embodiment of the present invention.

[0066] The method includes an initial step 410 to acquire the first set of data points. These are used in step 420 to perform the initial training of the neural network. The trained neural network is used in step 430 to calculate the value of the acquisition function for all sets of unlabeled data points. Data points from the initial set of data points can be added to the set of candidate data points.

[0067] This is followed by step 440, which is performed for all data points in the set of candidate data points. If a data point from the set of candidate data points is in the top T samples (in terms of the value of the acquisition function), the data point is annotated with a label (e.g., by the user), and the combination of data point and corresponding label is added to the training set.

[0068] The method continues, and in step 450, the network is retrained with the updated training set of labeled data points.

[0069] Next, in step 460, for all data points in the set of candidate data points, lower T data points (also referred to herein as samples) are determined and removed from the unlabeled dataset.

[0070] In step 470, candidate data points are sampled from the unlabeled dataset to obtain a set of candidate data points, or in other words, a candidate set.

[0071] For this candidate set, the value of the acquisition function is updated in step 480 (using the retrained network), and the method continues in an iterative manner in step 440.

[0072] Several experiments were conducted using the MNIST and CIFAR datasets. The methods were validated against actual records from applicant convoys.

[0073] Figure 5 shows the results of experiments conducted using the proposed method with various subsample sizes and random sampling as the AL strategy at the baseline. With subsampling using embodiments of the present invention, it can be observed that accuracy converges faster as the number of labeled samples increases compared to cases without subsampling or when random samples are selected for labeling.

Claims

1. A method for assisted training of a machine learning model for computer-assisted vehicle control, wherein the method is performed in an iterative manner. - Sampling one or more candidate data points from a set of unlabeled data points, - Calculate the acquisition function for the aforementioned candidate data points (110, 150), - With respect to a first data point in the set of unlabeled data points, if a first condition (115) relating to a first value of the acquisition function for the first data point is met, remove the first data point from the set of unlabeled data points (120), - With respect to a second data point in the set of candidate data points, if a second condition (125) relating to the second value of the acquisition function for the second data point is met, a label for the second data point is obtained and the second data point is added to the set of labeled data points along with the label (130), - Training the machine learning model using the aforementioned set of labeled data points (140) Methods that include...

2. The method according to claim 1, wherein the first condition (115) is satisfied when the first value of the acquisition function for the first data point is ranked among a first predetermined number of minimum acquisition values.

3. The method according to claim 1 or 2, wherein the second condition (125) is satisfied if the second value of the acquisition function for the second data point is among a second predetermined number of maximum acquisition values.

4. The method according to any one of claims 1 to 3, wherein the sampling is performed such that in one iteration, a predetermined share of data points from the set of unlabeled data points is sampled as candidate data points, preferably the predetermined share being between 3% and 30%, particularly between 5% and 20%.

5. The method according to any one of claims 1 to 4, wherein the sampling is performed such that the higher the confidence value of the acquisition function for a given data point from the set of unlabeled data points, the more likely the given data point is to be sampled as a candidate data point.

6. The method according to claim 5, wherein the sampling includes a step of determining a sampling probability based on the sampling temperature, preferably the sampling temperature being optimized in a hyperparameter optimization process.

7. The method according to any one of claims 1 to 6, wherein the machine learning model includes a neural network, in particular a convolutional neural network.

8. The method according to claim 7, wherein the vehicle is a fully or partially autonomous vehicle.

9. The method according to any one of claims 1 to 8, wherein the machine learning model is trained to detect objects in 3D data and / or predict trajectories in 3D data, preferably the 3D data includes a point cloud.

10. The method according to any one of claims 1 to 9, wherein the data points include sensor data, in particular sensor data from a stereo camera, RADAR and / or LIDAR.

11. The method according to any one of claims 1 to 10, wherein acquiring the label for the second data point includes outputting a data point acquisition command to the user and acquiring the label from the user.

12. The method according to any one of claims 1 to 10, wherein obtaining the label for the second data point includes outputting a data point acquisition command to a separate automated or semi-automated labeling system and obtaining the label from a separate loop.

13. A machine learning device configured to perform the method according to any one of claims 1 to 12.

14. A computer-readable storage medium for storing program code, wherein the program code includes instructions that, when executed by a processor, perform the method described in any one of claims 1 to 12.

Images