Method for determining distribution of training data set

Abstract

The inventive concept relates to a method for determining a distribution of a training data set. A computer-implemented method (100) of determining a distribution of training data sets for subsequent training of a machine learning model of an automatic driving system and other aspects thereof are disclosed. The method (100) comprises: providing (S102) a first dataset by selecting data samples from a second dataset of available training data based on a candidate distribution; training (S104) a machine learning model for the first data set; evaluating (S108) the machine learning model according to an evaluation criterion; and updating (S112) the candidate distribution in view of the evaluation result, thereby forming an updated candidate distribution. The inventive concept further relates to a method (200) of forming a training dataset for subsequent training of a machine learning model and other aspects thereof.

Description

Technical Field

[0001] The inventive concept relates to the field of autonomous vehicles. In particular, the present invention relates to a method and apparatus for determining the distribution of a training data set for subsequent training of a machine learning model for an autonomous driving system. Background Art

[0002] In recent years, with the development of technology, image capture and processing technologies have been widely used in different technical fields. In particular, vehicles produced today are typically equipped with some form of vision or perception system to ensure new functions. Moreover, an increasing number of modern vehicles have advanced driver assistance systems (ADAS) to improve vehicle safety and, more generally, road safety. ADAS - such as represented by adaptive cruise control (ACC), collision avoidance systems, forward collision warnings, lane support systems, etc. - are electronic systems that can assist the driver of a vehicle. Today, research and development are underway in many technical fields associated with both the ADAS and autonomous driving (AD) fields. ADAS and AD can also be collectively referred to by the common term autonomous driving system (ADS) corresponding to all different levels of automation, such as the driving automation levels (0 - 5) defined by, for example, SAE J3016.

[0003] While safety and performance are indispensable considerations for the development of ADS, the cost of the system also has a great impact, especially for online solutions implemented in vehicles. The computational footprint available for ADS is limited in terms of the hardware cost and the power consumption of the chips and algorithms used in operation. This in turn leads to limitations on the algorithms that can be deployed on such platforms. ADS or its perception system typically depends to some extent on machine learning models. Machine learning models, especially those that rely on deep neural networks, are typically associated with relatively high computational costs. Therefore, the computational footprint of such models will be limited in terms of the number of parameters (weights and biases) and possible architecture configurations (such as blocks, layers, etc.). As a further result, the limited computational footprint of the deployed model means limited learning ability, which in practice means that the best inference ability of the network must be achieved by training on a limited and restricted training data set. Given these limitations, a way to achieve the best possible performance is highly sought after. For models used in a vehicle environment, the problem is even further exacerbated because it needs to handle the huge variations in possible scenarios and the large number of objects that may appear in driving scenarios. This places high demands on the performance of the model. Therefore, improvements are needed in training and using machine learning models more effectively, especially in the field of autonomous driving. Summary of the Invention

[0004] The techniques disclosed herein are directed to alleviating, mitigating, or eliminating one or more of the above deficiencies and drawbacks in the prior art to address various problems related to the development of an autonomous driving system (ADS). In particular, the disclosed techniques provide a way to improve the efficiency of training and use of machine learning models for an ADS. More specifically, the inventors have implemented a new and improved way to determine a distribution for a training data set for subsequent training of a machine learning model. The distribution can help optimize the selection of training data from a limited training data set and how to balance between various types of training samples in order to improve model performance. The latter can be related to, for example, dealing with rare or other (for the model) challenging scenarios so that these scenarios are not overwhelmed by more common and simple data samples in the data set.

[0005] Given a set of available training data, the proposed techniques help optimize the performance of a machine learning model, but they can also help guide the data collection activities of a fleet such that a balanced mix of data samples corresponding to the determined distribution is provided, resulting in the collection of the best data.

[0006] Aspects and embodiments of the disclosed invention are defined below and in the appended independent and dependent claims.

[0007] According to a first aspect, there is provided a computer-implemented method for determining a distribution for a training data set for subsequent training of a machine learning model for an autonomous driving system. The method includes providing a first data set by selecting data samples from a second data set of available training data based on a candidate distribution. The method further includes training a machine learning model on the first data set. The method further includes evaluating the machine learning model according to an evaluation criterion. The method further includes updating the candidate distribution in view of the evaluation to form an updated candidate distribution. The updated candidate distribution can then be used as the determined distribution for a data set for subsequent training of the machine learning model.

[0008] The currently disclosed techniques provide an overall method for determining a distribution that can be optimized for an entire training data set rather than at the level of individual data samples. The current way of forming a training set can involve evaluating each data sample to classify them as "important" or "unimportant". A training data set can then be formed by selecting those data samples found to be important. However, simply collecting the most important data samples does not necessarily result in a well-distributed training data set. Instead, the proposed solution looks at the bigger picture by evaluating the distribution at the training data set level. The resulting distribution then enables a machine learning model subsequently trained with the training data set having the determined distribution to achieve better performance.

[0009] Further possible associated advantages may lie in that, since the training dataset can be optimized for a larger dataset of available data samples, the training process can be more efficient in terms of time and computational resources. More specifically, by sampling the larger dataset based on the distribution to form the training dataset, the number of training data samples required for training the machine learning model can be reduced. This means that the training process can be carried out in less time and with fewer computational resources.

[0010] Furthermore, the proposed solution allows for more efficient data collection, which leads to an improvement in the performance of the deployed machine learning model.

[0011] For the reasons given above, the present technology further facilitates in-vehicle learning, i.e., ultimately enabling the model to be trained online in a vehicle. This can be achieved thanks to the determined distribution, which can reduce the size of the training dataset while also improving the results of subsequent training of the model.

[0012] Moreover, the proposed solution can ensure a balance between data samples (such as between "rare" and more readily available data samples).

[0013] According to a second aspect, there is provided a computer program product comprising instructions which, when the program is executed by a computing device, cause the computing device to perform the method according to any embodiment of the first aspect. According to a third aspect, there is provided a (non-transitory) computer-readable storage medium. The non-transitory computer-readable storage medium stores one or more programs which are configured to be executed by one or more processors of a processing system, the one or more programs comprising instructions for performing the method according to any embodiment of the first aspect. Where applicable, any of the above-mentioned features and advantages of the first aspect also apply to the second and third aspects. For the sake of avoiding unnecessary repetition, reference is made to the above content.

[0014] According to a fourth aspect, there is provided a device for determining a distribution of a training dataset for subsequent training of a machine learning model for an autonomous driving system. The device comprises control circuitry. The control circuitry is configured to provide a first dataset by selecting data samples from a second dataset of available training data based on a candidate distribution. The control circuitry is further configured to train a machine learning model with respect to the first dataset. The control circuitry is further configured to evaluate the machine learning model according to an evaluation criterion. The control circuitry is further configured to update the candidate distribution in view of the evaluation, thereby forming an updated candidate distribution. Where applicable, any of the above-mentioned features and advantages of the previous aspects also apply to the fourth aspect. For the sake of avoiding unnecessary repetition, reference is made to the above content.

[0015] According to a fifth aspect, there is provided a method for forming a training data set for subsequent training of a machine learning model. The method includes obtaining a distribution of a training data set determined according to the method of any embodiment according to the first aspect. The method further includes forming a training data set based on the obtained distribution. In applicable cases, any of the above-mentioned features and advantages of the previous aspects also apply to the fifth aspect. To avoid unnecessary repetition, reference is made to the above.

[0016] According to a sixth aspect, there is provided a computer program product comprising instructions which, when executed by a computing device, cause the computing device to perform the method according to any embodiment of the fifth aspect. According to a seventh aspect, there is provided a (non-transitory) computer-readable storage medium. The non-transitory computer-readable storage medium stores one or more programs which are configured to be executed by one or more processors of a processing system, the one or more programs comprising instructions for performing the method according to any embodiment of the fifth aspect. In applicable cases, any of the above-mentioned features and advantages of the previous aspects also apply to the sixth aspect and the seventh aspect. To avoid unnecessary repetition, reference is made to the above.

[0017] According to an eighth aspect, there is provided an apparatus for forming a training data set for subsequent training of a machine learning model. The apparatus includes a control circuit. The control circuit is configured to obtain a distribution of a training data set determined according to the method of any embodiment according to the first aspect. The control circuit is further configured to form a training data set based on the obtained distribution. In applicable cases, any of the above-mentioned features and advantages of the previous aspects also apply to the eighth aspect. To avoid unnecessary repetition, reference is made to the above.

[0018] As used herein, the term "non-transitory" is intended to describe a computer-readable storage medium (or "memory") that excludes propagating electromagnetic signals, but is not intended to otherwise limit the type of physical computer-readable storage devices encompassed by the phrase computer-readable medium or memory. For example, the term "non-transitory computer-readable medium" or "tangible memory" is intended to encompass storage device types that include, for example, random access memory (RAM) which does not necessarily store information permanently. Program instructions and data stored in a non-transitory form on a tangible computer-accessible storage medium may further be transmitted via a transmission medium or a signal such as an electrical signal, an electromagnetic signal, or a digital signal, and the transmission medium or signal may be conveyed via a communication medium such as a network and / or a wireless link. Thus, as used herein, the term "non-transitory" is a limitation on the medium itself (i.e., tangible, rather than the signal), rather than a limitation on data storage persistence (e.g., RAM versus ROM).

[0019] The disclosed aspects and preferred embodiments can be appropriately combined with each other in any way obvious to a person of ordinary skill in the art, such that one or more features or embodiments related to one aspect can also be regarded as related to another aspect or an embodiment of another aspect.

[0020] Further embodiments are defined in the dependent claims. It should be emphasized that when used in this specification, the term "comprise / comprising" is used to specify the presence of the recited features, integers, steps or components. It does not exclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

[0021] These and other features and advantages of the disclosed technology will be further clarified below with reference to the embodiments described hereinafter. Brief Description of the Drawings

[0022] When combined with the accompanying drawings, the above aspects, features and advantages of the disclosed technology will be more fully understood by reference to the following illustrative and non - limiting detailed description of example embodiments of the present disclosure, wherein:

[0023] Figure 1 is a schematic flowchart representation of a method for determining the distribution of a training data set for subsequent training of a machine learning model for an autonomous driving system according to some embodiments.

[0024] Figure 2 is a schematic flowchart representation of a method for forming a training data set for subsequent training of a machine learning model according to some embodiments.

[0025] Figure 3 is a schematic illustration of an apparatus for determining the distribution of a training data set for subsequent training of a machine learning model for an autonomous driving system according to some embodiments.

[0026] Figure 4 is a schematic diagram of an apparatus for forming a training data set for subsequent training of a machine learning model according to some embodiments.

[0027] Figure 5 is a schematic illustration of a vehicle according to some embodiments.

[0028] Figure 6 is a schematic illustration of a system according to some embodiments. Detailed Description of Specific Embodiments

[0029] The present disclosure will now be described in detail with reference to the accompanying drawings, in which some example embodiments of the disclosed technology are shown. However, the disclosed technology may be embodied in other forms and should not be construed as limited to the disclosed example embodiments. The disclosed example embodiments are provided to fully convey the scope of the disclosed technology to those skilled in the art. Those skilled in the art will recognize that the steps, services, and functions explained herein can be implemented using separate hardware circuits, using software working in conjunction with a programmed microprocessor or a general-purpose computer, using one or more application-specific integrated circuits (ASICs), using one or more field-programmable gate arrays (FPGAs), and / or using one or more digital signal processors (DSPs).

[0030] It will also be recognized that when the present disclosure is described in the form of a method, it can also be embodied in an apparatus including one or more processors and one or more memories coupled to the one or more processors, wherein computer code is loaded to implement the method. For example, in some embodiments, the one or more memories may store one or more computer programs that, when executed by the one or more processors, cause the device to perform the steps, services, and functions disclosed herein.

[0031] It should also be understood that the terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting. It should be noted that the articles "a", "an", "the", and "said" as used in the specification and the appended claims are intended to mean that there is one or more elements, unless the context clearly indicates otherwise. Thus, for example, in some contexts, a reference to "a unit" or "the unit" may refer to more than one unit, etc. Further, the words "comprising", "including", and "containing" do not exclude other elements or steps. It should be emphasized that when used in this specification, the term "comprise / comprising" is used to specify the presence of the recited features, integers, steps, or components. It does not exclude the presence or addition of one or more other features, integers, steps, components, or groups thereof. The term "and / or" should be interpreted to mean "both" and each as an alternative.

[0032] It should also be understood that although terms such as first, second, etc. may be used herein to describe various elements or features, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the embodiments, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. The first element and the second element are both elements, but they are not the same element.

[0033] As used herein, the phrase "one or more" of a set of elements (such as "one or more of A, B, and C" or "at least one of A, B, and C") shall be construed in conjunctive logic or disjunctive logic. In other words, it may refer to all elements, one element, or a combination of two or more elements of a set of elements. For example, the phrase "one or more of A, B, and C" may be construed as A or B or C, A and B and C, A and B, B and C, or A and C.

[0034] Throughout this disclosure, reference is made to machine learning models (which may also be referred to as machine learning algorithms, networks, neural networks, deep learning networks, etc.). A machine learning model herein refers to any computational system or algorithm trained on data to make predictions or decisions, such as by learning patterns and relationships from training data and applying that knowledge to new input data.

[0035] A machine learning model may be a model configured to process any type of sensor data of a vehicle, such as image data, LIDAR data, RADAR data, etc. A machine learning model may be, for example, a perception model, an object detection model, an object classification model, an object tracking model, a lane estimation model, a free space estimation model, a trajectory prediction model, an obstacle avoidance model, a path planning model, a scene classification model, a traffic sign classification model, etc. Moreover, a machine learning model may encompass different architectures, including but not limited to convolutional neural networks (CNNs), recurrent neural networks (RNNs), and other existing or future alternative solutions.

[0036] The deployment of a machine learning model generally involves a training phase, in which the model learns from labeled or unlabeled training data to achieve accurate predictions during a subsequent inference phase. The training data (and the input data during inference) may be images or sequences of images, LIDAR data (i.e., point clouds), radar data, etc. In addition, the training data / input data may include a combination or fusion of one or more different data types. For example, the training data / input data may include both an image depicting an annotated object and a corresponding LIDAR point cloud including the same annotated object.

[0037] In some embodiments, appropriate software development machine learning code elements available to the public (such as code elements available in Python, TensorFlow, and Keras, or any other appropriate software development platform) may be used to implement machine learning algorithms in any appropriate manner known to those of ordinary skill in the art.

[0038] The following will refer to Figures 1 to 6 Describe a method for determining the distribution of a training data set for subsequent training of a machine learning model (sometimes referred to simply as a model) for an autonomous driving system and other related aspects thereof.

[0039] In making the training and use of machine learning models more efficient, it is common to attempt to optimize hyperparameters of the network such as the learning rate, dropout rate, etc. Further, there have been efforts to explore optimization of the model architecture (i.e., model structure) of the network through so-called neural architecture search (NAS), in which an optimal model architecture is found within a search space based on a search strategy and a performance estimation strategy.

[0040] However, as the inventors have realized, it is also important to consider optimization across the training datasets used to train the model. Thus, the presently disclosed techniques relate to how to select a training dataset in a way that optimizes or at least improves the performance of a machine learning model trained on the training dataset, particularly given the datasets that are already available. The quality and characteristics of the training dataset play a crucial role in the performance of the machine learning model. Simply using all available training data to train the model does not necessarily result in a model with optimal performance. In fact, there are several aspects that can help achieve a well-performing model when constructing the training dataset.

[0041] First, the training dataset should represent the overall distribution of the data that the model will encounter during deployment. If the training dataset is biased or unrepresentative, the model may perform poorly on unseen data. For the same reason, the training dataset should also be relevant to the task at hand. Including irrelevant or redundant samples can introduce noise and have a negative impact on model performance.

[0042] Second, the size of the training dataset should be large enough to capture the potential patterns and variations in the data. Insufficient data can lead to overfitting, in which the model memorizes the training examples rather than learning the underlying relationships. At the same time, the training dataset should be small enough so that it does not require excessive computational resources during training and does not overwhelm the model with unnecessary data.

[0043] Third, the training dataset should be diverse to cover the various scenarios, objects, and variations that exist in real-world applications. This helps the model generalize well to different situations.

[0044] Further, the quality of the data samples and their annotations (for supervised learning tasks) can also be crucial for model performance. The data samples should, for example, be free of noise to allow the model to extract as much knowledge as possible from the data samples. Also, the annotations should be accurate and consistent, as noisy or incorrect annotations can mislead the model during training.

[0045] Even further, as mentioned above, the balance of the training dataset is also an important aspect. For example, in a classification task, the distribution of classes in the dataset should be balanced. An unbalanced dataset (where the number of one class significantly exceeds the others) can lead to a biased model that performs well on the majority classes but poorly on the minority classes. In a more general sense, if not handled properly, data samples constituting rare or other tricky scenarios can easily get drowned in a large number of simple and easily obtainable data samples. Although they are rare, these samples can form a crucial part of achieving a well-performing model. This is especially important in the context of autonomous driving systems as these systems must be able to handle various scenarios and also perform well in rare scenarios.

[0046] The techniques disclosed in this paper provide a solution to this problem in a more automated way. It utilizes data distribution search, similar to the principle of NAS, as described above. Through the proposed techniques, given a set of already available training data samples and evaluation criteria, the best data distribution for the training dataset can be determined. The resulting distribution helps to optimize the performance of the machine learning model but can also help to guide the data collection activities of the fleet.

[0047] The currently disclosed techniques are at least partially based on the recognition that principles similar to those in NAS can be applied to achieve data distribution search through which the best data distribution for the training dataset can be determined.

[0048] In short, the proposed method for determining the distribution of the training dataset is based on data distribution search (DDS) which is given a search space (e.g., a dataset of already available training data samples). DDS explores this search space to find candidates for the best data distribution for the training dataset. The training dataset can then be formed by sampling the available dataset. Using this training set, the machine learning model can be trained and subsequently evaluated. Based on the evaluation, the candidate distribution can be updated (e.g., based on an optimization algorithm). These steps can be executed iteratively until the evaluation criteria and / or convergence criteria are met. After the loop is completed, the current distribution can be used to form the training dataset by sampling the given dataset or by collecting new data in view of the determined distribution. The proposed method will be explained in more detail below.

[0049] It should be noted that terms such as "optimal", "optimized", or similar should be understood as something that is made "as good as possible" within certain constraints. The constraints can be, for example, time constraints defined by evaluation criteria and / or convergence criteria. The constraints can further relate to computational resources or numerical approximations. As is known in the art, it can further be "optimal" or "optimized" in the sense that it is determined based on an optimization algorithm. "Optimal distribution" or "optimal model performance", etc. do not need to be a single best solution (i.e., global minimum / maximum), but can be a solution towards a single best solution.

[0050] Figure 1 FIG. 4 is a schematic flowchart representation of a method 100 for determining the distribution of a training dataset for subsequent training of a machine learning model for an autonomous driving system. In other words, the training dataset on which the machine learning model can subsequently be trained can be selected based on the determined distribution. Thus, method 100 can be a method for determining the distribution to be used for the training dataset. The method 100 described below can be performed offline (e.g., by a server). The server can also be referred to as a remote server, cloud server, central server, backend server, fleet server, or backend server. Once trained, the machine learning model can be deployed online in one or more vehicles of a fleet. In some embodiments, method 100 can be performed online, i.e., in the vehicle in which the model is deployed. In this case, the complexity and size limitations of the training dataset can be higher (compared to the offline method). However, this can still be achieved by the disclosed techniques. In some embodiments, some steps of method 100 can be performed by the server and some steps can be performed by the vehicle.

[0051] Below, the different steps of method 100 are described in more detail. Even though presented in a specific order, the steps of method 100 can be performed in any suitable order and multiple times. Thus, although Figure 1 a specific order of method steps may be shown, the order of the steps can be different from that depicted. Additionally, two or more steps can be performed simultaneously or partially simultaneously. Such variations will depend on the software and hardware systems selected as well as the choices of the designer. All such variations are within the scope of the present invention. Similarly, software implementations can use standard programming techniques based on rule-based logic and other logics to accomplish the various steps. Further variations of method 100 will be apparent from the present disclosure. The embodiments mentioned and described above are given only as examples and should not be limiting to the present invention. Other solutions, uses, purposes, and functions within the scope of the present invention claimed in the patent claims described below should be apparent to those skilled in the art.

[0052] Method 100 includes providing S102 a first data set by selecting data samples from a second data set of available training data based on a candidate distribution. In other words, the first data set can be formed by sampling the second data set according to the candidate distribution such that the first data set satisfies the candidate distribution.

[0053] As used herein, the term "distribution" or "data distribution" can be understood as the way in which the training data samples of a training data set (in this case) are distributed. For example, in a training data set for an object classification model, the distribution can indicate how many training data samples of each class are included in the training data set. In another example, the distribution can be defined by a series of distribution weights, where each distribution weight is assigned to the corresponding metadata associated with each data sample. Thus, the distribution weights can define which data samples with certain metadata are to be included in the training data set / to what extent they are included. The metadata can be, for example, classification metadata. Classification metadata can be, for example, different labels used to define the scenario or situation associated with the data sample. Classification metadata can be, for example, time of day, country, day: yes / no, night: yes / no, rain: yes / no, sun: yes / no, highway: yes / no, city driving: yes / no, depicting a bus: yes / no, depicting a child: yes / no, parking lot: yes / no, etc. The metadata can include binary data, such as yes / no. Alternatively or in combination, the metadata can be an indication of a continuous parameter such as the degree of rainfall. Such a continuous parameter can be represented as categorical (or discrete) data by a plurality of discrete values or ranges. It should be noted that the data distribution can also refer to many other aspects. Thus, the data distribution can be regarded as specifying which types of data samples should be included and to what extent these types of data samples should be included.

[0054] The candidate distribution should be regarded as a potential distribution to be used for subsequent training of a machine learning model. The principle of method 100 can then provide a way to update the candidate distribution until a desired distribution (e.g., in the sense that it can lead to a well-performing model) is obtained. In particular, method 100 can be iteratively executed (i.e., repeated) through multiple iterations (also called loops). The candidate distribution can then be iteratively updated until a desired distribution is obtained. Thus, the candidate distribution can be a distribution determined as a result of a previous iteration. Alternatively, the candidate distribution can be an initial distribution set as a starting point. How to iterate the method will be further explained below.

[0055] A second data set of available training data can be, for example, a large data set of training data samples that have already been collected. Thus, the second data set can be a training data set that includes data samples that have been collected and stored at a server. The data samples can be pre-collected by a fleet using on-vehicle sensors of vehicles. The data samples can also be obtained from other sources, such as data sets that are already available from a third party or data sets collected by other entities other than the fleet. Alternatively or in combination, the data samples can be synthetically generated data samples (e.g., by generating machine learning models, data augmentation, etc.). Then, the first data set is formed as a subset of the second data set by sampling the second data set according to a candidate distribution. Thus, the determined distribution provides an indication of how to sample (or select data samples from) a larger second data set of available training data. In other words, the best candidate distribution is searched for within the pre-collected data set (i.e., the second data set described above). In subsequent training of the machine learning model, a training data set (e.g., the first data set) selected from the second data set according to the determined distribution can be used instead of training on the entire second data set. This can result in a machine learning model with better performance and more efficient training.

[0056] Method 100 further includes training machine learning model S104 with respect to the first data set. Training machine learning model S104 can be performed using any suitable procedure that is readily understandable by those skilled in the art. Generally speaking, a machine learning model can be trained to make predictions while minimizing a loss function. Training machine learning model S104 can be performed, for example, by supervised, semi-supervised, or unsupervised learning. Thus, the presently disclosed techniques are not limited to any specific manner of training a machine learning model.

[0057] Method 100 further includes evaluating S108 the machine learning model according to evaluation criteria. Thus, evaluating S108 the machine learning model can provide information about the performance of the machine learning model after being trained on the first training dataset. The evaluation criteria can, for example, provide information indicating the performance of the machine learning model or any other metric. The evaluation criteria can include multiple sub-criteria. In other words, S108 the machine learning model can be evaluated according to a set of evaluation criteria. Each sub-criterion (i.e., each evaluation criterion of a set of evaluation criteria) can provide information indicating different performance aspects of the machine learning model. The evaluation criteria (or any of its sub-criteria) can, for example, include classification metrics such as accuracy, false positive / negative rate, precision, recall, log loss, F1 score, etc. Alternatively or in combination, the evaluation criteria (or any of its sub-criteria) can include regression test metrics such as coefficient of determination (or R-squared), mean squared error, mean absolute error, etc. In the case of iteratively executing Method 100, the evaluation criteria can include comparing the trained machine learning model with the machine learning model trained in a previous iteration. In other words, the performance of the trained machine learning model can be compared with the previously trained machine learning model of the previous iteration. The phrase "regression test metric" means in this context a metric obtained from a regression test procedure, where, including previous test cases, the previous version of the model has been tested on these test cases. For example, previous test cases on which the previous version of the model (i.e., trained with a previous candidate distribution) performed poorly can be run to see if the updated model (i.e., trained with an updated candidate distribution) performs better. A more specific way of performing evaluation S108 will be given below. In some embodiments, the evaluation step can be run in parallel on multiple models trained with different candidate distributions to evaluate multiple models at once. This can be particularly relevant for effectively reducing the time taken to find the next best distribution when using in-vehicle evaluation.

[0058] Method 100 further includes updating S112 the candidate distribution in view of the evaluation, thereby forming an updated candidate distribution. In other words, the candidate distribution can be updated in view of the evaluation criteria. S114 the candidate merging strategy can be updated in view of the evaluation such that the updated candidate merging strategy can potentially lead to better performance of the model. The phrase "in view of the evaluation" can be understood as being based on the satisfaction of the evaluation criteria. More specifically, in the case where the evaluation criteria are not satisfied, S112 the candidate distribution can be updated. The updated candidate distribution can then be used to repeat the subsequent loop of Method 100.

[0059] The candidate distribution can be updated by an optimization algorithm S112. In other words, an optimization algorithm can be used to determine the updated candidate distribution. In other words, the entire process of determining the distribution can be performed by an optimization algorithm, that is, selecting a candidate distribution from the search space, training the model, evaluating the trained model, and updating the candidate distribution based on the evaluation. The optimization algorithm can be, for example, random search, naive random search, grid search, stochastic optimization (such as genetic algorithms), Bayesian optimization, swarm optimization, or reinforcement learning.

[0060] If the evaluation criteria are met, method 100 can be terminated, and the candidate distribution can be stored as the determined distribution for subsequent training of the machine learning model. Thus, method 100 can include storing the candidate distribution as the determined distribution in response to meeting the evaluation criteria. In other words, in the case where the evaluation criteria are met, the candidate distribution can remain as it is. In other words, the updated candidate distribution can be selected as the current candidate distribution. This will be further explained below.

[0061] As described above, the steps of method 100 (denoted as S102 to S112) can be repeated for the updated candidate distribution until the evaluation criteria and / or convergence criteria of the trained machine learning model are met. In other words, method 100 can be repeated with the updated candidate distribution as the new candidate distribution for subsequent loops / iterations. Thus, the step of updating the candidate distribution S112 can be performed only when the evaluation criteria and / or convergence criteria are not met. Otherwise, the iteration can be terminated. By repeating these steps, the candidate distribution can be continuously refined until the evaluation criteria and / or convergence criteria are met. In other words, the proposed solution provides for updating the candidate distribution in a loop based on the evaluation criteria to continuously refine the candidate distribution, so that the performance of the model can be improved. Finally (i.e., after the iteration), given the available data of the second dataset, the best candidate distribution can be obtained. Moreover, the first dataset sampled from the second dataset in the last iteration can be used as the best subset of the second dataset for training the model. Similarly, it can be optimal given the available data of the second dataset. Therefore, this can be advantageous because by selecting the best subset of data from the second dataset of already available training data samples, the performance of the model can be improved in an efficient manner. Therefore, it may not be necessary to collect additional data samples. Moreover, as the second dataset of available data evolves, the principle of method 100 can be repeated over time. In other words, if new data is added to the second dataset, method 100 can be repeated multiple times in iteration to possibly find a new best data distribution.

[0062] In some embodiments, only an evaluation criterion can be used as a trigger to terminate the loop. In some embodiments, both the evaluation criterion and the convergence criterion can be used as triggers to terminate the loop. If both are satisfied, or if only one of the evaluation criterion and the convergence criterion is satisfied, the loop can then be terminated. In some embodiments, only the convergence criterion can be used as a trigger to terminate the loop.

[0063] The convergence criterion can be any suitable convergence criterion known in the art. For example, the convergence criterion can be a threshold for a loss function. In another example, the convergence criterion can be to monitor changes in model parameters (such as model weights). If the change in model parameters between iterations becomes small, it can be an indication of convergence. In still another example, the convergence criterion can be to monitor changes in the model performance on a validation set. If the change in model performance between iterations becomes small, it can also be an indication of convergence. It should be noted that the examples of the convergence criterion mentioned above and other possible criteria can be combined. Thus, the convergence criterion can include one or more convergence metrics. Each convergence metric can correspond to a different way of measuring convergence. When one or more convergence metrics reach the corresponding thresholds, the convergence criterion can then be satisfied.

[0064] Evaluating the S108 machine learning model can include determining S110 one or more evaluation metrics associated with the evaluation criterion. The S112 candidate distribution can then be updated based on the comparison of the one or more evaluation metrics with the corresponding thresholds. The evaluation metric can be regarded as a numerical value based on which the evaluation criterion can be formulated. The evaluation metric can correspond to different sub-criteria of the evaluation criterion mentioned above. When one or more evaluation metrics reach the corresponding thresholds, the evaluation criterion can be satisfied. By evaluating the trained machine learning model based on one or more evaluation metrics, the progress of the machine learning model over time (or rather, over iterations) can be evaluated. As method 100 traverses the space of possible candidate distributions, the directionality of the evaluation metrics can thus be observed. Based on this, the candidate distribution can thus be updated.

[0065] In some embodiments, method 100 further includes applying the S106 machine learning model to a validation data set. In other words, the step of evaluating the S108 machine learning model can include applying the S106 machine learning model to a validation data set. In other words, the machine learning model can be run on the validation data set. The evaluation criterion can then be an indication of the performance of the machine learning model for the validation data set. The validation data set can be selected as a pre-existing data set. The pre-existing data set can be stored, for example, in the server executing the method. The pre-existing data set can be, for example, a regulatory baseline data set, or a data set of a defined set of test scenarios on which the model should be evaluated to meet certain rules such as the so-called EURO-NCAP.

[0066] However, it should be recognized that method 100 or at least a portion of method 100 may be performed online in a vehicle. In response to verifying that a validation data sample satisfies one or more validation triggers, a validation data set may then be formed by obtaining the validation data sample and storing the validation data sample into the validation data set. The validation data sample may be obtained by being collected by on-vehicle sensors of the vehicle. The vehicle may then check whether the validation data sample satisfies one or more validation triggers. If so, the validation data sample may be stored into the validation data set. This may be repeated until validation. A certain number of validation data samples for each validation trigger or combination of validation triggers may be collected and stored until the validation data set satisfies the validation data set distribution (or such that the aggregated validation data set has a certain distribution across one or more validation triggers). If performed in a server, the validation data sample may be obtained from vehicles of a fleet that have already collected the validation data sample by on-vehicle sensors. Then, the machine learning model may apply S106 to the validation data samples that satisfy one or more validation triggers (or belong to the validation data set). Alternatively, the vehicle may store the validation data samples found to satisfy one or more validation triggers and / or transmit them to the server for forming the validation data set. It should be recognized that the check of whether the validation data sample satisfies one or more validation triggers (and / or the aggregation of the validation data set such that it has a certain distribution across one or more validation triggers) may also be performed in the server. Thus, the vehicle may transmit any data sample, or only the data samples that satisfy one or more validation triggers.

[0067] Moreover, a set of validation triggers may be distributed among vehicles of a fleet along with the trained machine learning model. A set of validation triggers should be regarded herein as conditions indicating which data samples are to be included in an evaluation. The validation triggers may be defined, for example, by metadata such as the above-mentioned metadata. In response to a vehicle encountering a scenario (i.e., a validation data sample) that satisfies the validation trigger, the vehicle may transmit the sensor data related to the scenario to the server. Alternatively or in combination, the trained machine learning model may evaluate (in the vehicle) in parallel with the vehicle's operating platform in this scenario (i.e., in a so-called shadow mode test). The evaluation metrics may be stored in the vehicle and transmitted to the server. The server may then (if needed) update the candidate distribution based on the evaluation metrics and later retrain the machine learning model for a new data set determined based on the updated candidate distribution.

[0068] It should be noted that, as explained above, the evaluation S108 of the machine learning model may be completed using both a pre-existing validation data set and validation data samples dynamically collected based on validation triggers.

[0069] The executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0070] Generally, a computer-accessible medium can include any tangible or non-transitory storage medium or a storage medium such as an electrical, magnetic, or optical medium (e.g., a disk or a CD / DVD-ROM coupled to a computer system via a bus). As used herein, the terms "tangible" and "non-transitory" are intended to describe computer-readable storage media (or "memory") that exclude propagating electromagnetic signals, but are not intended to otherwise limit the types of physical computer-readable storage devices encompassed by the phrase computer-readable medium or memory. For example, the term "non-transitory computer-readable medium" or "tangible memory" is intended to encompass storage device types that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in a tangible computer-accessible storage medium in non-transitory form can be further transmitted via a transmission medium or a signal such as an electrical signal, an electromagnetic signal, or a digital signal, and the transmission medium or signal can be conveyed via a communication medium such as a network and / or a wireless link.

[0071] Figure 2 is a schematic flowchart representation of a method 200 for forming a training data set for subsequent training of a machine learning model.

[0072] Below, different steps of the method 200 are described in more detail. Even though illustrated in a specific order, the steps of the method 200 can be executed in any suitable order and multiple times. Thus, although Figure 2 a specific order of method steps may be shown, the order of the steps can be different from that depicted. Additionally, two or more steps can be executed simultaneously or partially simultaneously. For example, an optional step denoted as S206 can be executed independently of other optional steps denoted as S208 and S210. Such variations will depend on the selected software and hardware systems and the choices of the designer. All such variations are within the scope of the present invention. Similarly, software implementations can use standard programming techniques based on rule-based logic and other logics to accomplish the various steps. Further variations of the method 200 will be apparent from the present disclosure. The embodiments mentioned and described above are given only as examples and should not be limiting to the present invention. Other solutions, uses, purposes, and functions within the scope of the present invention claimed in the patent claims described below should be apparent to those skilled in the art.

[0073] The method 200 includes obtaining S202 the distribution of the training data set determined according to the above-described method 100 in combination with Figure 1 In other words, the distribution that the training data set should satisfy can be obtained by performing the steps of the above-described method 100. In other words, the method 200 can include obtaining S202 the distribution for the training data set that has been determined according to the method 100 as described above.

[0074] Method 200 may further include forming the training data set S204 based on the obtained distribution. In other words, the training data set may be formed according to the distribution.

[0075] The training data set S204 may be formed by selecting S206 training data samples from an existing data set of training data samples based on the obtained distribution. In other words, forming the training data set S204 may include selecting S206 training data samples from an existing training data set according to the distribution, i.e., such that the training data set satisfies the distribution. This may be advantageous in cases where a large amount of training data is already available or where it is not possible to collect more data. Given an existing data set of available training data samples, using the obtained distribution to form the training data set may then help improve or even optimize the performance of a model subsequently trained on the formed training data set. The existing training data set may be the second data set as described above. Thus, the training data set formed by this method 200 may be the first data set sampled from the second data set during the final iteration of the loop (as described above). As determined according to the optimization process, the formed training data set may thus correspond to a subset of the second data set that results in the most performant model.

[0076] In some embodiments, the training data set S204 is formed by having a fleet collect S208 training data samples based on the distribution and storing S210 the training data samples as training data of the training data set. Thus, the data samples may be collected S208 and stored S210 based on the distribution in the sense that the training data set then satisfies the distribution. In other words, information indicating the distribution (e.g., a set of trigger conditions associated with the distribution) may be transmitted to the fleet, based on which the vehicles may collect relevant sensor data to be used as training data samples. After collecting the relevant training data samples, the vehicles may transmit the samples to the server. Thereby, guided data collection may be achieved, which utilizes the distribution determined according to the above principles and fleet insights. This may be advantageous in cases where there are not enough available training data samples. Moreover, the guided data collection defined by steps S208 and S210 as described above may be used in combination with selecting S206 training data samples from an existing data set. In particular, if some relevant data samples are missing or there is not enough relevant data in the existing data set, for example, the guided data collection may be used as a supplement to selecting training data samples from the existing data set. In other words, the guided data collection may be used to fill gaps in the existing data set in order to satisfy the distribution.

[0077] Method 200 may further include training a machine learning model on the formed training data set.

[0078] Throughout this disclosure, the term "obtain" should be construed broadly and encompass receiving, retrieving, collecting, acquiring, etc. directly and / or indirectly between two entities configured to communicate with each other or further with other external entities. However, in some embodiments, the term "obtain" should be construed as determining, deriving, forming, calculating, etc. In other words, obtaining a distribution may include receiving or collecting the distribution from another device or storage device. Alternatively, obtaining a distribution may include determining the distribution in accordance with the above-described method 100 in combination with Figure 1 the above.

[0079] The executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0080] Figure 3 is a schematic illustration of an apparatus 300 for determining a distribution of a training data set for subsequent training of a machine learning model for an autonomous driving system according to some embodiments. The apparatus 300 may be configured to perform the method 100 as described in combination with Figure 1 above.

[0081] The apparatus 300 as described herein refers to any computer system or general-purpose computing device. The apparatus 300 may be provided as part of a vehicle (such as the vehicle described below in combination with Figure 5 above). Alternatively, the apparatus 300 may be a server (also referred to as a remote server, cloud server, central server, back-end server, fleet server, or backend server) or a networked device configured to provide various computing services, data storage, processing capabilities, or resources to a client or user over a communication network. In the current context, the term "client" refers to a networked vehicle in a fleet (such as the vehicle 500 described below). Even though the apparatus 300 is illustrated herein as one apparatus, the apparatus 300 may be a distributed computing system formed by multiple different computing devices.

[0082] The apparatus 300 includes a control circuit 302. The control circuit 302 may physically include a single circuit device. Alternatively, the control circuit 302 may be distributed over several circuit devices.

[0083] As Figure 3 shown in the example of, the apparatus 300 may further include a transceiver 306 and a memory 308. The control circuit 302 is communicatively connected to the transceiver 306 and the memory 308. The control circuit 302 may include a data bus, and the control circuit 302 may communicate with the transceiver 306 and / or the memory 308 via the data bus.

[0084] The control circuit 302 can be configured to perform overall control of the functions and operations of the device 300. The control circuit 302 can include a processor 304 such as a central processing unit (CPU), a microcontroller, or a microprocessor. The processor is configured to execute program code stored in the memory 308 to perform the functions and operations of the device 300. The control circuit 302 is configured to perform the steps of the above-described method 100 as associated with Figure 1 These steps can be implemented in one or more functions stored in the memory 308.

[0085] The transceiver 306 is configured to enable the device 300 to communicate with other entities such as a vehicle or other servers. The transceiver 306 can both send data from the device 300 and receive data into the device 300.

[0086] The memory 308 can be a non-transitory computer-readable storage medium. The memory 308 can be one or more of a buffer, flash memory, hard disk drive, removable media, volatile memory, non-volatile memory, random access memory (RAM), or another suitable device. In a typical arrangement, the memory 308 can include non-volatile memory for long-term data storage and volatile memory that serves as the system memory for the device 300. The memory 308 can exchange data with the circuit 302 via a data bus. There can also be accompanying control lines and address buses between the memory 308 and the circuit 302.

[0087] The functions and operations of the device 300 can be implemented in the form of executable logic routines (e.g., lines of code, software programs, etc.) that are stored on a non-transitory computer-readable recording medium (e.g., the memory 308) of the device 300 and executed by the circuit 302 (e.g., using the processor 304). In other words, when it is stated that the circuit 302 is configured to perform a specific function, the processor 304 of the circuit 302 can be configured to execute a portion of the program code stored on the memory 308, where the stored portion of the program code corresponds to the specific function. Additionally, the functions and operations of the circuit 302 can be a stand-alone software application or form part of a software application that performs additional tasks related to the circuit 302. The described functions and operations can be regarded as methods such as the method 100 discussed above in connection with Figure 1 The method. Additionally, although the described functions and operations can be implemented in software, such functions can also be performed via dedicated hardware or firmware, or some combination of one or more of hardware, firmware, and / or software. In the following, the functions and operations of the device 300 are described.

[0088] The control circuit 302 is configured to provide a first data set by selecting data samples from a second data set of available training data based on a candidate distribution. This may be performed, for example, by executing the providing function 310.

[0089] The control circuit 302 is further configured to train a machine learning model with respect to the first data set. This may be performed, for example, by executing the training function 312.

[0090] The control circuit 302 is further configured to evaluate the machine learning model according to an evaluation criterion. This may be performed, for example, by executing the evaluation function 314.

[0091] The control circuit 302 is further configured to update the candidate distribution in view of the evaluation, thereby forming an updated candidate distribution. This may be performed, for example, by executing the updating function 316.

[0092] The control circuit 302 may further be configured to apply the machine learning model to a validation data set. This may be performed, for example, by executing the applying function 318. The evaluation criterion may then be an indication of the performance of the machine learning model with respect to the validation data set.

[0093] It should be noted that the principles, features, aspects, and advantages of the above-described method 100 in connection with Figure 1 also apply to the apparatus 300 as described herein. To avoid unnecessary repetition, reference is made to the above content.

[0094] Figure 4 is a schematic representation of an apparatus 400 for forming a training data set for subsequent training of a machine learning model according to some embodiments. The apparatus 400 may be configured to perform the method 200 as described in connection with Figure 2 described.

[0095] It should be noted that the principles and aspects of the above-described apparatus 300 in connection with Figure 3 also apply to the apparatus 400 as described herein. For example, the apparatus 400 includes a control circuit 402 having the same processor 404 as the above-described apparatus 300. Moreover, the apparatus 400 may include a transceiver 406 and a memory 408. These two apparatuses may be provided as two separate apparatuses or as a common apparatus. To avoid unnecessary repetition, reference is made to the above content. In the following, the focus will be changed to functions different from those of the above-described apparatus 300.

[0096] The control circuit 402 is configured to obtain the distribution of the training data set determined according to the above-described method 100 in connection with Figure 1 described. This may be performed, for example, by executing the obtaining function 410.

[0097] The control circuit 402 is further configured to form a training data set based on the obtained distribution. This can be performed, for example, by executing the forming function 412.

[0098] It should be noted that the principles, features, aspects, and advantages of the above-described method 100, as associated with Figure 2 are also applicable to the apparatus 400 as described herein. To avoid unnecessary repetition, reference is made to the above.

[0099] Figure 5 is a schematic illustration of a vehicle 500 according to some embodiments. The vehicle 500 is equipped with an autonomous driving system (ADS) 510. As used herein, a "vehicle" is any form of motorized conveyance. For example, the vehicle 500 can be any road vehicle such as, for example, an automobile, a motorcycle, a (cargo) truck, a bus, a smart bicycle, etc., as illustrated herein.

[0100] The vehicle 500 includes a plurality of elements that are typically found in an autonomous or semi-autonomous vehicle. It should be understood that the vehicle 500 can have Figure 5 any combination of the various elements shown in Figure 5 Moreover, in addition to the elements shown in Figure 5 the vehicle 500 can include further elements. Although the various elements are shown herein as being located inside the vehicle 500, one or more elements can be located outside the vehicle 500. Further, even though the various elements are depicted herein in certain arrangements, as will be readily understood by those skilled in the art, the various elements can be implemented in different arrangements. It should be further noted that the various elements can be communicatively connected to each other in any suitable manner. Figure 5 The vehicle 500 of Figure 5 should be regarded only as an illustrative example, as the elements of the vehicle 500 can be implemented in several different ways.

[0101] Vehicle 500 further includes a control system 502. The control system 502 is configured to perform overall control of the functions and operations of the vehicle 500. The system 502 includes a control circuit 504 and a memory 506. The control circuit 502 may physically include a single circuit device. Alternatively, the control circuit 502 may be distributed over several circuit devices. As an example, the control system 502 may share its control circuit 504 with other parts of the vehicle. The control circuit 502 may include one or more processors such as a central processing unit (CPU), a microcontroller, or a microprocessor. The one or more processors may be configured to execute program code stored in the memory 506 to perform the functions and operations of the vehicle 500. The processor may be or include any number of hardware components for performing data or signal processing or for executing computer code stored in the memory 506. In some embodiments, the control circuit 504 or some of its functions may be implemented on one or more so-called system-on-chips (SoCs). As an example, the ADS 510 may be implemented on an SoC. The memory 506 optionally includes high-speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 506 may include database components, object code components, script components, or any other type of information structure for supporting the various activities of this specification.

[0102] In the illustrated example, the memory 506 further stores map data 508. The map data 508 may be used, for example, by the ADS 510 of the vehicle 500 to perform the automatic functions of the vehicle 500. The map data 508 may include high-definition (HD) map data. It is contemplated that even though the memory 508 is illustrated as a separate element from the ADS 510, it may be provided as an integrated element of the ADS 510. In other words, according to some embodiments, any distributed or local memory device may be used for the implementation of the inventive concept. Similarly, the control circuit 504 may be distributed, for example, such that one or more processors of the control circuit 504 are provided as integrated elements of the ADS 510 or any other system of the vehicle 500. In other words, according to exemplary embodiments, any distributed or local control circuit device may be used for the implementation of the inventive concept.

[0103] Vehicle 500 further includes a sensor system 520. The sensor system 520 is configured to acquire sensor data regarding the vehicle itself or its surrounding environment. The sensor system 520 may, for example, include a Global Navigation Satellite System (GNSS) module 522 (such as GPS) configured to collect geographical location data of the vehicle 500. The sensor system 520 may further include one or more sensors 524. The one or more sensors 524 may be any type of in-vehicle sensors such as cameras, lidar and radar, ultrasonic sensors, gyroscopes, accelerometers, odometers, etc. It should be appreciated that the sensor system 520 may also provide the possibility of acquiring sensor data directly or via dedicated sensor control circuitry in the vehicle 500.

[0104] Vehicle 500 further includes a communication system 526. The communication system 526 is configured to communicate with external units such as other vehicles (i.e., via vehicle-to-vehicle (V2V) communication protocols), remote servers (e.g., the cloud server further shown below Figure 6 ), databases, or other external devices, i.e., vehicle-to-infrastructure (V2I) or vehicle-to-everything (V2X) communication protocols. The communication system 526 may communicate using one or more communication technologies. The communication system 526 may include one or more antennas. Cellular communication technologies may be used for remote communication such as to a remote server or a cloud computing system. Additionally, if the cellular communication technology used has low latency, it may also be used for V2V, V2I, or V2X communication. Examples of cellular radio technologies are GSM, GPRS, EDGE, LTE, 5G, 5G NR, etc., and also include future cellular solutions. However, in some solutions, short- to medium-range communication technologies such as wireless local area network (LAN) (e.g., IEEE 802.11-based solutions) may be used to communicate with other vehicles near the vehicle 500 or with local infrastructure elements. ETSI is researching cellular standards for vehicle communication, and 5G, for example, is considered a suitable solution due to its low latency and efficient handling of high bandwidth and communication channels.

[0105] The communication system 526 may further provide the possibility of sending outputs to a remote location (e.g., a remote operator or a control center) via one or more antennas. Moreover, the communication system 526 may further be configured to allow various elements of the vehicle 500 to communicate with each other. As an example, the communication system may provide local network setups such as CAN bus, I2C, Ethernet, fiber optic, etc. Local communication within the vehicle may also be of the wireless type with protocols such as WiFi, LoRa, Zigbee, Bluetooth, or similar short- to medium-range technologies.

[0106] Vehicle 500 further includes a control system 520. The control system 528 is configured to control the maneuverability of vehicle 500. The control system 528 includes a steering module 530 configured to control the heading of vehicle 500. The control system 528 further includes a throttle module 532 configured to control the actuation of the throttle of vehicle 500. The control system 528 further includes a brake module 534 configured to control the actuation of the brakes of vehicle 500. The various modules of the steering system 528 may receive manual inputs from the driver of vehicle 500 (i.e., from the steering wheel, accelerator pedal, and brake pedal respectively). However, the control system 528 may be communicatively coupled to the vehicle's ADS 510 to receive instructions on how the various modules should act. Thus, ADS 510 may control the maneuverability of vehicle 500.

[0107] As described above, vehicle 500 includes ADS 510. ADS 510 may be part of the vehicle's control system 502. ADS 510 is configured to perform the functions and operations of the automated functions of vehicle 500. ADS 510 may include a plurality of modules, where each module is responsible for a different function of ADS 510.

[0108] ADS 510 may include a positioning module 512 or positioning block / system. The positioning module 512 is configured to determine and / or monitor the geographical location and heading of vehicle 500 and may utilize data from the sensor system 520, such as data from the GNSS module 522. Alternatively or in combination, the positioning module 512 may utilize data from one or more sensors 524. The positioning system may alternatively be implemented as a real-time kinematic (RTK) GPS for increased accuracy.

[0109] ADS 510 may further include a perception module 514 or perception block / system. The perception module 514 may refer to any known module and / or function, such as those included in one or more electronic control modules and / or nodes of vehicle 500, that is adapted and / or configured to interpret sensed data related to the driving of vehicle 500 to identify, for example, obstacles, lanes, relevant signs, appropriate navigation paths, etc. The perception module 514 may thus be adapted to rely on multiple data sources and obtain inputs from multiple data sources such as automotive imaging, image processing, computer vision, and / or in-vehicle networking, in combination with sensor data from, for example, the sensor system 520.

[0110] An object detection model (or other type of machine learning model) for detecting and / or classifying objects in the surrounding environment of vehicle 500 can be part of ADS 510, or more specifically part of the perception module 514. Vehicle 500 is configured to perform the functions of method 100 to determine the distribution of a training data set for subsequent training of a machine learning model for ADS. These functions can be implemented in a separate computing device provided in the vehicle, such as the above-described device 300 in combination with Figure 3 The computing device can include a control circuit 302 configured to perform the steps of the above-described method 100 in combination with Figure 1 Alternatively, as will be readily understood by those skilled in the art, the functions can be distributed across one or more modules, systems, or components of vehicle 500. For example, the control circuit 504 of the control system 502 can be configured to perform the steps of method 100. Vehicle 500 can further be configured to perform the functions of method 200 to form a training data set for subsequent training of the above-described machine learning model in combination with Figure 2 Moreover, vehicle 500 can perform subsequent training of the machine learning model in the vehicle.

[0111] The localization module 512 and / or the perception module 514 can be communicatively connected to the sensor system 520 to receive sensor data from the sensor system 520. The localization module 512 and / or the perception module 514 can further transmit control instructions to the sensor system 520.

[0112] The ADS can further include a path planning module 516. The path planning module 516 is configured to determine a planned path for vehicle 500 based on the perception and location of the vehicle determined by the perception module 514 and the localization module 512, respectively. The planned path determined by the path planning module 516 can be sent to the maneuvering system 528 for execution.

[0113] The ADS can further include a decision and control module 518. The decision and control module 518 is configured to perform the control of ADS 510 and make decisions. For example, the decision and control module 518 can decide whether the planned path determined by the path planning module 516 should be executed.

[0114] It should be understood that parts of the described solution can be implemented in vehicle 500, in a system located outside the vehicle, or in a combination of inside and outside the vehicle; such as, for example, the so-called cloud solution in a server communicating with the vehicle, as further explained in combination with Figure 6 The different features and steps of the implementation can be combined in other combinations than the described steps. Further, the elements (i.e., systems and modules) of vehicle 500 can be implemented in different combinations than those described herein.

[0115] Figure 6 System 600 for ADS development is illustrated by way of example. More specifically, system 600 can be configured to implement the principles of the presently disclosed technology. System 600 should thus be regarded as a non-limiting example of an implementation of aspects of the present technology disclosed herein. For example, system 600 can be configured to perform the above-described method 100 and method 200 in connection with Figure 1 and Figure 2 respectively. Therefore, unless otherwise stated, any of the above features or principles in connection with Figure 1 and Figure 2 also apply to system 600 as described herein, and vice versa.

[0116] System 600 includes a server 602 (or remote server, cloud server, central server, back-end server, fleet server or back-end server), hereinafter referred to as remote server 602 or simply as server 602. The server can be configured to be any one of the above-described apparatus 300 and apparatus 400 in connection with Figure 3 and Figure 4 respectively. Therefore, any feature described in connection with apparatus 300 and apparatus 400 can also apply to server 602. As illustrated, server 602 can be provided in the cloud, i.e., a server implemented as a cloud. Advantageously, server 602 can perform more computationally intensive tasks, or manage and coordinate a fleet (described further below), such as aggregating data from different vehicles of the fleet or distributing data thereto.

[0117] System 600 further includes one or more vehicles 604a to 604c, also referred to as fleet 604a to 604c. One or more vehicles 604a to 604c can be the above-described vehicles in connection with Figure 5 respectively. Therefore, one or more vehicles 604a to 604c can be configured to perform method 100 as described in connection with Figure 1 respectively. Vehicles 604a to 604c can further be configured to perform method 200 as described in connection with Figure 2 respectively.

[0118] One or more vehicles 604a through 604c are communicatively connected to a remote server 602 for sending and / or receiving data 606 between the vehicles and the server. One or more vehicles 604a through 604c may be further communicatively connected to each other. The data 606 may be any type of data such as communication signals or sensor data. More specifically, the data 606 may include training data samples acquired by the vehicles 604a through 604c, or any information indicating a determined distribution for a training data set, a data set according to the distribution, a (re)trained machine learning model, triggering conditions for collecting which data samples, etc. The communication may be performed by any suitable wireless communication protocol. The wireless communication protocol may be a long-range communication protocol such as cellular communication technologies (e.g., GSM, GPRS, EDGE, LTE, 5G, 5G NR, etc.) or a medium-short range communication protocol such as a solution based on wireless local area network (LAN) (e.g., IEEE 802.11). The server 602 includes suitable memory and control circuitry (e.g., one or more processors or processing circuits) and one or more other components such as a data interface and a transceiver. The server 602 may also include software modules or other components such that the control circuitry may be configured to execute machine-readable instructions loaded from the memory to implement steps of a method to be performed.

[0119] By way of example, Figure 6 the fleet illustrated in includes three vehicles, a first vehicle 604a, a second vehicle 604b, and a third vehicle 604c. However, the system 600 may include any number of vehicles 604a through 604c.

[0120] In the following, some ways of implementing the principles of the presently disclosed technology within the system 600 will be given. However, it should be recognized that these examples should not be considered limiting, as there may be several different ways of implementation depending on the specific implementation. For further details regarding different steps, refer to the above Figure 1 and Figure 2 so as to avoid excessive repetition.

[0121] Generally, the steps of method 100 for determining the distribution of the training dataset can be performed locally within vehicles 604a to 604c. Alternatively or in combination, the steps of method 100 can be performed in server 602. If performed in the vehicles of the fleet, the vehicles can then send the determined distribution to server 602. As described above, part of the step of evaluating the machine learning model in S108 includes forming a validation dataset by sampling an existing dataset or by data collection. In the latter case, server 602 can distribute trigger conditions to the fleet for collecting relevant data. In some embodiments, the step S104 of training the machine learning model for the first dataset can be performed by server 602. The server can then transmit the trained machine learning model to one or more vehicles in the fleet. One or more vehicles can then perform the step of evaluating the machine learning model in S108 (e.g., through so-called shadow mode testing). The evaluation results can then be transmitted back to server 602 for potential update S112 of the candidate distribution.

[0122] Similarly, the steps of method 200 for forming the training dataset can be performed by server 602 and / or the vehicles in the fleet. In the former case, the server can transmit instructions (e.g., trigger conditions) to the fleet indicating which data samples to collect. In other words, server 602 can transmit data indicating the determined distribution to the vehicles. When a vehicle encounters a new data sample (e.g., a new scenario), the vehicle can check whether it meets the trigger condition (or whether it belongs to the distribution). If so, the vehicle can transmit the data sample to the server.

[0123] The above processes of system 600 should be understood as non-limiting examples for enhancing understanding of the presently disclosed technology. Further variations are obvious from the present disclosure and can be readily implemented by those skilled in the art.

[0124] The present invention has been presented with reference to specific embodiments. However, other embodiments are possible and within the scope of the present invention. Within the scope of the present invention, method steps for performing the method by hardware or software different from the above can be provided. Thus, according to an exemplary embodiment, a non-transitory computer-readable storage medium storing one or more programs is provided, the one or more programs being configured to be executed by one or more processors of a vehicle control system, the one or more programs including instructions for performing the method according to any of the above embodiments. Alternatively, according to another exemplary embodiment, a cloud computing system can be configured to execute any of the methods presented herein. The cloud computing system can include distributed cloud computing resources that jointly execute the methods presented herein under the control of one or more computer program products.

[0125] It should be noted that any reference signs do not limit the scope of the claims, and the present invention may be implemented at least in part by both hardware and software, and the same hardware item may represent several "devices" or "units".

Claims

1. A computer-implemented method (100) for determining a distribution of a training data set for subsequent training of a machine learning model for an autonomous driving system, the method (100) comprising: providing (S102) a first data set by selecting data samples from a second data set of available training data based on the candidate distribution; Training (S104) the machine learning model for the first data set; evaluating (S108) the machine learning model according to evaluation criteria; and The candidate distribution is updated ( S112 ) in view of the evaluation, thereby forming an updated candidate distribution.

2. The method (100) according to claim 1, wherein: The steps of the method (100) are repeated for the updated candidate distribution until the evaluation criteria and / or convergence criteria of the trained machine learning model are met.

3. The method (100) according to claim 1, wherein: The candidate distribution is updated by an optimization algorithm (S112).

4. The method (100) according to any one of claims 1 to 3, wherein: evaluating (S108) the machine learning model comprises determining (S110) one or more evaluation metrics associated with the evaluation criteria; and The candidate distribution is updated based on a comparison of the one or more evaluation metrics with corresponding thresholds ( S112 ).

5. The method (100) according to claim 4, wherein: The evaluation criterion is met when the one or more evaluation metrics reach the corresponding threshold value.

6. The method (100) according to claim 1, further comprising applying (S106) the machine learning model to a validation dataset, and in, The evaluation criterion is an indication of the performance of the machine learning model on the validation dataset.

7. The method (100) according to claim 6, wherein: The validation dataset is formed by the following steps: Get a sample of validation data, and In response to the verification data sample satisfying one or more verification triggers, the verification data sample is stored in the verification data set.

8. A computer program product comprising instructions which, when executed by a computing device, cause the computing device to perform the method (100) according to claim 1.

9. An apparatus (300) for determining a distribution of a training data set for subsequent training of a machine learning model for an autonomous driving system, the apparatus (300) comprising a control circuit (302), the control circuit (302) being configured to: providing a first data set by selecting data samples from a second data set of available training data based on a candidate distribution; Training the machine learning model on the first data set; evaluating the machine learning model according to an evaluation criterion; as well as The candidate distribution is updated in view of the evaluation, thereby forming an updated candidate distribution.

10. The device (300) according to claim 9, wherein: The control circuit (302) is further configured to apply the machine learning model to a validation dataset, and Wherein, the evaluation criterion is an indication of the performance of the machine learning model on the validation dataset.

11. A method (200) of forming a training data set for subsequent training of a machine learning model, the method (200) comprising: Obtaining (S202) the distribution of the training data set determined according to the method (100) according to claim 1; as well as The training data set is formed ( S204 ) based on the obtained distribution.

12. The method (200) of claim 11, wherein: The training data set is formed ( S204 ) by selecting ( S206 ) training data samples from a data set of existing training data samples based on the obtained distribution.

13. The method (200) according to claim 11 or 12, wherein: The training data set is formed by the following steps (S204): The fleet collects (S208) training data samples based on the distribution, and The training data samples are stored (S210) as training data of the training data set.

14. A computer program product comprising instructions which, when executed by a computing device, cause the computing device to perform the method (200) according to claim 11.

15. An apparatus (400) for forming a training data set for subsequent training of a machine learning model, the apparatus (400) comprising a control circuit (402), the control circuit (402) being configured to: Obtaining the distribution of the training data set determined according to the method (100) according to claim 1; and The training data set is formed based on the obtained distribution.

Images