Computer-implemented method of providing perceptual models for annotations of training data

Abstract

The invention relates to a computer-implemented method of providing a perceptual model for training annotations of data. The method includes training a base model using a first training dataset to predict a trajectory of the vehicle based on a sequence of sensor data, where the sequence of sensor data includes sensor data relating to an environment of the vehicle, and where the first training dataset includes a second training dataset. The first training data set comprises sensor data sequences and information indicating driving tracks associated with the corresponding sensor data sequences; forming an offline perceptual model by adding a task-specific layer to the trained base model, where the task-specific layer is configured to perform a perceptual task of the offline perceptual model; and fine tuning the offline perceptual model using a second set of training data to perform a perceptual task, where the second set of training data includes sensor data annotated for the perceptual task. The invention further relates to a method for annotating data for use in subsequent training of an online perceptual model and an apparatus therefor.

Description

Technical Field

[0001] The present inventive concept relates to the field of autonomous vehicles. In particular, the present inventive concept relates to methods and apparatus for annotation of training data used in training of a perception model. Background Art

[0002] With the development of technology in recent years, image capture and processing technology has become widely used in different technical fields. In particular, vehicles produced today are usually equipped with some form of visual or perception system for realizing new functions. In addition, more and more modern vehicles have advanced driver assistance systems (ADAS) for improving vehicle safety and more general road safety. ADAS, which can be represented by adaptive cruise control (ACC), collision avoidance system, forward collision warning, lane support system, etc., is an electronic system that can help the driver of the vehicle. Today, research and development are being carried out in many technical fields associated with both ADAS and autonomous driving (AD). ADAS and AD can also be mentioned under the general term autonomous driving system (ADS) corresponding to all different automation levels defined, for example, by the SAE J3016 level (0-5) of driving automation.

[0003] Some of the functionality of these systems can be implemented using simple rule-based techniques. However, to handle the complexity of real-world driving scenarios that include changing road conditions, unpredictability of human or non-human behavior, and rapidly changing environments, the use of machine learning models has been shown to enhance the safety, capabilities, and performance of ADS. Machine learning models, such as deep learning models or neural networks, are particularly useful as part of a perception system of an ADS for, for example, detecting, determining, or tracking objects in the vehicle's surroundings.

[0004] Solving the perception tasks necessary to achieve autonomous driving with deep learning algorithms requires large amounts of labeled training data. Such datasets need to cover any conceivable scenario that may occur while driving. Collecting data is a relatively easy task. However, annotating data to make it useful for training machine learning models is many orders of magnitude more expensive because it typically requires human involvement. These problems only get worse when turning to spatiotemporal models of sequence data that require annotation, bringing a new dimension to the annotation cost. Therefore, an important goal in the development of AD is to find ways to accomplish this task in an automated manner. The inventive concept provides techniques for obtaining high-fidelity annotations in a more automated manner that can eliminate or significantly reduce the need for human involvement. Summary of the invention

[0005] The technology disclosed herein seeks to mitigate, alleviate or eliminate one or more of the above-mentioned defects and shortcomings in the prior art to solve various problems related to obtaining annotated training data. Recent advances in large language models have demonstrated the fact that deep learning is most powerful when there is no clear limit on the scale of the model or the size of its input data set. Demonstrations of this are large language models such as generative pre-trained transformers (transformers), or GPTs for short. The inventors have realized that these aspects can also be used in other fields, such as the field of autonomous driving development for annotation of data. The currently disclosed technology is at least partially based on training offline models with easily collected data to be able to annotate training data, which can then be used to train online (or production) models used in vehicles equipped with autonomous driving systems ADS. By increasing the availability of high-fidelity annotated data, subsequent training of online models can be improved because they can learn from more data.

[0006] As mentioned above, data collection is several orders of magnitude cheaper than annotation. To this end, the currently disclosed techniques exploit data that does not need to be explicitly labeled. The basic problem then becomes defining an objective function for the model that enables it to exploit large amounts of unlabeled data while establishing an understanding of the world around the vehicle that can be used to solve relevant AD tasks such as object or lane tracking. The proposed techniques thus allow one to train an offline automatically annotated model that is only limited by, for example, the amount of raw data collected by a test vehicle or fleet and the available computational resources for training, rather than the resources available for human annotation. The objective function proposed for the offline perception model for subsequent annotation of the training data is chosen in this article to be the problem of predicting the trajectory of a vehicle based on a sensor data sequence that includes sensor data related to the vehicle's surroundings.

[0007] Various aspects and embodiments of the disclosed invention are defined in the following and accompanying independent and dependent claims.

[0008] According to a first aspect, a computer-implemented method for providing an offline perception model for subsequent annotation of training data is provided. The training data can in turn be used for training an online perception model. The method includes training a base model using a first training data set to predict a trajectory of a vehicle based on a sensor data sequence. The sensor data sequence includes sensor data related to the surrounding environment of the vehicle. The first training data set includes the sensor data sequence and information indicating a driving trajectory associated with the corresponding sensor data sequence. The method further includes forming an offline perception model by adding a task-specific layer to the trained base model. The task-specific layer is configured to perform a perception task of the offline perception model. The method further includes fine-tuning the offline perception model using a second training data set to perform the perception task. The second training data set includes sensor data annotated for the perception task.

[0009] According to a second aspect, a computer program product comprising instructions is provided, and when a computing device executes the program, the instructions cause the computing device to perform a method according to any embodiment of the first aspect. According to an alternative embodiment of the second aspect, a (non-transitory) computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a processing system, and the one or more programs include instructions for performing the method according to any embodiment of the first aspect. When applicable, any of the above-mentioned features and advantages of the first aspect also apply to the second aspect. In order to avoid unnecessary repetition, reference is made to the above.

[0010] According to a third aspect, a device for providing an offline perception model for subsequent annotation of training data is provided. The training data can be used for training an online perception model. The device includes a control circuit. The control circuit is configured to train a base model using a first training data set to predict a trajectory of a vehicle based on a sensor data sequence. The sensor data sequence includes sensor data related to the surrounding environment of the vehicle. The first training data set includes a sensor data sequence and information indicating a driving trajectory associated with the corresponding sensor data sequence. The control circuit is further configured to form an offline perception model by adding a task-specific layer to the trained base model. The task-specific layer is configured to perform a perception task of the offline perception model. The control circuit is further configured to fine-tune the offline perception model using a second training data set to perform a perception task. The second training data set includes sensor data annotated for the perception task. When applicable, any of the above-mentioned features and advantages of other aspects also apply to this third aspect. In order to avoid unnecessary repetition, refer to the above.

[0011] According to a fourth aspect, a computer-implemented method for annotating data used in subsequent training of an online perception model is provided. The online perception model is configured to perform perception tasks of a vehicle equipped with an autonomous driving system. The method includes obtaining sensor data related to a physical environment. The method further includes determining a perception output by inputting the obtained sensor data into an offline perception model provided by a method according to any embodiment of the first aspect. The method further includes storing the sensor data together with the perception output as annotation data for subsequent training of the online perception model. When applicable, any of the above-mentioned features and advantages of other aspects also apply to this fourth aspect. In order to avoid unnecessary repetition, reference is made to the above.

[0012] According to a fifth aspect, a computer program product comprising instructions is provided, and when a computing device executes the program, the instructions cause the computing device to perform a method according to any embodiment of the fourth aspect. According to an alternative embodiment of the fifth aspect, a (non-transitory) computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a processing system, and the one or more programs include instructions for performing the method according to any embodiment of the fourth aspect. When applicable, any of the above-mentioned features and advantages of other aspects also apply to this fifth aspect. In order to avoid unnecessary repetition, refer to the above.

[0013] According to a sixth aspect, a device for annotating data used in subsequent training of an online perception model is provided. The online perception model is configured to perform perception tasks of a vehicle equipped with an autonomous driving system. The device includes a control circuit. The control circuit is configured to obtain sensor data related to the physical environment. The control circuit is further configured to determine the perception output by inputting the obtained sensor data into an offline perception model provided by the method according to any embodiment of the first aspect. The control circuit is further configured to store the sensor data together with the perception output as annotation data for subsequent training of the online perception model. When applicable, any of the above-mentioned features and advantages of other aspects also apply to this sixth aspect. In order to avoid unnecessary repetition, refer to the above.

[0014] As used herein, the term "non-transitory" is intended to describe computer-readable storage media (or "memory") that do not include propagating electromagnetic signals, but is not intended to otherwise limit the types of physical computer-readable storage devices encompassed by the term computer-readable media or memory. For example, the term "non-transitory computer-readable medium" or "tangible memory" is intended to encompass types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in a non-transitory form on a tangible computer-accessible storage medium may be further transmitted via a transmission medium or signals such as electrical, electromagnetic, or digital signals that may be transmitted via a communication medium such as a network and / or wireless link. Therefore, as used herein, the term "non-transitory" is a limitation on the medium itself (i.e., tangible, not a signal), not a limitation on the persistence of data storage (e.g., RAM vs. ROM).

[0015] The disclosed aspects and preferred embodiments can be appropriately combined with each other in any manner obvious to those skilled in the art, so that one or more features or embodiments disclosed in relation to one aspect can also be considered to be disclosed in relation to another aspect or an embodiment of another aspect. In addition, any advantages mentioned in conjunction with one aspect also apply to other aspects when applicable.

[0016] As previously mentioned, an advantage of the currently disclosed technology may be that it enables annotation of training data with less human involvement. In addition to a relatively small training data set (a second training data set compared to the first training data set) of annotated data, an offline perception model for annotating data can be provided using a large amount of data that does not require explicit annotation. Therefore, one can simply drive a vehicle with an appropriate sensor setup and collect a sequence of sensor data while also recording the driving trajectory. The recorded driving trajectory can then be used as an automatic annotation for training a base model. Therefore, high-fidelity annotations of the training data that can be used in the training of an online perception model can then be generated in a more time-saving manner, and in quantity far greater than previously feasible with today's technology. This in turn can improve subsequent training of online perception models because larger and more general training data sets can be used. The effect of imitation learning using implicitly annotated data and / or base models makes it easy to collect these large amounts of data with little effort. In addition, by deploying the perception model for data annotation as an offline model, it is allowed to push the model and data set scale to new heights. The effect of the above aspects may be that an offline perception model can learn the complex task of predicting vehicle trajectories based on a sequence of sensor data provided as input to the model. Solving this task on a sufficiently large and diverse dataset can produce models that can understand the dynamics of the environment and understand the temporal evolution of the scene. Therefore, the offline perception models provided by the present technology can be more powerful (e.g., in terms of capabilities, accuracy, and general performance) than any automatic annotation model currently known to be trained only on a limited human-labeled dataset. The provided offline perception models are also more powerful than models trained with contrastive losses or classification tasks, because these objectives do not directly replace the relevant AD tasks that the online perception models are intended to perform. In other words, the offline perception models are able to learn to minimize contrastive losses without requiring a general understanding of object tracking or other perception tasks.

[0017] In summary, a very powerful base model can be trained on any sequence of sensor data for which driving trajectories are stored. Such a model will contain a strong representation of the perceived environment around the car, which in turn can be used to train a strong offline perception model, which can then be used for automatic annotation.

[0018] Further embodiments are defined in the dependent claims. It should be emphasized that when used in this specification, the term "comprising" is used to specify the presence of stated features, integers, steps or components. It does not exclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

[0019] These and other features and advantages of the disclosed technology will be further elucidated below with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above-described aspects, features, and advantages of the disclosed technology will be more fully understood by referring to the following illustrative and non-limiting detailed description of example embodiments of the present disclosure when taken in conjunction with the accompanying drawings, in which:

[0021] Figure 1 is a schematic flow chart representation of a method for providing an offline perceptual model for subsequent annotation of training data according to some embodiments.

[0022] Figure 2 is a schematic flow chart representation of a method for annotating data for use in subsequent training of an online perception model according to some embodiments.

[0023] Figure 3 is a schematic diagram of an apparatus for providing an offline perceptual model for subsequent annotation of training data according to some embodiments.

[0024] Figure 4 is a schematic diagram of an apparatus for annotating data for use in subsequent training of an online perception model according to some embodiments.

[0025] Figure 5 is a schematic diagram of a vehicle according to some embodiments.

[0026] Figure 6 is a schematic diagram of a system according to some embodiments. DETAILED DESCRIPTION

[0027] The present disclosure will now be described in detail with reference to the accompanying drawings, in which some example embodiments of the disclosed technology are shown. However, the disclosed technology may be embodied in other forms and should not be construed as limited to the disclosed example embodiments. The disclosed example embodiments are provided to fully convey the scope of the disclosed technology to those skilled in the art. Those skilled in the art will understand that the steps, services, and functions described herein may be implemented using separate hardware circuits, using software that works in conjunction with a programmed microprocessor or general-purpose computer, using one or more application-specific integrated circuits (ASICs), using one or more field programmable gate arrays (FPGAs), and / or using one or more digital signal processors (DSPs).

[0028] It will also be understood that when the present disclosure is described in terms of methods, it may also be embodied in a device comprising one or more processors, one or more memories coupled to the one or more processors, wherein computer code is loaded to implement the methods. For example, in some embodiments, the one or more memories may store one or more computer programs that, when executed by the one or more processors, cause the device to perform the steps, services, and functions disclosed herein.

[0029] It should also be understood that the terms used in this article are only used for the purpose of describing specific embodiments, and are not used to limit. It should be noted that, as used in the specification and the appended claims, unless the context clearly stipulates otherwise, the articles "one", "an", "the" and "said" are intended to indicate the presence of one or more elements. Thus, for example, in some contexts, reference to "a unit" or "the unit" may refer to more than one unit, and so on. In addition, the words "include", "comprise" do not exclude other elements or steps. It should be emphasized that when used in this specification, the term "include" is used to specify the presence of stated features, wholes, steps or parts. It does not exclude the presence or addition of one or more other features, wholes, steps, parts or groups thereof. The term "and / or" should be interpreted as meaning that "both" and each is optional.

[0030] It will also be understood that although the terms first, second, etc. may be used to describe various elements or features in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the embodiment, the first element may be referred to as the second element, and similarly, the second element may be referred to as the first element. Both the first element and the second element are elements, but they are not the same element.

[0031] As used herein, the phrase "one or more of a group of elements" (such as "one or more of A, B, and C" or "at least one of A, B, and C") should be interpreted as conjunction or disjunction. In other words, it can refer to all elements, one element, or a combination of two or more elements in a group of elements. For example, the phrase "one or more of A, B, and C" can be interpreted as A or B or C, A and B and C, A and B, B and C, or A and C.

[0032] Throughout this disclosure, reference is made to a machine learning model (or just a "model"). The term "machine learning model" herein means any form of machine learning algorithm, such as deep learning models and neural networks, that is capable of learning and adapting from input data and subsequently making predictions, decisions, or classifications based on new data. In general, as used herein, a machine learning model may be any neural network-based model that operates on sensor data of an autonomous vehicle. In the following, the terms "perception model" and "base model" will be used to distinguish between more specific types of machine learning models or for the purpose of defining a machine learning model.

[0033] The deployment of machine learning models typically involves a training phase during which the model learns from labeled or unlabeled training data to achieve accurate predictions during a subsequent inference phase. The training data (and input data during inference) can be, for example, images or image sequences, lidar data (i.e., point clouds), radar data, etc. In addition, the training / input data can include a combination or fusion of one or more different data types. The training / input data can, for example, include both an image depicting a scene of the vehicle's surroundings and a corresponding lidar point cloud of the same scene.

[0034] In some embodiments, the machine learning model may be implemented in any manner known to be suitable to a person of ordinary skill in the art using publicly available suitable software development machine learning code elements (e.g., such as code elements available in Pytorch, TensorFlow, and Keras or any other suitable software development platform).

[0035] The term "perception model" herein refers to a computing system or algorithm designed to perceive or interpret an environment depicted in sensor data such as digital images, video frames, lidar data, radar data, ultrasonic data, or other types of visual data related to the driving of a vehicle. In other words, the perception model can be designed to detect, locate, determine, and / or identify instances of specific objects within sensor data, lanes, relevant signs, appropriate navigation paths, etc. Therefore, the perception model can be configured to perform the perception tasks of the vehicle's autonomous driving system ADS. Examples of perception tasks include, but are not limited to, object detection, object classification, lane estimation, and free space estimation. More specifically, the machine learning model can be an object detection model, an object classification model, a lane estimation model, or a free space estimation model. The perception model can use a combination of advanced technologies from computer vision, machine learning, and pattern recognition to analyze visual sensor data, and output, for example, a bounding box or region of interest around an object of interest present in an input image. The perception model can be further configured to classify what type of object is detected. The perception model can include different architectures, including, but not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), converters, and other existing or future alternatives.

[0036] The output of the perception model can be used for downstream tasks such as trajectory prediction, path planning, or emergency braking systems or used by downstream systems of the ADS. In some embodiments, the perception model can be part of an end-to-end model that is configured (in contrast to the above) to perform perception tasks and downstream tasks. For example, a machine learning model can perform trajectory prediction or path planning directly based on sensor data.

[0037] In addition, in the following, a distinction will be made between "online" and "offline" perception models. This distinction should be understood to refer to how or where (i.e., in what setting) the perception model is deployed. An online perception model should be interpreted as a perception model deployed at the edge (i.e., directly on an edge device), in this case, a vehicle equipped with an ADS. Therefore, an online perception model can be regarded as a production model deployed in a vehicle. In other words, the calculation of the online perception model is performed locally, close to the data source. In contrast, an offline perception model refers to a perception model deployed at, for example, a remote server (also referred to as a cloud server, a central server, a background server, a fleet server, or a backend server). In addition, in contrast to online perception models, offline perception models are not used in production scenarios (i.e., real-time scenarios). Instead, offline perception models can be run independently during the development process. Due to their computing environment, online perception models typically have a simpler or less computationally intensive architecture than offline perception models because they run at the edge with limited memory and processing power. On the other hand, an offline perception model can be a larger and more complex model because it can be deployed on a server with more available computing resources. In fact, there may be no explicit limit on the size of the offline perception model, as it can even be parallelized across several computing devices.

[0038] The term "base model" herein refers to a machine learning model that can serve as the basis or core architecture on which a more specialized or customized machine learning model is built. The base model is also often referred to as a "basic model" or "general model". The base model is usually pre-trained on a large number of large and diverse data sets (usually through self-supervision or semi-supervised learning) to learn the general patterns, features or representations of the data. These learned representations can be used for various specific tasks such as natural language processing, image recognition, recommendation systems and various other applications and fine-tuned for them. The typical characteristics of the base model are that their model size is large, including a large number of trainable parameters. The model size and complexity contribute to its ability to capture complex patterns and representations from a large number of data sets. As a non-limiting example, the base model can be built on a convolutional neural network (CNN) such as a residual neural network (commonly referred to as ResNet in the art) and one or more converter models. For example, images captured by one or more cameras of a vehicle can be fed to a CNN to encode them. Alternatively, a visual converter can be used. Then, the lidar point cloud and / or radar scan corresponding to the surrounding environment depicted in the image can be encoded by a CNN or a different model. The encoded images, lidar point clouds and / or radar scans can be fed to a converter model capable of building a unified abstract representation of the surrounding environment. The converter model can further consider the encoded sensor data of the previous time instance of the sensor data sequence or the sensor data itself. As a non-limiting example, the so-called BEVFormer (proposed by Li et al.) can be used. Then, before providing the output of the base model, the unified abstract representation can be further processed by the above converter model or another converter model. In short, it is possible to encode the sensor data using an arbitrarily large model (e.g., CNN). Then, one or more converter models or any size can be used to interpret the encoded sequence of sensor data. Training such a base model can be done end-to-end. In other words, the entire model can be trained simultaneously as a whole. It goes without saying that the above examples of base model structures are only regarded as non-limiting examples, because many replacements are also possible as those skilled in the art will readily appreciate.

[0039] Essentially, base models enable transfer learning approaches where knowledge gained from one domain or task can be transferred and adapted to improve performance in another domain or task. The concept of base models plays a crucial role in the efficiency and effectiveness of machine learning systems, enabling faster development and improved performance across a range of applications by reusing learned features and representations.

[0040] In the present disclosure, the base model is a self-trajectory model. The wording "self-trajectory model" means in this article a machine learning model that is trained to perform the task of predicting the trajectory of the self-vehicle based on the sensor data sequence captured while the vehicle is driving. In other words, predict the trajectory of the vehicle that has captured the sensor data sequence. In some embodiments, the base model can be trained to predict the driving trajectory of the vehicle. In other words, based on the sensor data sequence captured while the vehicle is driving, the self-trajectory model can retrospectively predict the driving trajectory of the vehicle. In some embodiments, the base model can be trained to predict the future trajectory of the vehicle. In other words, based on the sensor data sequence captured while the vehicle is driving, the self-trajectory model can predict the trajectory along which the vehicle is expected to travel in a subsequent time series. For example, the sensor data sequence may include sensor data of the vehicle within a first time period T1. Then, the base model can predict the trajectory of the vehicle within a second time period T2 after the first time period T1. A time period may also refer to a (time) sequence or multiple subsequent time instances. By learning any one (or both) of the tasks of predicting the driving trajectory or predicting the future trajectory, the base model can gain an understanding of the dynamics of the physical world, such as how different objects behave and how this is reflected in the sensor data. The task of predicting the future trajectory of a vehicle is a particularly challenging task that, when solved on a sufficiently large dataset, can provide machine learning models that have learned a complex understanding of the behavior and characteristics of all objects on the road, as well as the road itself and the surrounding environment. Therefore, predicting future trajectories can be advantageous because it enables high performance in subsequent annotation of sensor data. In order to be able to predict future trajectories in a wide variety of situations (and to some extent also to predict driving trajectories), the model must implicitly solve most of the perception tasks present in an autonomous vehicle. For example, for trajectories that always stay on the road, the model needs to know where the road is, for trajectories that do not hover between lanes, the model must detect lanes, for trajectories that avoid all obstacles, the model must learn to detect objects, and so on. More details about the online and offline perception models, the base models, and their training will become apparent through the following detailed description.

[0041] Figure 1 1 is a schematic flow chart representation of a computer-implemented method 100 for providing an offline perceptual model for subsequent annotation of training data. The offline perceptual model may also be referred to as an annotation model. The training data may in turn be used for subsequent training of an online perceptual model. The method 100 may be as follows: Figure 3The method 100 is performed by the device 300 described. More generally, the method 100 can be performed by any suitable computing device, such as in an offline server setting. Advantageously, the server is a device with more available computing resources than the vehicle equipped with the ADS. This can facilitate the deployment of more computationally intensive offline perception models. An online perception model trained on annotated training data can be deployed in the vehicle instead.

[0042] The term "annotation" as used herein refers to the process of adding some form of metadata or labels to data to make it understandable and usable for machine learning algorithms. In this case, metadata can be used to enrich sensor data to make it useful for training and evaluating machine learning models. This can include associating labels for determining, for example, objects in an image or determining bounding boxes or assigning segmentation data. Therefore, the term "labeling" or "labeling" can be considered a subset of data annotation. More specifically, it refers to the process of assigning one or more labels or categories to data instances (e.g., sensor data). For example, in image classification, labeling involves labeling images with their respective categories (e.g., cat, dog, or car).

[0043] Below, the different steps of method 100 are described in more detail. Even though illustrated in a particular order, the steps of method 100 may be performed in any suitable order and multiple times. Figure 1 A specific order of method steps can be shown, but the order of steps may be different from the order depicted. In addition, two or more steps may be performed simultaneously or partially concurrently. This variation will depend on the selected software and hardware systems and the designer's choice. All of these variations are within the scope of the present invention. Similarly, software implementation can be accomplished using standard programming techniques with rule-based logic and other logic to accomplish the various steps. Further variations of method 100 will become apparent from this disclosure. The embodiments mentioned and described herein are given as examples only and should not be limitations of the present invention. Other solutions, uses, objectives, and functions within the scope of the present invention claimed in the patent claims described below should be apparent to those skilled in the art.

[0044] Method 100 includes training S102 a base model using a first training data set to predict the trajectory of a vehicle based on a sensor data sequence. The predicted trajectory may be a driving trajectory of a vehicle. The predicted trajectory may be a future trajectory of a vehicle. The predicted trajectory may be both a driving trajectory and a future trajectory of a vehicle. The sensor data sequence includes sensor data related to the surrounding environment of the vehicle. In other words, the base model may be trained to predict the trajectory of an ego vehicle. Therefore, the predicted trajectory may be referred to as an ego trajectory. In addition, the sensor data sequence may be captured by an onboard sensor of an ego vehicle whose trajectory is to be predicted. More specifically, the sensor data sequence may include sensor data related to the surrounding environment of the vehicle at multiple subsequent time instances. Therefore, the sensor data sequence may include sensor data collected by an onboard sensor of the vehicle at multiple subsequent time instances t, ..., t+n, where n is any positive integer greater than 0. The sensor data sequence may also be referred to as a sensor data stream or spatiotemporal sensor data. Spatiotemporal sensor data herein means sensor data captured in space and time by an onboard sensor of a vehicle. As previously described, the base model may be trained to determine the driving trajectory of a vehicle. Thus, the base model can predict the driving trajectory of the vehicle for which sensor data was collected during time instances t to t+n. Alternatively or in combination, the base model can be trained to predict the future trajectory of the vehicle. Thus, the base model can predict the future trajectory of the vehicle during time instances t+n+1 to t+m, where m is a positive integer greater than n+1. In some embodiments, the base model can predict the trajectory of the vehicle during both time instances that at least partially overlap with the sensor data sequence and time instances that do not overlap with the sensor data sequence (i.e., at least one future time instance).

[0045] The wording "predict" as in "predicting a trajectory" should be interpreted herein as determining a driving trajectory or future trajectory of a vehicle. In one example, a sequence of sensor data may include an image at each time instance of a sequence of time instances, wherein each image depicts at least a portion of a surrounding environment from a viewpoint of the vehicle along its trajectory at the corresponding time instance. Thus, the trajectory of the vehicle may be predicted or determined based on how the depicted surrounding environment changes over the sequence.

[0046] The first training data set includes a sensor data sequence and information indicating a driving trajectory associated with the corresponding sensor data sequence. In other words, the first training data set may include multiple training samples. Each training sample includes a sensor data sequence and associated information indicating the driving trajectory of the vehicle. The information indicating the driving trajectory may include data related to the position of the vehicle at multiple subsequent time instances. The data related to the position of the vehicle may be, for example, GPS data of the vehicle at subsequent time instances. Alternatively or in combination, the data related to the position of the vehicle may be determined based on the odometer data of the vehicle or any other type of motion data of the vehicle. Alternatively or in combination, the data related to the position of the vehicle may be obtained by the inertial navigation system of the vehicle. Therefore, the information indicating the driving trajectory may include data related to the motion of the vehicle at multiple subsequent time instances. As an example, the information indicating the driving trajectory may include the geographic location (or other type of motion data) of the vehicle at each time instance t to t+n for which the sensor data sequence is given. In some embodiments, the information indicating the driving trajectory may further include the geographic location (or other type of motion data) of the vehicle at multiple time instances (i.e., t+n+1 to t+m) after the sensor data sequence. In other words, the driving trajectory may be on the sensor data sequence or in the future. In some embodiments, the information indicating the driving trajectory may include only the geographic location (or other type of motion data) of the vehicle at multiple time instances (i.e., t+n+1 to t+m) after the sensor data sequence. This will be further described below. Therefore, the wording "associated" as in the information indicating the driving trajectory "associated with the corresponding sensor data sequence" can be interpreted as indicating the driving trajectory during the same time period as the sensor data sequence, during the time period after the sensor data sequence, or during the time period that at least partially overlaps with the sensor data sequence and multiple subsequent time instances.

[0047] It will be appreciated that the information indicative of the driving trajectory of the vehicle can be formatted in any suitable manner, depending on how the base model and its output are constructed. More specifically, the base model can be trained to predict or determine the trajectory of the vehicle based on a sequence of sensor data input to the model that is in the same or similar format as the information indicative of the driving trajectory available in the first training data set. Thus, the information indicative of the driving trajectory can serve as real data. If the base model is trained to predict the future trajectory of the vehicle, a portion of the information indicative of the driving trajectory can serve as an input to the model together with the sequence of sensor data.

[0048] By collecting sensor data related to the surroundings of the vehicle and sensor data that can serve as information indicative of the driving trajectory of the vehicle (or from which such information can be obtained), the first training data set can be easily collected (or recorded) during a series of time instances. Thus, a large amount of data can be collected in a simple manner.

[0049] Now give an illustrative example of how to collect training samples of the first training data set. This example also illustrates the structure of the input to the basic model and what it can be trained to do. Sensor data related to the surrounding environment and driving trajectory of the vehicle can be recorded during the time period T. The time period T can be divided into a first time period T1 and a second time period T2. The second time period T2 is later than the first time period T2 in time. Then, the sensor data sequence referred to as the input to the basic model can be formed by the sensor data corresponding to the first time period. In other words, the sensor data corresponding to the second time period may not be given to the basic model. Optionally, the driving trajectory recorded during the first sub-period T1 can also be fed to the basic model as an input. Then, the basic model can undertake the task of predicting the driving trajectory of the vehicle during the second time period T2. Thus, the second time period T2 corresponds to the "future" as described above. Therefore, the driving trajectory of the vehicle in the second sub-period recorded can be used as the real data of the basic model. In the case where the basic model predicts the driving trajectory, the driving trajectory of the first time period T1 recorded can also be used as the real data. Therefore, the information indicative of the driving trajectory associated with the sensor data sequence may refer to the recorded driving trajectory of the first time period T1 , the second time period T2 , or both the first time period and the second time period.

[0050] Thus, when training the base model, information indicative of the driving trajectory can be used as real data and / or as input to the base model. In the former case, the base model can be fed with a sequence of sensor data and output a predicted driving trajectory for the vehicle. A comparison of the model's output with the real data can then serve as a basis for learning and improving the base model in terms of predicting the vehicle's trajectory. In the latter case, the driving trajectory can provide the base model with further information about the vehicle's past behavior, which the base model can use to make better predictions.

[0051] As described herein, the first training data set can be interpreted as an implicitly annotated data set. The wording "implicit annotation" in this article means a low-level annotation that can be obtained without any human or automatic annotation process. In other words, the sensor data sequence can be implicitly annotated in the sense that it is annotated without having to manually annotate the sensor data sequence, process the sensor data sequence through a machine learning model, or analyze the content of the sensor data sequence in any other way. Instead, the sensor data sequence is associated with information indicating a driving trajectory that can be collected at the same time as the sensor data sequence. In some aspects, the first training data set can be regarded as an unannotated training data set because no formal labels are associated with the training samples. Instead, the supervisory signal comes from the data itself, rather than from an external source such as a human or other automatic annotation process. This allows the first training data set to be much larger than the training data set that is feasible for an explicitly annotated training data set (e.g., the second training data set described below).

[0052] By using a first training dataset as described above, the collection of a large amount of training data can be done with little effort. Using implicitly annotated data means that there is virtually no limit to how much training data can be collected for training the base model, since if there is a fleet of vehicles, it is only limited by the storage and transmission of data between the vehicles and the central server.

[0053] The training S102 of the basic model can be performed by so-called imitation learning. This can be because the basic model is trained to predict (or learn) the behavior of the driver. Imitation learning herein refers to a machine learning paradigm in which the model learns by observing and imitating the actions of humans or other demonstrators. In essence, imitation learning can be described as a process that allows an actor (herein, the basic model) to learn from the behavior of one person or another actor to replicate the behavior in a similar scene. Therefore, in this case, the demonstrator can be the driver of the vehicle or the automatic driving system of the vehicle. Demonstration can be recorded by collecting information on driving trajectories. Therefore, the sensor data sequence related to the surrounding environment corresponds to the scene in which the actor (i.e., the basic model) will work. Then, the basic model is trained on the sensor data sequence and the information indicating the driving trajectory to predict the action of the demonstrator.

[0054] From another perspective, the training S102 of the base model can be regarded as a form of supervised learning in which implicitly annotated training samples are used, or as (future prediction) self-supervised learning (because not all parts of the data are shown to the model, but the model is responsible for predicting the data). Then, this form of supervised learning should be distinguished from traditional supervised learning using explicitly annotated training samples. In traditional supervised learning, the model is trained on a labeled data set, where each input is paired with a corresponding target label. In traditional unsupervised learning, the model is trained only on an unlabeled data set. However, in the training of the base model, the implicit annotations of the input data are generated without any human supervision. In this case, the generated supervisory signal corresponds to a recorded trajectory (i.e., a form of sensor data collected by a vehicle) that can be used as an implicit label of a sensor data sequence (i.e., another form of sensor data collected by the same vehicle). Using this form of training, the base model can be advantageous in its ability to utilize unlabeled (e.g., implicitly labeled) data that is generally easier to obtain than labeled (e.g., explicitly labeled) data, while still enjoying at least some benefits of having a supervisory signal associated with the data.

[0055] The surroundings of the vehicle can be understood as the general area (of the physical world) around the vehicle in which objects (e.g. other road users, landmarks, obstacles, etc.) can be detected and determined by the vehicle sensors (radar sensors, lidar sensors, cameras, etc.) (i.e. within the sensor range of the vehicle). The sensor data are related to the surroundings in the sense that the sensor data reflect one or more properties of the surroundings, e.g. by depicting one or more objects in the surroundings.

[0056] The sensor data (sequence of sensor data) may be collected by onboard sensors of a vehicle equipped with an ADS. Thus, the sensor data may be related to the surroundings of the vehicle. The sensor data may include one or more of image data, lidar data, radar data, or ultrasonic data. The image data may be, for example, one or more images or image frames. Furthermore, the one or more images may depict different parts of the surroundings. The lidar data may be a point cloud of the surroundings. By using only one type of sensor data, the base model may be trained to better understand the world from that point of view. By using more than one type of sensor data, the base model may also learn how different sensor data types relate to each other.

[0057] The sensor data may be raw sensor data. Alternatively, the sensor data may be processed or fused sensor data of two or more different types of sensor data.

[0058] The method 100 further includes forming S104 an offline perception model by adding a task-specific layer to the trained base model. The task-specific layer is configured to perform a perception task of the offline perception model. The task-specific layer may include one or more sublayers required to perform the perception task. The perception task may be one of object detection, object classification, object tracking, lane estimation, free space estimation, trajectory prediction, obstacle avoidance, scene classification, and traffic sign classification.

[0059] The step of forming S104 offline perception model can be regarded as converting the trained base model into a task-specific model. In addition to adding task-specific layers, the structure of the base model can be further modified to accommodate this. For example, the output layer of the base model can be removed and replaced by a task-specific layer. In addition, the input layer of the base model can be modified or replaced. For example, if the base model and the offline perception model will take different types of data as input, this may be the case. As an example, the base model can take both sensor data sequences and driving trajectories as input (when its task is to predict future trajectories), while the perception model can only take sensor data as input. After adding a task-specific layer suitable for predicting according to the expected perception task, as described below, the base model can be frozen before subsequently fine-tuning the model. Methods for converting a trained base model into a specific task model (also known as transfer learning) are considered to be well known in the art. In this case, any suitable method can be used.

[0060] The method 100 further includes fine-tuning S106 the offline perception model using a second training data set to perform the perception task. The second training data set includes training data annotated for the perception task. Fine-tuning the offline perception model allows the base model that has been trained and then modified to adapt to the perception task. Fine-tuning may involve training a portion of the offline perception model, such as a task-specific layer. Before doing so, the base model that is part of the offline perception model can be frozen so that its trainable parameters do not change during the fine-tuning process. Thus, fine-tuning of the offline perception model allows learning the trainable parameters (e.g., model weights) of the task-specific layer. Alternatively, the entire offline perception model can be trained during fine-tuning. In other words, one or more trainable parameters of the base model can be updated during fine-tuning of the offline perception model.

[0061] Compared to the first training dataset, which may be an implicitly annotated dataset, the second training dataset may be an explicitly annotated dataset. In other words, the second training dataset may include training samples with explicit labels determined, for example, by a human or automatic annotation program. Since the offline perception model includes an already trained base model, the second training dataset may be several orders of magnitude smaller than the first training dataset, while still being able to achieve higher accuracy and performance compared to a perception model trained only on the second dataset. In other words, the second training dataset may be several orders of magnitude smaller than the first training dataset. Fine-tuning S106 of the offline perception model may be performed by supervised learning. More specifically, fine-tuning S106 of the offline perception model may be performed by supervised learning using explicitly annotated data.

[0062] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0063] In general, a computer-accessible medium may include any tangible or non-transitory storage medium or memory medium, such as an electronic, magnetic, or optical medium, for example, a disk or CD / DVD-ROM connected to a computer system via a bus. As used herein, the terms "tangible" and "non-transitory" are intended to describe a computer-readable storage medium (or "memory") that does not include propagating electromagnetic signals, but are not intended to otherwise limit the types of physical computer-readable storage devices encompassed by the term computer-readable medium or memory. For example, the term "non-transitory computer-readable medium" or "tangible memory" is intended to cover types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in a non-transitory form on a tangible computer-accessible storage medium may be further transmitted by a transmission medium or a signal, such as an electrical, electromagnetic, or digital signal, that may be transmitted via a communication medium such as a network and / or a wireless link.

[0064] Figure 2 1 is a schematic flow chart representation of a computer-implemented method 200 for annotating data for use in subsequent training of an online perception model according to some embodiments. The online perception model is configured to perform a perception task for a vehicle equipped with an autonomous driving system. The method 200 can be performed by the following description of Figure 4 The described apparatus 400 performs.

[0065] Below, the various steps of method 200 are described in more detail. Even though illustrated in a particular order, the steps of method 200 may be performed in any suitable order and multiple times. Figure 2A specific order of method steps can be shown, but the order of steps may be different from the order depicted. In addition, two or more steps may be performed simultaneously or partially concurrently. For example, the steps represented as S208 and S210 may be performed independently of each other. This variation will depend on the selected software and hardware systems and the designer's choice. All these variations are within the scope of the present invention. Similarly, software implementations may be accomplished using standard programming techniques with rule-based logic and other logic to accomplish various steps. Further variations of method 200 will become apparent from this disclosure. The embodiments mentioned and described herein are given as examples only and should not be limitations of the present invention. Other solutions, uses, objectives, and functions within the scope of the present invention claimed in the patent claims described below should be apparent to those skilled in the art.

[0066] The method 200 includes obtaining S202 sensor data related to the physical environment. The sensor data may be collected by one or more onboard sensors of the vehicle. Therefore, the sensor data may be related to the surrounding physical environment of the vehicle for which the sensor data has been collected. The vehicle may be a vehicle provided with an online perception model. Alternatively, the vehicle may be a different vehicle configured for sensor data collection.

[0067] Throughout this disclosure, the term "obtain" should be interpreted broadly and includes receiving, retrieving, collecting, acquiring, etc., directly and / or indirectly between two entities configured to communicate with each other or further communicate with other external entities. However, in some embodiments, the term "obtain" will be interpreted as determining, deriving, forming, calculating, etc. Therefore, as used in this article, "obtaining" can indicate receiving a parameter from a second entity / unit at a first entity / unit, or determining a parameter at a first entity / unit based on data received from another entity / unit, for example. In some embodiments, the sensor data is obtained by receiving from a vehicle that has collected sensor data. The vehicle can be part of a fleet configured to collect sensor data for use as training data. It should be noted that the vehicle that has collected sensor data does not need to be the same vehicle as the one provided with the online perception model. In some embodiments, the sensor data is obtained by retrieving from a database. In other words, the database may include sensor data that has been collected by one or more vehicles or by any other collection device.

[0068] The method 200 further includes inputting the obtained sensor data into the Figure 1The method 100 of claim 100 may be described in any embodiment of the method 100 to provide an offline perception model to determine S204 a perception output. In other words, the perception output may be determined by feeding sensor data to a trained offline perception model. The offline perception model may be trained to perform the same perception task as the online perception model. Therefore, the perception output may be the same type of output that the online perception model will output. More specifically, the perception output may include, for example, a bounding box of an object detected in the sensor data, a label of a determined object, and / or a segmentation of the sensor data. Therefore, the perception output of the offline perception model may be used as annotation data for the sensor data input to the offline perception model. The perception output may be used directly as annotation data. However, in some embodiments, the perception output may be further processed before being used as annotation data.

[0069] The offline perception model provided as described above is also able to perceive objects in new or previously unseen scenes or environments due to its high performance, making it possible to provide annotated data to a variety of scenarios. This means that the offline perception model becomes more capable of annotating data than previous attempts to implement automatic annotation models that are trained only on limited training data sets of explicitly annotated data. As a non-limiting example, at least in part because the trained base model is part of the model architecture of the offline perception model, the offline perception model is still able to recognize tractors in urban environments depicted in the obtained sensor data despite being fine-tuned on a second training data set that only includes examples of tractors in rural environments. Another automatic annotation model that is trained only on a training data set like the second training data set described above may not be able to recognize tractors in such new scenes. It should be understood that this simplified example is merely for the purpose of illustrating the principles of the presently disclosed technology and may not represent actual conditions.

[0070] The method 200 further includes storing the sensor data together with the perception output as annotated data S206 for subsequent training of an online perception model of the vehicle. The above steps may be repeated for additional sensor data to obtain a training data set of annotated sensor data. The training data set may then be used to train the online perception model using supervised learning.

[0071] The method 200 may further include transmitting S208 the perception data together with the perception output to the vehicle for subsequent training of the online perception model in the vehicle. Thus, the local model of the online perception model may be trained at the edge, ie by the vehicle.

[0072] The method 200 may further include training S210 the online perception model on the stored sensor data together with the perception output, thereby generating an updated online perception model. Thus, a global (or master) model of the online perception model may be trained, for example, by a centralized server. It should be understood that the online perception model may be trained (or retrained) using the stored annotation data in any suitable manner available to those skilled in the art today.

[0073] The method 200 may further include transmitting S212 the updated online perception model to the vehicle or fleet. It should be understood that the updated online perception model may be transmitted as an entire model, or as information indicating how to update a perception model already provided in the vehicle. For example, only the values ​​of any updated parameters of the online perception model may be transmitted, thereby reducing the amount of data transmitted.

[0074] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0075] Figure 3 is a schematic diagram of an apparatus 300 for providing subsequent annotations for training data for an offline perceptual model for use in training an online perceptual model according to some embodiments. The apparatus 300 may be configured to perform Figure 1 Method 100 is described.

[0076] For the purposes of this patent application, the device 300 described herein refers to a computer system or networked device configured to provide various computing services, data storage, processing capabilities or resources to a client or user through a communication network. In the present case, the term "client" refers to a connected vehicle of a fleet (e.g., vehicle 400 described below). Therefore, the device 300 described herein may refer to a general computing device. The device 300 may be a server such as a remote server, a cloud server, a central server, a background server, a fleet server, or a back-end server. Although the device 300 is illustrated herein as one device, the device 300 may be a distributed computing system formed by multiple different devices.

[0077] The device 300 comprises a control circuit 302. The control circuit 302 may physically comprise one single circuit device. Alternatively, the control circuit 302 may be distributed over several circuit devices.

[0078] As in Figure 3As shown in the example of , the apparatus 300 may further include a transceiver 306 and a memory 308. The control circuit 302 is communicatively connected to the transceiver 306 and the memory 308. The control circuit 302 may include a data bus, and the control circuit 302 may communicate with the transceiver 306 and / or the memory 308 via the data bus.

[0079] The control circuit 302 may be configured to perform overall control of the functions and operations of the device 300. The control circuit 302 may include a processor 304 such as a central processing unit (CPU), a microcontroller, or a microprocessor. The processor 304 may be configured to execute program code stored in the memory 308 to perform the functions and operations of the device 300. The control circuit 302 is configured to perform the functions and operations described above with respect to Figure 1 The steps of method 100 are described. The steps may be implemented in one or more functions stored in memory 308.

[0080] The transceiver 306 is configured to enable the device 300 to communicate with other entities such as vehicles or other devices. The transceiver 306 can send data from the device 300 and receive data to the device 200.

[0081] The memory 308 may be a non-transitory computer-readable storage medium. The memory 308 may be one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, a random access memory (RAM), or other suitable devices. In a typical arrangement, the memory 308 may include a non-volatile memory for long-term data storage and a volatile memory used as a system memory for the device 300. The memory 308 may exchange data with the circuit 302 via a data bus. There may also be accompanying control lines and an address bus between the memory 308 and the circuit 302.

[0082] The functions and operations of the apparatus 300 may be implemented in the form of an executable logic program (e.g., code lines, software program, etc.) stored on a non-transitory computer-readable recording medium (e.g., memory 308) of the apparatus 300, and executed by the circuit 302 (e.g., using the processor 304). In other words, when it is stated that the circuit 302 is configured to perform a specific function, the processor 304 of the circuit 302 may be configured to execute a program code portion stored on the memory 308, wherein the stored program code portion corresponds to the specific function. In addition, the functions and operations of the circuit 302 may be independent software applications, or form part of a software application that performs additional tasks associated with the circuit 302. The functions and operations described may be considered as being such as those described above with respect to Figure 1The method 100 discussed, the corresponding device is configured to perform the method. In addition, although the functions and operations described can be implemented in software, such functions can also be implemented via dedicated hardware or firmware or some combination of one or more hardware, firmware and software. In the following, the functions and operations of the device 300 are described.

[0083] The control circuit 302 is configured to train a base model using a first training data set to predict a trajectory of a vehicle based on a sensor data sequence. The sensor data sequence includes sensor data related to the surrounding environment of the vehicle. The first training data set includes the sensor data sequence and information indicating a driving trajectory associated with the corresponding sensor data sequence. Training the base model can be performed, for example, by executing a training function 310.

[0084] The control circuit 302 is further configured to form an offline perception model by adding a task-specific layer to the trained base model. The task-specific layer is configured to perform the perception task of the offline perception model. This can be performed, for example, by executing the formation function 312.

[0085] The control circuit 302 is further configured to fine-tune the offline perception model using a second training data set to perform the perception task. The second training data set includes training data annotated for the perception task. This can be performed, for example, by executing a fine-tuning function 314.

[0086] It should be noted that, as mentioned above Figure 1 The principles, features, aspects and advantages of the described method 100 also apply to the apparatus 300 as described herein. To avoid unnecessary repetitions, reference is made to the above.

[0087] Figure 4 4 is a schematic diagram of an apparatus 400 for annotating data for use in subsequent training of an online perception model. The online perception model is configured to perform a perception task of a vehicle equipped with an autonomous driving system. The apparatus 400 may be configured to perform Figure 2 The method 200 is described. Figure 4 The device 400 described above is similar to the Figure 3 The described apparatus 300 may be incorporated into a single apparatus such as a common server.

[0088] For the purposes of this patent application, the device 400 described herein refers to a computer system or networked device configured to provide various computing services, data storage, processing capabilities or resources to a client or user through a communication network. In the present case, the term "client" refers to a connected vehicle of a fleet (e.g., vehicle 400 described below). Therefore, the device 400 described herein may refer to a general computing device. The device 400 may be a server such as a remote server, a cloud server, a central server, a background server, a fleet server, or a back-end server. Although the device 400 is illustrated herein as one device, the device 400 may be a distributed computing system formed by multiple different devices.

[0089] The device 400 comprises a control circuit 402. The control circuit 402 may physically comprise one single circuit device. Alternatively, the control circuit 402 may be distributed over several circuit devices.

[0090] As in Figure 4 As shown in the example of , the apparatus 400 may further include a transceiver 406 and a memory 408. The control circuit 402 is communicatively connected to the transceiver 406 and the memory 408. The control circuit 402 may include a data bus, and the control circuit 402 may communicate with the transceiver 406 and / or the memory 408 via the data bus.

[0091] The control circuit 402 may be configured to perform overall control of the functions and operations of the device 400. The control circuit 402 may include a processor 404 such as a central processing unit (CPU), a microcontroller, or a microprocessor. The processor 404 may be configured to execute program code stored in the memory 408 to perform the functions and operations of the device 400. The control circuit 402 is configured to perform the functions and operations described above with respect to Figure 2 The steps of method 200 are described. The steps may be implemented in one or more functions stored in memory 408.

[0092] The transceiver 406 is configured to enable the device 400 to communicate with other entities such as vehicles or other devices. The transceiver 406 can send data from the device 400 and receive data to the device 200.

[0093] The memory 408 may be a non-transitory computer-readable storage medium. The memory 408 may be one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, a random access memory (RAM), or other suitable devices. In a typical arrangement, the memory 408 may include a non-volatile memory for long-term data storage and a volatile memory used as a system memory for the device 400. The memory 408 may exchange data with the circuit 402 via a data bus. There may also be accompanying control lines and an address bus between the memory 408 and the circuit 402.

[0094] The functions and operations of the device 400 may be implemented in the form of an executable logic program (e.g., code lines, software program, etc.) stored on a non-transitory computer-readable recording medium (e.g., memory 408) of the device 400, and executed by the circuit 402 (e.g., using the processor 404). In other words, when it is stated that the circuit 402 is configured to perform a specific function, the processor 404 of the circuit 402 may be configured to execute a program code portion stored on the memory 408, wherein the stored program code portion corresponds to the specific function. In addition, the functions and operations of the circuit 402 may be independent software applications, or form part of a software application that performs additional tasks associated with the circuit 402. The functions and operations described may be considered as being such as those described above with respect to Figure 2 The method 200 discussed, the corresponding device is configured to perform the method. In addition, although the functions and operations described can be implemented in software, such functions can also be implemented via dedicated hardware or firmware or some combination of one or more hardware, firmware and software. In the following, the functions and operations of the device 400 are described.

[0095] The control circuit 402 is configured to obtain sensor data related to the physical environment. The sensor data may be related to the surrounding physical environment of the vehicle for which the sensor data has been collected. This may be done, for example, by executing an obtain function 410.

[0096] The control circuit 402 is further configured to input the obtained sensor data into the control circuit 402 as described above. Figure 1 The described method 100 provides for determining a perceptual output in an offline perceptual model. This may be performed, for example, by executing a determination function 412 .

[0097] The control circuit 402 is further configured to store the sensor data together with the perception output as annotation data for subsequent training of the online perception model of the vehicle. This can be performed, for example, by executing the storage function 414.

[0098] The control circuit 402 may be further configured to transmit the perception data together with the perception output to the vehicle for subsequent training of an online perception model in the vehicle. This may be performed, for example, by executing a transmission function 416.

[0099] The control circuit 402 is further configured to train the online perception model on the stored sensor data together with the perception output, thereby generating an updated online perception model. This can be performed, for example, by executing the training function 418.

[0100] It should be noted that, as mentioned above Figure 2 The principles, features, aspects and advantages of the described method 200 also apply to the apparatus 400 as described herein. To avoid unnecessary repetitions, reference is made to the above.

[0101] Figure 5 5 is a schematic diagram of a vehicle 500 in accordance with some embodiments. The vehicle 500 is equipped with an automated driving system (ADS) 510. As used herein, a "vehicle" is any form of motorized transportation. For example, the vehicle 500 may be any road vehicle such as a car (as illustrated herein), a motorcycle, a (freight) truck, a bus, a smart bike, etc.

[0102] Vehicle 500 includes a number of elements commonly found in autonomous or semi-autonomous vehicles. It will be appreciated that vehicle 500 can have Figure 5 Any combination of the various elements shown in FIG. In addition, the vehicle 500 may include Figure 5 Although various elements are shown herein as being located inside the vehicle 500, one or more of the elements can be located outside the vehicle 500. Furthermore, as will be readily appreciated by those skilled in the art, even though various elements are described herein in a particular arrangement, the various elements can be implemented in a different arrangement. It should be further noted that the various elements can be communicatively connected to each other in any suitable manner. Since the elements of the vehicle 500 can be implemented in several different ways, Figure 5 The vehicle 500 should be considered as an illustrative example only.

[0103] The vehicle 500 includes a control system 502. The control system 502 is configured to perform overall control of the functions and operations of the vehicle 500. The control system 502 includes a control circuit 504 and a memory 506. The control circuit 502 may physically include a single circuit device. Alternatively, the control circuit 502 may be distributed over several circuit devices. As an example, the control system 502 may share its control circuit 504 with other parts of the vehicle. The control circuit 502 may include one or more processors such as a central processing unit (CPU), a microcontroller, or a microprocessor. One or more processors may be configured to execute program codes stored in the memory 506 in order to perform functions and operations of the vehicle 500. The processor may be or include any number of hardware components for performing data or signal processing or for executing computer codes stored in the memory 506. In some embodiments, the control circuit 504 or some of its functions may be implemented on one or more so-called systems on chips (SoCs). As an example, the ADS 510 may be implemented on the SoC. Optionally, the memory 506 includes a high-speed random access memory such as DRAM, SRAM, DDRRAM or other random access solid-state storage devices; and, optionally, includes a non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 506 may include a database component, an object code component, a script component or any other type of information structure for supporting various actions of the present specification.

[0104] In the illustrated example, the memory 506 further stores map data 508. The map data 508 may be used, for example, by the ADS 510 of the vehicle 500 in order to perform autonomous functions of the vehicle 500. The map data 508 may include high-definition (HD) map data. It is contemplated that even if the memory 508 is illustrated as an element separate from the ADS 510, it may be provided as an integrated element of the ADS 510. In other words, according to some embodiments, in the implementation of the present inventive concept, any distributed or local memory device may be utilized. Similarly, the control circuit 504 may be distributed, for example, so that one or more processors of the control circuit 504 are provided as an integrated element of the ADS 510 or any other system of the vehicle 500. In other words, according to exemplary embodiments, in the implementation of the present inventive concept, any distributed or local control circuit device may be utilized.

[0105] The vehicle 500 further includes a sensor system 520. The sensor system 520 is configured to obtain sensory data about the vehicle itself or its surroundings. The sensor system 520 may, for example, include a global navigation satellite system (GNSS) module 522 (e.g., GPS) configured to collect geographic location data of the vehicle 500. The sensor system 520 may further include one or more sensors 524. The one or more sensors 524 may be any type of on-board sensor such as a camera, a lidar and a radar, an ultrasonic sensor, a gyroscope, an accelerometer, an odometer, and the like. Therefore, one or more sensors 524 may be used to collect sensor data sequences related to the surroundings of the vehicle 500 to be used as training data for the base model. In addition, the sensor system 520 may collect geographic location data (e.g., through the GNSS module 522) for obtaining data about the driving trajectory. In other words, the GNSS module 522 may collect (or record) information indicating the driving trajectory of the vehicle. Furthermore, one or more sensors 524 can be used to collect sensor data about the surroundings of the vehicle 500, which, once annotated, will be used for fine-tuning of offline perception models and / or training of online perception models. It should be understood that the sensor system 520 may also provide the possibility to acquire sensory data directly or via dedicated sensor control circuits in the vehicle 500.

[0106] The vehicle 500 further includes a communication system 526. The communication system 526 is configured to communicate with other vehicles (ie, via a vehicle-to-vehicle (V2V) communication protocol), remote servers (eg, as described above with respect to Figure 3 and Figure 4 The communication system 526 may communicate with an external unit of a cloud server of the device described), a database or other external device, i.e., a vehicle-to-infrastructure (V2I) or vehicle-to-everything (V2X) communication protocol. The communication system 526 may communicate using one or more communication technologies. The communication system 526 may include one or more antennas. Cellular communication technology may be used for remote communications such as to a remote server or cloud computing system. In addition, if the cellular communication technology used has low latency, it may also be used for V2V communication, V2I communication, or V2X communication. Examples of cellular radio technologies are GSM, GPRS, EDGE, LTE, 5G, 5G NR, etc., and also include future cellular solutions. However, in some solutions, medium- and short-range communication technologies such as wireless local area networks (LANs) (e.g., solutions based on IEEE 802.11) may be used to communicate with other vehicles near the vehicle 500 or with local infrastructure elements. ETSI is developing cellular standards for vehicle communications, for example, 5G is considered a suitable solution due to low latency and efficient handling of high bandwidth and communication channels.

[0107] The communication system 526 may further provide the possibility to send the output to a remote location (e.g. a remote server, operator or control center) by means of one or more antennas. Furthermore, the communication system 526 may be further configured to allow the various elements of the vehicle 500 to communicate with each other. For example, the communication system may provide a local network setup such as CAN bus, I2C, Ethernet, fiber optics, etc. The local communication within the vehicle may also be of wireless type with protocols such as WiFi, LoRa, Zigbee, Bluetooth or similar medium / short range technologies.

[0108] The vehicle 500 further includes a steering system 520. The steering system 528 is configured to control the steering of the vehicle 500. The steering system 528 includes a steering module 530 configured to control the direction of the vehicle 500. The steering system 528 further includes a throttle module 532 configured to control the actuation of the throttle of the vehicle 500. The steering system 528 further includes a brake module 534 configured to control the actuation of the brakes of the vehicle 500. The various modules of the steering system 528 can receive manual inputs from the driver of the vehicle 500 (i.e., from the steering wheel, the accelerator pedal, and the brake pedal, respectively). However, the steering system 528 can be communicatively connected to the ADS 510 of the vehicle to receive instructions on how the various modules should act. Therefore, the ADS 510 can control the steering of the vehicle 500.

[0109] As described above, the vehicle 500 includes an ADS 510. The ADS 510 may be part of the vehicle's control system 502. The ADS 510 is configured to perform functions and operations of autonomous functions of the vehicle 500. The ADS 510 may include a plurality of modules, wherein each module is tasked with a different function of the ADS 510.

[0110] The ADS 510 may include a positioning module 512 or a positioning block / system. The positioning module 512 is configured to determine and / or monitor the geographic location and direction of the vehicle 500, and may utilize data from a sensor system 522, such as data from a GNSS module 520. Alternatively or in combination, the positioning module 512 may utilize data from one or more sensors 524. Alternatively, the positioning system may be implemented as a real-time kinematic (RTK) GPS to improve accuracy. In some embodiments, the positioning module 512 may obtain information indicating the driving trajectory of the vehicle 500. For example, the positioning module 512 may obtain geographic location data of the vehicle and potential additional sensory data, and formalize this into the driving trajectory of the vehicle in a format used by the base model. In some embodiments, the data for obtaining information indicating the driving trajectory may be transmitted to a server as raw data or partially processed data. The data may then be formulated on the server.

[0111] The ADS 510 may further include a perception module 514 or perception block / system. The perception module 514 may refer to any well-known module and / or function, such as included in one or more electronic control modules and / or nodes of the vehicle 500, adapted and / or configured to interpret perception data related to driving of the vehicle 500 to identify, for example, obstacles, lanes, relevant signs, appropriate navigation paths, etc. Thus, the perception module 514 may be adapted to rely on and obtain input from a plurality of data sources, such as automotive imaging, image processing, computer vision, and / or in-vehicle networking, in combination with, for example, sensory data from the sensor system 520. An online perception model for performing vehicle perception tasks may be provided as part of the ADS 510, or more specifically, as part of the perception module 514.

[0112] The positioning module 512 and / or the perception module 514 may be communicatively connected to the sensor system 520 to receive sensor data from the sensor system 520. The positioning module 512 and / or the perception module 514 may further send control instructions to the sensor system 520.

[0113] The ADS may further include a path planning module 516. The path planning module 516 is configured to determine a planned path for the vehicle 500 based on the perception and position of the vehicle determined by the perception module 514 and the positioning module 512, respectively. The planned path determined by the path planning module 516 may be sent to the manipulation system 528 for execution. Thus, the planned path may be interpreted as a path along which the vehicle should travel, for example, to reach its destination, comply with traffic regulations, avoid obstacles, etc. Thus, the planned path may correspond to a desired future path of the vehicle.

[0114] The ADS may further include a decision and control module 518. The decision and control module 518 is configured to execute control of the ADS 510 and make decisions for the ADS 310. For example, the decision and control module 518 may determine whether the planned path determined by the path planning module 516 should be executed. The decision and control module 518 may be further configured to detect any deviation of the vehicle, such as deviation from the planned path or expected trajectory of the path planning module 516.

[0115] It should be understood that parts of the described solutions may be implemented in the vehicle 500, in a system disposed externally to the vehicle, or in a combination of the vehicle's interior and exterior; for example, in a server that communicates with the vehicle, a so-called cloud solution. The different features and principles of the embodiments may be combined in other combinations than those described. Furthermore, the elements of the vehicle 500 (i.e., systems and modules) may be implemented in combinations different from those described herein.

[0116] Figure 6 The distributed system 600 according to some embodiments is illustrated by way of example. The system 600 should be viewed as a non-limiting example of implementing aspects disclosed herein of the inventive concept. For example, the system 600 is configured to perform the above-described Figure 1 The system 600 may be further configured to perform the method 100 described above. Figure 2 Therefore, unless otherwise stated, any features or principles described above are also applicable to the system 600 described herein, and vice versa.

[0117] System 600 includes a server 602 (or a remote server, a cloud server, a central server, a backend server, a fleet server, or a backend server), hereinafter referred to as a remote server 602 or simply as server 602. Server 600 may be a server 602 that is associated with a server 602. Figure 3 The apparatus 300 described and / or related Figure 4 In other words, the server 602 may be configured to perform the functions of the above-described apparatus. Therefore, the server 602 may be configured to perform the functions of the above-described apparatus. Figure 1 The method 100 described and / or related to Figure 2 Described method 200. As shown, server 602 may be provided in the cloud, ie, as a cloud-implemented server.

[0118] The system 600 further includes one or more vehicles 604a-c, also referred to as a fleet. The one or more vehicles 604a-c may be the same as those described above with respect to Figure 5 The vehicle 500 described above. Thus, one or more vehicles 604a-c can be used for sensor data collection to be used as training data. In addition, one or more vehicles 604a-c can be provided with an online perception model as described above.

[0119] One or more vehicles 604a-c are communicatively connected to a remote server 602 to transmit and / or receive data 606 between the vehicle and the server. One or more vehicles 604a-c may be further communicatively connected to each other. Data 606 may be any type of data such as a communication signal or sensor data. Communication may be performed by any suitable wireless communication protocol. The wireless communication protocol may be, for example, a remote communication protocol such as a cellular communication technology (e.g., GSM, GPRS, EDGE, LTE, 5G, 5G NR, etc.) or a short-range to medium-range communication protocol such as a solution based on a wireless local area network (LAN) (e.g., IEEE 802.11). The server 602 includes a suitable memory and a control circuit such as one or more processors or processing circuits and one or more other components such as a data interface and a transceiver. The server 602 may further include a software module or other component so that the control circuit can be configured to execute machine-readable instructions loaded from the memory to implement the steps of the method to be executed.

[0120] As an example, Figure 6 The fleet illustrated in FIG. 6 includes three vehicles, a first vehicle, a second vehicle, and a third vehicle 604a-c. However, the system 600 may include any number of vehicles 604a-c. In the following, the system 600 will be described primarily with reference to the first vehicle 604a. It should be understood that these principles apply to any vehicle in the fleet.

[0121] In the following, an example of how the system 600 performs techniques according to some embodiments will be described. For further details on the different steps, refer to the above Figure 1 and Figure 2 to avoid inappropriate duplication.

[0122] In the first scenario, server 602 performs a process of providing an offline perceptual model for subsequent annotation of training data. In this case, server 602 performs the process described above with respect to Figure 3 Functionality of the described apparatus 300. Data of a first dataset for training a base model and data of a second training dataset for fine-tuning an offline perception model may be collected by vehicles of a fleet.

[0123] In the second scenario, the server 602 performs a process of annotating data for use in subsequent training of the online perception model. As described above, the vehicle 604a can collect sensor data of the physical surroundings of the vehicle. The vehicle 604a can then transmit the collected sensor data to the server 602. After receiving the sensor data, the server can determine the perception output by inputting the obtained sensor data into the offline perception model. The server then stores the sensor data together with the perception output as annotation data for subsequent training of the online perception model. This process can be repeated for the sensor data received from the fleet until a sufficiently large data set has been formed. The online perception model can then be trained (or retrained) using the data set. The online perception model can be trained at the edge (i.e., at the vehicle 604a). The sensor data can then be transmitted to the vehicle 604a together with the associated perception output. Alternatively, the online perception model can be trained at the server 602. Then, an updated version of the online perception model after training can be transmitted to the vehicles of the fleet.

[0124] The above-described process of system 600 should be understood as a non-limiting example of the presently disclosed technology to enhance understanding. Further variations are apparent from this disclosure and are readily recognized by those skilled in the art.

[0125] The present invention has been presented above with reference to specific embodiments. However, other embodiments than those described above are possible and within the scope of the present invention. Within the scope of the present invention, method steps different from the above method steps may be provided for performing the method by hardware or software. Therefore, according to an exemplary embodiment, a non-transitory computer-readable storage medium is provided, which stores one or more programs configured to be executed by one or more processors of a vehicle control system, the one or more programs including instructions for performing a method according to any of the above embodiments. Alternatively, according to another exemplary embodiment, a cloud computing system may be configured to perform any of the methods presented herein. The cloud computing system may include distributed cloud computing resources that jointly perform the methods presented herein under the control of one or more computer program products.

[0126] It should be noted that any reference signs do not limit the scope of the claims, that the present invention may be implemented at least partially by both hardware and software, and that the same hardware item may represent multiple "means" or "units".

Claims

1. A computer-implemented method (100) for providing subsequent annotations for training data for an offline perceptual model for use in training an online perceptual model, the method (100) comprising: training (S102) a base model using a first training data set to predict a trajectory of a vehicle based on sensor data sequences, wherein the sensor data sequences include sensor data related to a surrounding environment of the vehicle, and wherein the first training data set includes sensor data sequences and information indicating driving trajectories associated with respective sensor data sequences; forming (S104) the offline perception model by adding a task-specific layer to the trained base model, wherein the task-specific layer is configured to perform a perception task of the offline perception model; and The offline perception model is fine-tuned ( S106 ) using a second training dataset to perform the perception task, wherein the second training dataset includes sensor data annotated for the perception task.

2. The method (100) according to claim 1, wherein: The information indicative of the driving trajectory includes data relating to the position of the vehicle at a plurality of subsequent time instances.

3. The method (100) according to claim 1 or 2, wherein: The sequence of sensor data includes sensor data relating to the surroundings of the vehicle at a plurality of subsequent time instances.

4. The method (100) according to claim 3, wherein: The sensor data includes one or more of image data, lidar data, radar data and ultrasonic data.

5. The method (100) of claim 1, wherein: The first training data set is an implicitly annotated data set, and the second training data set is an explicitly annotated data set.

6. The method (100) of claim 1, wherein: The perception task is one of object detection, object classification, object tracking, lane estimation, free space estimation, trajectory prediction, obstacle avoidance, scene classification, and traffic sign classification.

7. The method (100) of claim 1, wherein: The training of the base model is performed by imitation learning (S102), and Wherein, the fine-tuning (S106) of the offline perception model is performed through supervised learning.

8. A computer program product comprising instructions which, when executed by a computing device, cause the computing device to implement the method (100) according to claim 1.

9. An apparatus (300) for providing subsequent annotations for training data for an offline perceptual model for use in training an online perceptual model, the apparatus (300) comprising a control circuit (302) configured to: A base model is trained using the first training data set to predict the trajectory of the vehicle based on the sensor data sequence, wherein: The sensor data sequences include sensor data related to a surrounding environment of the vehicle, and wherein the first training data set includes sensor data sequences and information indicating driving trajectories associated with respective sensor data sequences; forming (S104) the offline perception model by adding a task-specific layer to the trained base model, wherein the task-specific layer is configured to perform a perception task of the offline perception model; and The offline perception model is fine-tuned using a second training dataset to perform the perception task, wherein the second training dataset includes sensor data annotated for the perception task.

10. A computer-implemented method (200) for annotating data for use in subsequent training of an online perception model, wherein: The online perception model is configured to perform a perception task of a vehicle equipped with an autonomous driving system, the method (200) comprising: Obtaining (S202) sensor data related to the physical environment; determining (S204) a perception output by inputting the obtained sensor data into an offline perception model provided by the method (100) according to claim 1; and The sensor data is stored ( S206 ) together with the perception output as annotation data for subsequent training of the online perception model.

11. The method (200) of claim 10, further comprising: The sensor data is transmitted ( S208 ) to the vehicle together with the perception output for subsequent training of the online perception model in the vehicle.

12. The method (200) according to claim 10 or 11, further comprising: The online perception model is trained (S210) on the stored sensor data together with the perception output, thereby generating an updated online perception model.

13. The method (200) of claim 12, further comprising: The updated online perception model is transmitted ( S212 ) to the vehicle.

14. A computer program product comprising instructions which, when executed by a computing device, cause the computing device to implement the method (200) according to claim 10.

15. An apparatus (400) for annotating data for use in subsequent training of an online perception model, wherein: The online perception model is configured to perform a perception task of a vehicle equipped with an autonomous driving system, and the device (400) includes a control circuit (402), wherein the control circuit (402) is configured to: Obtain sensor data related to the physical environment; determining a perception output by inputting the acquired sensor data into an offline perception model provided by the method (100) according to claim 1; and The sensor data is stored together with the perception output as annotation data for subsequent training of the online perception model.

Images