Computer-implemented method for providing perceptual model for annotation of training data

Abstract

The invention relates to a computer-implemented method of providing a perceptual model for annotation of training data. The method (100) comprises: training a base model training (S102) using a first training data set to predict sensor data relating to a physical environment of time instances in a sequence of time instances, the prediction being based on sensor data of remaining time instances in the sequence of time instances; forming (S104) an offline perceptual model by adding a task-specific layer to the trained base model, where the task-specific layer is configured to perform perceptual tasks of the offline perceptual model; and fine tuning (S106) the offline perceptual model using a second training data set to perform a perceptual task, where the second training data set includes training data labeled for the perceptual task. The invention further relates to a method of tagging data for subsequent training of an online perceptual model and to a device therefor.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving vehicles and in particular to a method and apparatus for labeling training data for training a perception model. Background Art

[0002] In recent years, with the development of technology, image capture and processing technology has been widely used in different technical fields. In particular, vehicles produced today are usually equipped with some form of vision or perception system to ensure new functions. Moreover, more and more modern vehicles have advanced driver assistance systems (ADAS) to improve vehicle safety and, more generally, road safety. ADAS - such as can be represented by adaptive cruise control (ACC), collision avoidance system, forward collision warning, lane support system, etc. - is an electronic system that can assist the driver of the vehicle. Today, research and development in many technical fields associated with both ADAS and autonomous driving (AD) are underway. ADAS and AD can also be collectively referred to as the general term autonomous driving system (ADS) corresponding to all different levels of automation, such as the driving automation levels (0-5) defined by SAE J3016.

[0003] Some of the functions of these systems can be implemented using simple rule-based techniques. However, to handle the complexity of real-world driving scenarios—which involve changing road conditions, unpredictable human or non-human behavior, and rapidly changing environments—the use of machine learning models has been shown to enhance the safety, capabilities, and performance of ADS. Machine learning models, such as deep learning models or neural networks, are particularly useful as part of an ADS perception system for, for example, detecting, identifying, or tracking objects in the vehicle's surroundings.

[0004] Solving the perception tasks necessary to achieve autonomous driving using deep learning algorithms requires large amounts of labeled training data. Such datasets need to cover any conceivable scenario that may exist while driving. Collecting data is a relatively easy task. However, the cost of annotating data to make it useful for training machine learning models is many orders of magnitude more expensive because it typically requires human involvement. These problems only get worse when turning to spatiotemporal models of sequence data that require annotation, bringing new dimensions to the annotation cost. Therefore, one of the holy grails of AD development is to find ways to do this in an automated manner. The present invention contemplates providing techniques for obtaining high-fidelity annotations in a more automated manner, which can eliminate or greatly reduce the need for human involvement. Summary of the invention

[0005] The technology disclosed herein is intended to mitigate, alleviate or eliminate one or more of the above-mentioned defects and shortcomings in the prior art to solve various problems related to obtaining labeled training data. Recent advances in large language models have shown that deep learning is most powerful when there is no clear limit on the scale of the model or the size of its input data set. The inventors realize that these aspects can also be used in other fields, such as the field of autonomous driving development for data annotation. The currently disclosed technology is at least partially based on using easily collected data to train offline models so that training data can be annotated, and then the training data can be used to train online (or production) models used in vehicles equipped with autonomous driving systems ADS.

[0006] As mentioned above, data collection is many orders of magnitude less expensive than annotation. For this reason, the currently disclosed techniques exploit data that does not require explicit labeling. The fundamental problem then becomes defining an objective function for the model that enables it to exploit this large amount of unlabeled data while building an understanding of the world around the vehicle that can be used to solve relevant AD tasks such as object or lane tracking. The proposed techniques enable one to train a model for offline automatic annotation that is limited only by the amount of raw data collected (e.g., a test vehicle or fleet) and the available computational resources for training, rather than by the resources for manual annotation. The objective function of the proposed offline perception model for subsequent annotation of training data is chosen in this article to be the problem of predicting missing sensor data in a sensor data sequence based on the available sensor data in the sensor data sequence.

[0007] Various aspects and embodiments of the disclosed invention are defined below and in the accompanying independent and dependent claims.

[0008] According to a first aspect, a computer-implemented method for providing an offline perception model for subsequent annotation of training data is provided. The training data can be used for training an online perception model. The method includes training a base model using a first training data set to predict sensor data related to a physical environment of a time instance in a sequence of time instances, the prediction being based on sensor data of remaining time instances in the sequence of time instances. The method further includes forming an offline perception model by adding a task-specific layer to the trained base model. The task-specific layer is configured to perform a perception task of the offline perception model. The method further includes fine-tuning the offline perception model using a second training data set to perform the perception task. The second training data set includes training data annotated for the perception task.

[0009] According to a second aspect, a computer program product comprising instructions is provided, which, when executed by a computing device, causes the computing device to perform a method according to any embodiment of the first aspect. According to an alternative embodiment of the second aspect, a (non-transient) computer-readable storage medium is provided. The non-transient computer-readable storage medium stores one or more programs, which are configured to be executed by one or more processors of a processing system, and the one or more programs include instructions for performing a method according to any embodiment of the first aspect. Where applicable, any of the above features and advantages of the first aspect also apply to the second aspect. In order to avoid unnecessary repetition, please refer to the above content.

[0010] According to a third aspect, a device is provided for providing an offline perception model for subsequent annotation of training data. The training data can then be used for training an online perception model. The device includes a control circuit. The control circuit is configured to train a base model using a first training data set to predict sensor data related to the physical environment of a time instance in a time instance sequence, the prediction being based on sensor data of the remaining time instances in the time instance sequence. The control circuit is further configured to form an offline perception model by adding a task-specific layer to the trained base model. The task-specific layer is configured to perform a perception task of the offline perception model. The control circuit is further configured to fine-tune the offline perception model using a second training data set to perform the perception task. The second training data set includes training data annotated for the perception task. Where applicable, any of the above features and advantages of other aspects also apply to this third aspect. To avoid unnecessary repetition, please refer to the above content.

[0011] According to a fourth aspect, a computer-implemented method for annotating data for subsequent training of an online perception model is provided. The online perception model is configured to perform perception tasks of a vehicle equipped with an autonomous driving system. The method includes obtaining sensor data related to a physical environment. The method further includes determining a perception output by inputting the obtained sensor data into an offline perception model provided by the method according to any embodiment of the first aspect. The method further includes storing the sensor data together with the perception output as annotated data for subsequent training of the online perception model. Where applicable, any of the above features and advantages of other aspects also apply to this fourth aspect. To avoid unnecessary repetition, please refer to the above content.

[0012] According to a fifth aspect, a computer program product comprising instructions is provided, and when the program is executed by a computing device, these instructions cause the computing device to perform a method according to any embodiment of the fourth aspect. According to an alternative embodiment of the fifth aspect, a (non-transient) computer-readable storage medium is provided. The non-transient computer-readable storage medium stores one or more programs, which are configured to be executed by one or more processors of a processing system, and the one or more programs include instructions for performing the method according to any embodiment of the fourth aspect. Where applicable, any of the above features and advantages of other aspects also apply to the fifth aspect. In order to avoid unnecessary repetition, please refer to the above content.

[0013] According to a sixth aspect, a device for annotating data for subsequent training of an online perception model is provided. The online perception model is configured to perform perception tasks of a vehicle equipped with an autonomous driving system. The device includes a control circuit. The control circuit is configured to obtain sensor data related to a physical environment. The control circuit is further configured to determine a perception output by inputting the obtained sensor data into an offline perception model provided by a method according to any embodiment of the first aspect. The control circuit is further configured to store the sensor data together with the perception output as annotated data for subsequent training of the online perception model. Where applicable, any of the above features and advantages of other aspects also apply to this sixth aspect. To avoid unnecessary repetition, please refer to the above content.

[0014] The term "non-transient" as used herein is intended to describe a computer-readable storage medium (or "memory") that excludes propagating electromagnetic signals, but is not intended to otherwise limit the type of physical computer-readable storage device encompassed by the phrase computer-readable medium or memory. For example, the term "non-transient computer-readable medium" or "tangible memory" is intended to encompass types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in non-transient form on a tangible computer-accessible storage medium may be further transmitted via a transmission medium or signal such as an electrical signal, an electromagnetic signal, or a digital signal, which may be transmitted via a communication medium such as a network and / or a wireless link. Therefore, the term "non-transient" as used herein is a limitation on the medium itself (i.e., tangible, not a signal), rather than a limitation on data storage persistence (e.g., RAM vs. ROM).

[0015] The disclosed aspects and preferred embodiments may be appropriately combined with each other in any manner apparent to those skilled in the art, so that one or more features or embodiments associated with one aspect may also be considered to be associated with another aspect or another aspect of the embodiment. In addition, any advantages associated with one aspect also apply to other aspects, where applicable.

[0016] As previously described, the currently disclosed technology may be advantageous because it is able to annotate training data with less human involvement. In addition to a relatively small training data set with annotated data (a second training data set compared to the first training data set), an offline perception model for annotated data can be provided using a large amount of data that does not require explicit annotation. Therefore, one can simply drive a vehicle with appropriate sensors installed around and collect relevant sensor data. Therefore, high-fidelity annotations of training data that can be used for training online perception models can then be generated in a more time-efficient and much larger manner than was previously feasible using existing technologies, which in turn can improve subsequent training of online perception models. The effect of semi-supervised learning using implicitly annotated data and / or base models makes it easy to collect these large amounts of data with almost no effort. Moreover, by deploying the perception model for data annotation as an offline model, it is possible to push the model and data set size to new heights. The effect of the above aspects can be that an offline perception model can learn to predict the complex task of missing sensor data from a sequence of sensor data provided as input to the model. Solving this task on a sufficiently large and diverse data set can produce a model that can understand the dynamics of the environment and learn the temporal evolution of the scene. Therefore, as is currently known, the offline perception model provided by the present technology may be more powerful than any automatic annotation model trained only on a limited set of manually labeled data (e.g., in terms of capability, accuracy, and overall performance). The provided offline perception model is also more powerful than models trained with contrastive loss or classification tasks, because these objectives do not directly replace the relevant AD tasks that the online perception model is intended to perform. In other words, the model can learn to minimize contrastive loss without actually having a general understanding of object tracking or other perception tasks.

[0017] Finally, compared to other solutions, the proposed objective function for training of offline perception models can be byte-efficient in the sense that the input and output of the model can include similar amounts of information.

[0018] Further embodiments are defined in the dependent claims. It should be emphasized that when used in this specification, the term "comprise / comprising" is used to specify the presence of recited features, integers, steps or components. It does not exclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

[0019] These and other features and advantages of the disclosed technology will be further elucidated below with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above-described aspects, features, and advantages of the disclosed technology will be more fully understood by reference to the following illustrative and non-limiting detailed description of example embodiments of the present disclosure when taken in conjunction with the accompanying drawings, in which:

[0021] Figure 1 is a schematic flow chart representation of a method of providing an offline perceptual model for subsequent annotation of training data according to some embodiments.

[0022] Figure 2 is a schematic flow chart representation of a method of annotating data for subsequent training of an online perception model according to some embodiments.

[0023] Figure 3 is a schematic illustration of an apparatus for providing an offline perceptual model for subsequent annotation of training data according to some embodiments.

[0024] Figure 4 is a schematic illustration of an apparatus for annotating data for subsequent training of an online perception model according to some embodiments.

[0025] Figure 5 is a schematic illustration of a vehicle in accordance with some embodiments.

[0026] Figure 6 is a schematic illustration of a system in accordance with some embodiments.

[0027] 7A to 7E Sensor data of a sequence of time instances is illustrated by way of example. DETAILED DESCRIPTION

[0028] The present disclosure will now be described in detail with reference to the accompanying drawings, in which some example embodiments of the disclosed technology are shown. However, the disclosed technology may be embodied in other forms and should not be construed as being limited to the disclosed example embodiments. The disclosed example embodiments are provided to fully convey the scope of the disclosed technology to those skilled in the art. Those skilled in the art will recognize that the steps, services, and functions explained herein may be implemented using separate hardware circuits, using software that works in conjunction with a programmed microprocessor or general-purpose computer, using one or more application specific integrated circuits (ASICs), using one or more field programmable gate arrays (FPGAs), and / or using one or more digital signal processors (DSPs).

[0029] It will also be appreciated that when the present disclosure is described in the form of a method, it may also be embodied in an apparatus comprising one or more processors, one or more memories coupled to the one or more processors, wherein computer code is loaded to implement the method. For example, in some embodiments, the one or more memories may store one or more computer programs that, when executed by the one or more processors, cause the apparatus to perform the steps, services, and functions disclosed herein.

[0030] It should also be understood that the terms used herein are only used for the purpose of describing particular embodiments and are not intended to be limiting. It should be noted that the articles "a", "an", "the" and "said" as used in the specification and the appended claims are intended to mean the presence of one or more elements, unless the context clearly indicates otherwise. Thus, for example, in some contexts, references to "unit" or "the unit" may refer to more than one unit, etc. In addition, the words "comprising", "including" and "containing" do not exclude other elements or steps. It should be emphasized that when used in this specification, the term "comprise / comprising" is used to specify the presence of the described features, integers, steps or components. It does not exclude the presence or addition of one or more other features, integers, steps, components or groups thereof. The term "and / or" should be interpreted as meaning "both" and each as optional.

[0031] It should also be understood that although the terms first, second, etc. can be used to describe various elements or features in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the embodiment, the first element can be called the second element, and similarly, the second element can be called the first element. The first element and the second element are both elements, but they are not the same element.

[0032] As used herein, the phrase "one or more" of a group of elements (such as "one or more of A, B, and C" or "at least one of A, B, and C") should be interpreted as conjunction or disjunction. In other words, it can refer to all elements, one element, or a combination of two or more elements in a group of elements. For example, the phrase "one or more of A, B, and C" can be interpreted as A or B or C, A and B and C, A and B, B and C, or A and C.

[0033] Throughout this disclosure, reference is made to a machine learning model (or simply a "model"). The term "machine learning model" herein means any form of machine learning algorithm, such as a deep learning model or a neural network, that is capable of learning and adapting from input data and subsequently making predictions, decisions, or classifications based on new data. In general, a machine learning model as used herein may be any neural network-based model that operates on sensor data of an autonomous vehicle. In the following, the terms "perception model" and "base model" will be used to distinguish more specific types of machine learning models or for the purpose of defining a machine learning model.

[0034] The deployment of machine learning models typically involves a training phase, where the model learns from labeled or unlabeled training data to achieve accurate predictions during the subsequent inference phase. The training data (and input data during inference) can be images or image sequences, LIDAR data (i.e., point clouds), radar data, etc. In addition, the training data / input data can include a combination or fusion of one or more different data types. The training data / input data can, for example, include both an image depicting a scene of the vehicle's surroundings and a LIDAR point cloud corresponding to the same scene.

[0035] In some embodiments, the machine learning algorithm may be implemented in any suitable manner known to a person of ordinary skill in the art using suitable software development machine learning code elements available to the public (e.g., such as code elements available in Python, TensorFlow, and Keras, or any other suitable software development platform).

[0036] The term "perception model" herein refers to a computing system or computing algorithm designed to perceive or interpret the environment depicted in sensor data such as digital images, video frames, LIDAR data, radar data, ultrasonic data, or other types of visual data related to driving for a vehicle. In other words, the perception model can be designed to detect, locate, identify and / or recognize specific instances of objects in sensor data, lanes, relevant signs, appropriate navigation paths, etc. Therefore, the perception model can be configured to perform the perception tasks of the vehicle's autonomous driving system ADS. Examples of perception tasks include, but are not limited to, object detection, object classification, lane estimation, and free space estimation. More specifically, the machine learning model can be an object detection model, an object classification model, a lane estimation model, or a free space estimation model. The perception model can use a combination of advanced technologies from computer vision, machine learning, and pattern recognition to analyze visual sensor data, and output, for example, a bounding box or region of interest around an object of interest present in an input image. The perception model can be further configured to classify the type of object detected. The perception model can include different architectures, including but not limited to convolutional neural networks (CNNs), recurrent neural networks (RNNs), converters, and other existing or future options.

[0037] The output of the perception model can be used for downstream tasks or by downstream systems of the ADS, such as trajectory prediction, path planning, or emergency braking systems. In some embodiments, the perception model can be part of an end-to-end model that is configured (in contrast to the above) to perform both perception tasks and downstream tasks. For example, a machine learning model can perform trajectory prediction or path planning directly based on sensor data.

[0038] Moreover, in the following, a distinction will be made between "online" and "offline" perception models. This distinction should be understood to refer to how or where the perception model is deployed. The online perception model should be interpreted as a perception model deployed at the edge, that is, directly deployed on an edge device, in this case a vehicle equipped with an ADS. Therefore, the online perception model can be regarded as a production model deployed in a vehicle. In other words, the calculation of the online perception model is performed locally close to the data source. In contrast, the offline perception model refers to a perception model deployed at a remote server (also known as a cloud server, a central server, a background server, a fleet server, or a back-end server). Moreover, in contrast to the online perception model, the offline perception model is not used in production scenarios (i.e., real-case scenarios on site). Instead, the offline perception model can run independently during the development process. Due to its computing environment, the online perception model usually has a simpler or less computationally intensive architecture than the offline perception model because it runs at the edge with limited memory and computing resources. On the other hand, the offline perception model may be a larger and more complex model because it can be deployed on a server with more available computing resources. In fact, there may be no clear limit on the size of the offline perception model because it can even be performed in parallel on multiple computing devices.

[0039] The term "base model" herein refers to a machine learning model that can be used as the basis or core architecture for building more specialized or customized machine learning models on it. The base model may also be generally referred to as a "base model" or "general model". The base model is usually pre-trained on a large and diverse data set (usually through self-supervision or semi-supervised learning) to learn the general patterns, features or representations of the data. These learned representations can be utilized and fine-tuned for various specific tasks such as natural language processing, image recognition, recommendation systems and various other applications. The typical feature of the base model is that the model size is large and includes a large number of trainable parameters. The size and complexity of the model contribute to its ability to capture complex patterns and representations from a wide range of data sets. As a non-limiting example, the base model can be built on a convolutional neural network (CNN) such as a residual neural network (commonly referred to as ResNet in the art) and one or more converter models. For example, images captured by one or more cameras of a vehicle can be fed to a CNN for encoding. Alternatively, a visual converter can be used. Then, the LIDAR point cloud and / or radar scan corresponding to the physical environment depicted in the image can be encoded by a CNN or a different model. The encoded images, LIDAR point clouds and / or radar scans can be fed to a converter model, which can build a unified abstract representation of the physical environment. The converter model can further take into account the encoded sensor data of the previous time instance or the sensor data itself. As a non-limiting example, the so-called BEVFormer (proposed by Li et al.) can be used. The unified abstract representation can then be further processed by the above converter model or a further converter model before providing the output of the base model. In short, any large model (such as CNN) can be used to encode the sensor data. Then, one or more converter models or any size can be used to interpret the encoded sensor data. Training such a base model can be done end-to-end. In other words, the entire model can be trained simultaneously as a whole. It goes without saying that the above examples of base model structures are only regarded as non-limiting examples, as those skilled in the art will readily recognize that many alternatives are also possible.

[0040] Essentially, the base model can adopt a transfer learning approach, where knowledge gained from one domain or task can be transferred and adapted to improve performance in another domain or task. The design of the base model plays a vital role in the efficiency and effectiveness of machine learning systems, ensuring faster development and improved performance in a range of applications by reusing learned features and representations.

[0041] In the present disclosure, the base model is a generative model. The term "generative model" in this article refers to a machine learning model whose task is to generate data based on some input. In the current case, the generative model is trained to predict (i.e., generate or reconstruct) sensor data for a certain time instance (or a sequence of sensor data, i.e., sensor data at a set of time instances) based on sensor data at other adjacent time instances. This is a challenging goal that, if solved on a large enough dataset, will provide a learning model that has learned to understand the behavior and characteristics of all objects on the road, as well as the road itself and the surrounding environment. More details about the online perception model, the offline perception model, the base model and their training will become apparent from the detailed description below.

[0042] Figure 1 is a schematic flow chart representation of a computer-implemented method 100 for providing an offline perceptual model for subsequent annotation of training data. The offline perceptual model may also be referred to as an annotation model. The training data may in turn be used for subsequent training of an online perceptual model. The method 100 may be combined with Figure 3 The method 100 is performed by the device 300 described. More generally, the method 100 may be performed by any suitable computing device, such as a remote server. Advantageously, the server is a device with more available computing resources than the vehicle equipped with the ADS. This may facilitate the deployment of offline perception models that are more computationally intensive. Instead, an online perception model trained on labeled training data can be deployed in the vehicle.

[0043] The term "labeling" as used in this article refers to the process of adding some form of metadata or labels to data to make it understandable and useful for machine learning algorithms. In this case, metadata can be used to enrich sensor data to make it useful for training and evaluating machine learning models. This can include associating labels to identify objects in images, for example, or determining bounding boxes or assigning segmented data. Therefore, the terms "labelling" and "labels" can be considered as a subset of data annotation. More specifically, it refers to the process of assigning one or more labels or categories to data instances (such as sensor data). For example, in image classification, labeling involves labeling images with corresponding categories (such as cats, dogs, or cars).

[0044] Below, the different steps of method 100 are described in more detail. Even though illustrated in a specific order, the steps of method 100 may be performed in any suitable order and multiple times. Figure 1A specific order of method steps may be shown, but the order of steps may be different from that depicted. In addition, two or more steps may be performed simultaneously or partially simultaneously. Such variations will depend on the selected software and hardware systems and the designer's choice. All such variations are within the scope of the present invention. Similarly, software implementations can complete various steps with standard programming techniques based on rule-based logic and other logic. Further variations of method 100 will be apparent from this disclosure. The embodiments mentioned and described herein are given as examples only and should not be limited to the present invention. Other solutions, uses, purposes and functions within the scope of the present invention claimed in the patent claims described below should be apparent to those skilled in the art.

[0045] The method 100 includes training S102 a base model using a first training data set to predict sensor data related to a physical environment of a time instance in a sequence of time instances, the prediction being based on sensor data of remaining time instances in the sequence of time instances. In other words, the base model may be trained to predict sensor data in a sequence of sensor data, the prediction being based on remaining sensor data in the sequence of sensor data. In other words, the base model is trained to generate or reconstruct sensor data related to the physical environment of the time instance based on sensor data related to the physical environment at other time instances in the sequence of time instances. 7A to 7E By way of example, sensor data 702a to 702e of a sequence of time instances 700 are illustrated. Sensor data illustrated in dashed lines correspond to retained (or masked) sensor data (which can also be described as gaps in the sensor data sequence) that are not available as input to the base model. Furthermore, the sensor data pointed to by the arrows correspond to sensor data for the time instance to be predicted. 7A to 7E The examples only illustrate some principles of how to train the base model. It goes without saying that the training of the base model can be done in many different ways. In addition, the principles of different examples can be combined in any appropriate way. As an example, Figure 7B The principle of predicting sensor data 702c based on past time instances and future time instances also applies to FIG. 7C to FIG. 7E In general, the gaps in the sensor data sequence can be synchronized between sensors so that all information is hidden for the time instance to be predicted. Alternatively or in combination, only a subset of the sensors for that time instance can be hidden. The same is true for the sensor data for the remaining time instances in the time instance sequence. Furthermore, it should be noted that 7A to 7E The principles described in can be extended to predict sensor data for multiple time instances. As an example, in Fig. 7AIn the case illustrated in , this can translate into predicting sensor data for time instances t-1 and t, for example, based on sensor data for time instances tn to t-2. All such variations fall within the scope of the inventive concept.

[0046] The remaining time instances in the sequence of time instances may include past time instances and / or future time instances of the time instance whose sensor data is predicted. Thus, in some embodiments, the base model is trained to predict sensor data for future time instances based on sensor data for multiple past time instances. Fig. 7A , where the sensor data denoted as 702e at the time instance denoted as t is to be predicted. In this example, the sensor data 702e at time instance t is predicted based on the sensor data 702a to 702d at n past time instances tn to t-1. It should be noted that the number n can be any positive integer. In some embodiments, the base model is trained to predict the sensor data of the past time instance based on the sensor data of multiple future time instances, i.e., Fig. 7A In this case, the sensor data for time instance tn, denoted as 702a, will be predicted based on the sensor data for the remaining time instances 702b to 702e. In some embodiments, the base model is trained to predict the sensor data for the time instance based on sensor data for both past time instances and future time instances. Figure 7B , where sensor data, denoted as 702c, for time instance t is predicted based on sensor data 702a-702b for n past time instances and sensor data 702d-702e for m future time instances. The number n and the number m can be any positive integers. Moreover, the number n and the number m can be the same or different values. Using both past time instances and future time instances can allow the model to gain a more complex understanding of how the physical environment evolves over time.

[0047] It should be noted that in some embodiments, the base model is trained to predict sensor data for multiple time instances. In other words, the base model can be trained to predict a sensor data sequence, where the sensor data sequence includes sensor data for multiple time instances. Thus, the base model can be trained to predict sensor data related to the physical environment of one or more time instances in the time instance sequence, based on sensor data for the remaining time instances in the time instance sequence.

[0048] The first training data set may include multiple training samples. Each training sample may consist of a sensor data sequence (i.e., sensor data of a time instance sequence). The first training data set may be easily collected by capturing sensor data of a time instance sequence of a vehicle. Thus, a large amount of data may be collected in a simple manner. When training the base model, the sensor data corresponding to the time instance to be predicted may be retained (or masked) from the base model and used as a true value. In other words, the sensor data of the remaining time instances may be fed to the base model to try to reconstruct the retained sensor data. Then, the comparison of the output of the model with the true value may be used as a basis for learning and improving the base model in predicting the sensor data. Therefore, the first training data set may be described as an implicitly annotated data set. The term "implicit annotation" herein refers to a low-level annotation that can be obtained without any manual or automatic annotation process. In other words, the sensor data may be implicitly annotated in the sense that the content of the sensor data is annotated without processing or analyzing it. Instead, it is only necessary to retain the sensor data (or part thereof) of the time instance to be predicted from the base model input. In some aspects, the first training data set may be regarded as an unlabeled training data set because no formal label is associated with the training sample. Instead, the supervisory signal comes from the data itself, rather than from an external source such as a manual or other automatic annotation process. This allows the first training dataset to be much larger than an explicitly annotated training dataset (such as the second training dataset described below).

[0049] By using a first training dataset as described above, a large amount of training data can be collected with almost no effort. Using implicitly labeled data means that there is virtually no limit to how much training data can be collected for training the base model, since if you have a fleet of vehicles, you are only limited by the storage and transmission of data between the vehicles and a central server.

[0050] Therefore, self-supervised learning can be used to perform training S102 of the basic model. Self-supervised learning herein refers to a machine learning paradigm in which a model learns from unlabeled data by creating its own supervisory signal or label (also referred to as implicit label or implicitly annotated data). In traditional supervised learning, the model is trained on a labeled data set, where each input is paired with a corresponding target label. In traditional unsupervised learning, the model is trained only on an unlabeled data set. However, in self-supervised learning, the model generates its own label or representation from the input data without any manual supervision. In this case, the self-created supervisory signal corresponds to the retained (or masked) sensor data associated with the time instance of the sensor data to be predicted by the basic model. Using self-supervised learning may have advantages because it can utilize a large amount of unlabeled data, which is usually easier than labeled data, while still enjoying at least some benefits of the supervisory signal associated with the data.

[0051] The wording "predict" as in "predicted sensor data" should be interpreted herein as generating or reconstructing sensor data for the time instance. In other words, the predicted sensor data can be considered as machine-generated data, like actual sensor data captured by real-world sensors, and corresponds to what the underlying model believes the sensor data for the time instance looks like.

[0052] The physical environment surrounding the vehicle may be understood as the general area around the vehicle in which objects (such as other road users, landmarks, obstacles, etc.) may be detected and identified by the vehicle sensors (radar sensors, LIDAR sensors, cameras, etc.) (i.e. within the sensor range of the vehicle). Sensor data relates to the physical environment in the sense that the sensor data reflects one or more properties of the physical environment (e.g. by depicting one or more objects in the physical environment).

[0053] The sensor data may be collected by onboard sensors of a vehicle equipped with an ADS. Therefore, the sensor data may be related to the physical environment around the vehicle. The sensor data may include one or more types of sensor data in a group including image data, LIDAR data, radar data, and ultrasonic data. The image data may be, for example, one or more images or image frames. The LIDAR data may be a point cloud. In some embodiments, the sensor data to be predicted is LIDAR data. The LIDAR data may be predicted based on the LIDAR data of the remaining time instance or based on sensor data including one or more types of sensor data. Training an offline perception model to predict LIDAR data may be advantageous because the offline perception model can focus on learning the appearance and dynamic behavior of objects in the physical environment without having to understand complex optical physics, such as the task of optical physics is to predict images. Therefore, the task of predicting a new LIDAR point cloud may be easier to achieve than predicting a new camera image, while still exceeding the perception task for which the labeled training data is expected to be used. Therefore, this can provide a balance between still achieving an efficient model and not being too complex or too computationally intensive.

[0054] By using only one type of sensor data, the base model can be trained to better understand the world from that perspective. By using more than one type of sensor data, the base model can also learn how different sensor data types relate to each other.

[0055] The sensor data may be raw sensor data. Alternatively, the sensor data may be processed or fused sensor data of two or more different types of sensor data. Fig. 7A and Figure 7B The illustration shows two examples using only one type of sensor data. In other words, the sensor data 702a to 702e may be any of image data, LIDAR data, radar data, or ultrasonic data. Thus, the base model may be trained S102 to predict one type of sensor data. However, as explained above, the sensor data may include more than one type of sensor data. Thus, the base model may be trained S102 to predict at least one type of sensor data. FIG. 7C to FIG. 7EAn example is shown where sensor data 702a to 702e includes more than one type of sensor data (such as three types of sensor data as shown). It goes without saying that the sensor data may include at least two types of sensor data from the above-mentioned sensor data type group. If the sensor data includes two or more types of sensor data, the S102 base model may be trained to predict the types of the two or more types of sensor data for a time instance in a time instance sequence, the prediction being based on the sensor data for the remaining time instances in the time instance sequence. In other words, the sensor data for the predicted time instance t may be predicted based solely on the sensor data for the remaining time instances. This is in Figure 7C , where all sensor data for the predicted time instance t are retained. Moreover, if the sensor data includes two or more types of sensor data, the base model can be trained to predict any number of types of sensor data. For example, it can be trained to predict only one type of sensor data, or to predict all types of sensor data.

[0056] If the sensor data includes two or more types of sensor data, the base model may be trained S102 to predict the types of the two or more types of sensor data of the time instance in the time instance sequence based on the sensor data of the remaining time instances in the time instance sequence and further based on the remaining types of sensor data in the two or more types of sensor data of the predicted time instance. Two such examples are shown in Fig.7D and Fig. 7E In. Fig.7D , only one type of sensor data 702e for the predicted time instance t is retained. The base model can then be trained to predict this one type based on all sensor data for the remaining time instances tn to t-1 and the available sensor data for the predicted time instance t. In some embodiments, partial sensor data of one or more types of sensor data can be retained. The base model can then be trained to predict portions of the sensor data. In general, some form of arbitrary masking can be applied to the sensor data sequence and then the base model is trained to learn predictions. This enables the base model to gain insight into how objects and the physical environment evolve over time. As a non-limiting example, a base model can be trained to predict radar data (or a portion of radar data) for a time instance to be predicted based on one-tenth of the points in the past LIDAR point cloud, the lower half of the future image, and one-third of the ultrasound data for the time instance to be predicted. Fig. 7EA slightly different scenario is shown, where the sensor data type to be predicted at time instance t is also retained from the remaining time instances used as input to the offline perception model. For example, the offline perception model can be trained to predict the LIDAR data for the time instance to be predicted, with the prediction being based on other types of sensor data for the remaining time instances. This can be advantageous because it forces the offline perception model to also learn how different sensor data types relate to each other.

[0057] In another specific non-limiting example, a base model can be trained to predict future LIDAR point clouds (i.e., based only on previous time instances). As input to the base model, all available sensor data can be used. Using all sensors in the input allows the model to learn the most information about the world. For example, in LIDAR data, in general, a car at a distance of 200 meters may not be recognizable, but in combination with a camera, the model can know that a single point of the LIDAR point cloud corresponds to a car. With this one LIDAR point, the model can now also learn more about the location of the car. Moreover, predicting future LIDAR data requires a full understanding of scene dynamics, etc. Therefore, it becomes an appropriate task for the base model. Camera images also have this property, but correctly learning the physical principles of all lighting, reflections, etc. may be a more difficult task than learning to model LIDAR data.

[0058] The method 100 further includes forming S104 an offline perception model by adding a task-specific layer to the trained base model. The task-specific layer is configured to perform the perception task of the offline perception model. The task-specific layer may include one or more sublayers required for performing the perception task. The perception task may be one of object detection, object classification, object tracking, lane estimation, free space estimation, trajectory prediction, obstacle avoidance, path planning, scene classification, and traffic sign classification.

[0059] The step of forming S104 offline perception model can be regarded as converting the trained base model into a task-specific model. In addition to adding task-specific layers, the structure of the base model can also be modified to accommodate this. For example, the output layer of the base model can be removed and replaced with a task-specific layer. Moreover, the input layer of the base model can be modified or replaced. For example, if the base model and the offline perception model will take different types of data as input, this may be the case. After adding a task-specific layer suitable for making predictions based on the expected perception task, the base model can be frozen before subsequently fine-tuning the model, as described below. Methods for converting a trained base model into a task-specific model (also known as transfer learning) are considered to be well-known in the art. Any suitable approach can be used in this case.

[0060] Method 100 further includes fine-tuning the offline perceptual model using a second training data set S106 to perform the perceptual task. The second training data set includes training data that is annotated for the perceptual task. Fine-tuning the offline perceptual model can adapt the already trained and then modified base model to the perceptual task. Fine-tuning may involve training a portion of the offline perceptual model such as a task-specific layer. Prior to this, the base model that is part of the offline perceptual model can be frozen so that its trainable parameters do not change during the fine-tuning process. Thus, fine-tuning of the offline perceptual model allows learning of trainable parameters (e.g., model weights) of the task-specific layer. Alternatively, the entire offline perceptual model can be trained during fine-tuning. In other words, during fine-tuning of the offline perceptual model, one or more trainable parameters of the base model can be updated.

[0061] In contrast to the first training dataset, which may be an implicitly annotated dataset, the second training dataset may be an explicitly annotated dataset. In other words, the second training dataset may include training samples with explicit labels determined, for example, by a manual or automatic annotation procedure. Since the offline perception model includes an already trained base model, the second training dataset may be several orders of magnitude smaller than the first training dataset, while still being able to achieve higher accuracy and performance compared to a perception model trained only on the second dataset. In other words, the second training dataset may be several orders of magnitude smaller than the first training dataset. Fine-tuning S106 of the offline perception model may be performed by supervised learning. More specifically, fine-tuning S106 of the offline perception model may be performed by supervised learning using explicitly annotated data.

[0062] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0063] In general, computer accessible media may include any tangible or non-transient storage media or storage media such as electronic, magnetic or optical media (e.g., disks or CD / DVD-ROMs connected to a computer system via a bus). The terms "tangible" and "non-transient" as used herein are intended to describe computer readable storage media (or "memory") that exclude propagating electromagnetic signals, but are not intended to otherwise limit the types of physical computer readable storage devices encompassed by the phrase computer readable media or memory. For example, the terms "non-transient computer readable media" or "tangible memory" are intended to encompass types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in non-transient form on tangible computer accessible storage media may be further transmitted via a transmission medium or a signal such as an electrical signal, an electromagnetic signal, or a digital signal, which may be transmitted via a communication medium such as a network and / or a wireless link.

[0064] Figure 2 is a schematic flow chart representation of a computer-implemented method 200 for annotating data for subsequent training of an online perception model according to some embodiments. The online perception model is configured to perform a perception task for a vehicle equipped with an autonomous driving system. The method 200 may be combined as follows: Figure 4 The described apparatus 400 performs.

[0065] Below, the different steps of method 200 are described in more detail. Even though illustrated in a specific order, the steps of method 200 may be performed in any suitable order and multiple times. Figure 2 The specific order of the method steps may be shown, but the order of the steps may be different from that depicted. In addition, two or more steps may be performed simultaneously or partially simultaneously. For example, the steps represented as S208 and S210 may be performed independently of each other. Such variations will depend on the selected software and hardware systems and the designer's choice. All such variations are within the scope of the present invention. Similarly, software implementations can complete various steps with standard programming techniques based on rule-based logic and other logics. Further variations of method 200 will be apparent from this disclosure. The embodiments mentioned and described herein are given as examples only and should not be limited to the present invention. Other solutions, uses, purposes and functions within the scope of the present invention claimed in the patent claims described below should be apparent to those skilled in the art.

[0066] The method 200 includes obtaining S202 sensor data related to a physical environment. The sensor data may be collected by one or more onboard sensors of a vehicle. Therefore, the sensor data may be related to the surrounding physical environment of the vehicle from which the sensor data was collected. The vehicle may be a vehicle equipped with an online perception model. Alternatively, the vehicle may be a different vehicle configured for sensor data collection.

[0067] Throughout this disclosure, the wording "obtain" should be interpreted broadly and encompasses direct and / or indirect reception, retrieval, collection, acquisition, etc. between two entities configured to communicate with each other or further communicate with other external entities. However, in some embodiments, the term "obtain" should be interpreted as determining, deriving, forming, calculating, etc. Therefore, as used herein, "obtain" may indicate that a parameter is received at a first entity / unit from a second entity / unit, or that a parameter is determined at a first entity / unit, for example based on data received from another entity / unit. In some embodiments, the sensor data is obtained by receiving the sensor data from a vehicle that collected the sensor data. The vehicle may be part of a fleet configured to collect sensor data for use as training data. It should be noted that the vehicle that collected the sensor data does not need to be the same vehicle as the vehicle provided with the online perception model. In some embodiments, the sensor data is obtained by retrieving from a database. In other words, the database may include sensor data that has been collected by one or more vehicles or any other collection device.

[0068] The method 200 further includes inputting the obtained sensor data into the above combination Figure 1 S204 The perception output is determined in the offline perception model provided in any embodiment of the described method 100. In other words, the perception output can be determined by feeding the sensor data to the trained offline perception model. The offline perception model can be trained to perform the same perception task as the online perception model. Therefore, the perception output can be the same type of output that the online perception model will output. More specifically, the perception output can, for example, include bounding boxes of objects detected in the sensor data, labels of identified objects, and / or segmentation of the sensor data. Therefore, the perception output of the offline perception model can be used as annotation data for the sensor data input into the offline perception model. The perception output can be used directly as annotation data. However, in some embodiments, the perception output can be further processed before being used as annotation data.

[0069] According to the offline perception model provided as described above, benefiting from its high performance, it is also possible to perceive objects in new or previously unseen scenes or environments, and thus can provide labeled data for various scenes. This means that the offline perception model becomes more capable of labeling data than previous attempts to implement automatic labeling models, and previous automatic labeling models are only trained on limited training data sets that are explicitly labeled. As a non-limiting example, although the offline perception model is fine-tuned on a second training data set that only includes examples of tractors in rural environments, it is still able to recognize tractors in urban environments depicted in the obtained sensor data, at least in part because the trained base model is part of the model architecture of the offline perception model. Another automatic labeling model that is only trained on a training data set like the second training data set described above may not be able to recognize tractors in such new scenes. It should be recognized that this simplified example is only for the purpose of illustrating the principles of the disclosed technology and may not represent actual conditions.

[0070] The method 200 further includes storing the sensor data together with the perception output as annotated data S206 for subsequent training of an online perception model of the vehicle. The above-mentioned steps can be repeated for additional sensor data to obtain a training data set of annotated sensor data. The training data set can then be used to train the online perception model using supervised learning.

[0071] The method 200 may further include transmitting S208 the perception data together with the perception output to the vehicle for subsequent training of the online perception model in the vehicle. Thus, the local model of the online perception model may be trained at the edge (ie, by the vehicle).

[0072] The method 200 may further include training S210 the online perception model for the stored sensor data and the perception output, thereby generating an updated online perception model. Thus, a global (or master) model of the online perception model may be trained, for example, by a centralized server. The method 200 may then further include transmitting the updated online perception model to the vehicle or fleet.

[0073] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0074] Figure 3 is a schematic illustration of an apparatus 300 for providing an offline perceptual model for subsequent annotation of training data for training an online perceptual model according to some embodiments. The apparatus 300 may be configured to perform a combination of Figure 1 Method 100 is described.

[0075] For the purposes of this patent application, the device 300 as described herein refers to a computer system or network device configured to provide various computing services, data storage, processing capabilities or resources to a client or user through a communication network. In the present case, the term "client" refers to a networked vehicle in a fleet (such as vehicle 400 described below). Therefore, the device 300 as described herein may refer to a general computing device. The device 300 may be a server such as a remote server, a cloud server, a central server, a background server, a fleet server, or a back-end server. Even if the device 300 is explained as one device in this article, the device 300 may also be a distributed computing system formed by multiple different devices.

[0076] The device 300 comprises a control circuit 302. The control circuit 302 may physically comprise one single circuit device. Alternatively, the control circuit 302 may be distributed over several circuit devices.

[0077] like Figure 3 As shown in the example of , the apparatus 300 may further include a transceiver 306 and a memory 308. The control circuit 302 is communicatively connected to the transceiver 306 and the memory 308. The control circuit 302 may include a data bus, and the control circuit 302 may communicate with the transceiver 306 and / or the memory 308 via the data bus.

[0078] The control circuit 302 may be configured to perform overall control of the functions and operations of the device 300. The control circuit 302 may include a processor 304 such as a central processing unit (CPU), a microcontroller, or a microprocessor. The processor is configured to execute program code stored in the memory 308 to perform the functions and operations of the device 300. The control circuit 302 is configured to perform the above-mentioned combined Figure 1 The steps of method 100 are described. These steps may be implemented in one or more functions stored in memory 308.

[0079] The transceiver 306 is configured to enable the device 300 to communicate with other entities such as a vehicle or other device. The transceiver 306 can both send data from the device 300 and receive data to the device 300.

[0080] The memory 308 may be a non-transitory computer-readable storage medium. The memory 308 may be one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, a random access memory (RAM), or another suitable device. In a typical arrangement, the memory 308 may include a non-volatile memory for long-term data storage and a volatile memory used as a system memory for the device 300. The memory 308 may exchange data with the circuit 302 via a data bus. There may also be accompanying control lines and an address bus between the memory 308 and the circuit 302.

[0081] The functions and operations of the apparatus 300 may be implemented in the form of executable logic routines (e.g., lines of code, software programs, etc.) stored on a non-transitory computer-readable recording medium (e.g., memory 308) of the apparatus 300 and executed by the circuit 302 (e.g., using the processor 304). In other words, when it is stated that the circuit 302 is configured to perform a specific function, the processor 304 of the circuit 302 may be configured to execute a program code portion stored on the memory 308, wherein the stored program code portion corresponds to the specific function. In addition, the functions and operations of the circuit 302 may be independent software applications or form part of a software application that performs additional tasks associated with the circuit 302. The described functions and operations may be considered as corresponding apparatuses being configured to perform, such as the above combined Figure 1 The method 100 discussed. In addition, although the functions and operations described can be implemented in software, such functions can also be performed via dedicated hardware or firmware, or some combination of one or more of hardware, firmware and / or software. In the following, the functions and operations of the device 300 are described.

[0082] The control circuit 302 is configured to train a base model using the first training data set to predict sensor data related to the physical environment of a time instance in the time instance sequence, the prediction being based on sensor data of the remaining time instances in the time instance sequence. This can be performed, for example, by executing a training function 310.

[0083] The control circuit 302 is further configured to form an offline perception model by adding a task-specific layer to the trained base model. The task-specific layer is configured to perform the perception task of the offline perception model. This can be performed, for example, by executing the formation function 312.

[0084] The control circuit 302 is further configured to fine-tune the offline perception model using a second training data set to perform the perception task. The second training data set includes training data annotated for the perception task. This can be performed, for example, by executing a fine-tuning function 314.

[0085] It should be noted that, as above, Figure 1 The principles, features, aspects and advantages of the described method 100 are also applicable to the apparatus 300 described herein. To avoid unnecessary repetition, please refer to the above contents.

[0086] Figure 4 is a schematic illustration of an apparatus 400 for annotating data for subsequent training of an online perception model. The online perception model is configured to perform a perception task for a vehicle equipped with an autonomous driving system. The apparatus 400 may be configured to perform a combination of Figure 2 The method 200 described herein is combined with Figure 4 The device 400 described above is combined with Figure 3 The described apparatus 300 may be incorporated into a single apparatus such as a common server.

[0087] For the purposes of this patent application, the device 400 as described herein refers to a computer system or network device configured to provide various computing services, data storage, processing capabilities or resources to a client or user through a communication network. In the present case, the term "client" refers to a networked vehicle in a fleet (such as vehicle 400 described below). Therefore, the device 400 as described herein may refer to a general computing device. The device 400 may be a server such as a remote server, a cloud server, a central server, a background server, a fleet server, or a back-end server. Even if the device 400 is explained as one device in this article, the device 400 may also be a distributed computing system formed by multiple different devices.

[0088] The device 400 comprises a control circuit 402. The control circuit 402 may physically comprise one single circuit device. Alternatively, the control circuit 402 may be distributed over several circuit devices.

[0089] like Figure 4 As shown in the example of , the apparatus 400 may further include a transceiver 406 and a memory 408. The control circuit 402 is communicatively connected to the transceiver 406 and the memory 408. The control circuit 402 may include a data bus, and the control circuit 402 may communicate with the transceiver 406 and / or the memory 408 via the data bus.

[0090] The control circuit 402 may be configured to perform overall control of the functions and operations of the device 400. The control circuit 402 may include a processor 404 such as a central processing unit (CPU), a microcontroller, or a microprocessor. The processor 404 may be configured to execute program code stored in the memory 408 to perform the functions and operations of the device 400. The control circuit 402 is configured to perform the above-mentioned combination Figure 2The steps of method 200 are described. These steps may be implemented in one or more functions stored in memory 408.

[0091] The transceiver 406 is configured to enable the device 400 to communicate with other entities such as vehicles or other devices. The transceiver 406 can both send data from the device 400 and receive data to the device 300.

[0092] The memory 408 may be a non-transitory computer-readable storage medium. The memory 408 may be one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, a random access memory (RAM), or another suitable device. In a typical arrangement, the memory 408 may include a non-volatile memory for long-term data storage and a volatile memory used as a system memory for the device 400. The memory 408 may exchange data with the circuit 402 via a data bus. There may also be accompanying control lines and an address bus between the memory 408 and the circuit 402.

[0093] The functions and operations of the device 400 may be implemented in the form of executable logic routines (e.g., lines of code, software programs, etc.) that are stored on a non-transitory computer-readable recording medium (e.g., memory 408) of the device 400 and executed by the circuit 402 (e.g., using the processor 404). In other words, when it is stated that the circuit 402 is configured to perform a specific function, the processor 404 of the circuit 402 may be configured to execute a program code portion stored on the memory 408, wherein the stored program code portion corresponds to the specific function. In addition, the functions and operations of the circuit 402 may be independent software applications or form part of a software application that performs additional tasks associated with the circuit 402. The described functions and operations may be considered as corresponding devices being configured to perform, such as the above combined Figure 2 The method of method 200 discussed. In addition, although the functions and operations described can be implemented in software, such functions can also be performed via dedicated hardware or firmware, or some combination of one or more of hardware, firmware and / or software. In the following, the functions and operations of device 400 are described.

[0094] The control circuit 402 is configured to obtain sensor data related to the physical environment. The sensor data may be related to the surrounding physical environment of the vehicle for which the sensor data has been collected. This may be performed, for example, by executing an obtain function 410.

[0095] The control circuit 402 is further configured to input the obtained sensor data into the control circuit 402 by combining the above Figure 1The described method 100 provides for determining a perceptual output in an offline perceptual model. This may be performed, for example, by executing a determination function 412 .

[0096] The control circuit 402 is further configured to store the sensor data together with the perception output as annotated data for subsequent training of an online perception model of the vehicle. This can be performed, for example, by executing the storage function 414.

[0097] The control circuit 402 may be further configured to transmit the perception data together with the perception output to the vehicle for subsequent training of the online perception model in the vehicle. This may be performed, for example, by executing the transmission function 416.

[0098] The control circuit 402 may be further configured to train the online perception model for the stored sensor data and the perception output, thereby generating an updated online perception model. This may be performed, for example, by executing the training function 418.

[0099] It should be noted that, as above, Figure 2 The principles, features, aspects and advantages of the described method 200 are also applicable to the apparatus 400 described herein. To avoid unnecessary repetition, please refer to the above content.

[0100] Figure 5 5 is a schematic illustration of a vehicle 500 according to some embodiments. The vehicle 500 is equipped with an automated driving system (ADS) 510. As used herein, a "vehicle" is any form of motorized transportation. For example, the vehicle 500 may be any road vehicle such as (as illustrated herein) a car, a motorcycle, a (freight) truck, a bus, a smart bike, etc.

[0101] Vehicle 500 includes a number of elements that are typically found in an autonomous or semi-autonomous vehicle. It should be understood that vehicle 500 may have Figure 5 Any combination of the various elements shown in FIG. Moreover, the vehicle 500 may include Figure 5 500. Although various elements are shown herein as being located inside the vehicle 500, one or more elements may be located outside the vehicle 500. Further, even though various elements are depicted herein in certain arrangements, as those skilled in the art will readily appreciate, the various elements may also be implemented in different arrangements. It should be further noted that the various elements may be communicatively connected to each other in any suitable manner. Figure 5 The vehicle 500 should be viewed as an illustrative example only, as the elements of the vehicle 500 may be implemented in a number of different ways.

[0102] The vehicle 500 further includes a control system 502. The control system 502 is configured to perform overall control of the functions and operations of the vehicle 500. The system 502 includes a control circuit 504 and a memory 506. The control circuit 502 may physically include a single circuit device. Alternatively, the control circuit 502 may be distributed over several circuit devices. As an example, the control system 502 may share its control circuit 504 with other parts of the vehicle. The control circuit 502 may include one or more processors such as a central processing unit (CPU), a microcontroller, or a microprocessor. One or more processors may be configured to execute program codes stored in the memory 506 to perform functions and operations of the vehicle 500. The processor may be or include any number of hardware components for performing data or signal processing or for executing computer codes stored in the memory 506. In some embodiments, the control circuit 504 or some of its functions may be implemented on one or more so-called systems on chips (SoCs). As an example, the ADS 510 may be implemented on the SoC. Memory 506 optionally includes high-speed random access memory such as DRAM, SRAM, DDR, RAM or other random access solid-state memory devices; and optionally includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. Memory 506 may include database components, object code components, script components or any other type of information structure for supporting various activities of this specification.

[0103] In the illustrated example, the memory 506 further stores map data 508. The map data 508 can be used, for example, by the ADS 510 of the vehicle 500 to perform automatic functions of the vehicle 500. The map data 508 may include high-definition (HD) map data. It is considered that even if the memory 508 is illustrated as an element separate from the ADS 510, it can also be provided as an integrated element of the ADS 510. In other words, according to some embodiments, any distributed or local memory device can be used for the implementation of the present invention. Similarly, the control circuit 504 can be distributed, for example, so that one or more processors of the control circuit 504 are provided as an integrated element of the ADS 510 or any other system of the vehicle 500. In other words, according to exemplary embodiments, any distributed or local control circuit device can be used for the implementation of the present invention.

[0104] The vehicle 500 further includes a sensor system 520. The sensor system 520 is configured to acquire sensor data about the vehicle itself or its surroundings. The sensor system 520 may, for example, include a global navigation satellite system (GNSS) module 522 (such as GPS) configured to collect geographic location data of the vehicle 500. The sensor system 520 may further include one or more sensors 524. The one or more sensors 524 may be any type of on-board sensor such as a camera, LIDAR, RADAR, ultrasonic sensor, gyroscope, accelerometer, odometer, etc. Therefore, one or more sensors 524 may be used to collect sensor data sequences related to the physical surroundings of the vehicle 500 to be used as training data for the base model. Moreover, one or more sensors 524 may be used to collect sensor data related to the surrounding physical environment of the vehicle 500 to be used for fine-tuning of offline perception models and / or training of online perception models after annotation. It should be appreciated that the sensor system 520 may also provide the possibility of acquiring sensor data directly or via a dedicated sensor control circuit in the vehicle 500.

[0105] The vehicle 500 further includes a communication system 526. The communication system 526 is configured to communicate with other vehicles (i.e., via a vehicle-to-vehicle (V2V) communication protocol), remote servers (e.g., as described above in conjunction with Figure 3 and Figure 4 The communication system 526 may communicate with an external unit such as a cloud server, a database or other external device of the device described, i.e., a vehicle-to-infrastructure (V2I) or vehicle-to-everything (V2X) communication protocol. The communication system 526 may communicate using one or more communication technologies. The communication system 526 may include one or more antennas. Cellular communication technology may be used for remote communications such as to a remote server or a cloud computing system. In addition, if the cellular communication technology used has low latency, it may also be used for V2V, V2I or V2X communications. Examples of cellular radio technologies include GSM, GPRS, EDGE, LTE, 5G, 5G NR, etc., including future cellular solutions. However, in some solutions, medium- and short-range communication technologies such as wireless local area networks (LANs) (e.g., solutions based on IEEE 802.11) may be used to communicate with other vehicles near the vehicle 500 or with local infrastructure elements. ETSI is studying cellular standards for vehicle communications, and 5G, for example, is considered an appropriate solution because of its low latency and efficient handling of high bandwidth and communication channels.

[0106] The communication system 526 may further provide the possibility to send outputs to a remote location (e.g., a remote server, operator, or control center) via one or more antennas. Moreover, the communication system 526 may further be configured to allow the various elements of the vehicle 500 to communicate with each other. As an example, the communication system may provide a local network setup such as CAN bus, I2C, Ethernet, fiber optics, etc. The local communication within the vehicle may also be of wireless type with protocols such as WiFi, LoRa, Zigbee, Bluetooth, or similar medium / short range technologies.

[0107] The vehicle 500 further includes a steering system 520. The steering system 528 is configured to control the steering of the vehicle 500. The steering system 528 includes a steering module 530 configured to control the heading of the vehicle 500. The steering system 528 further includes a throttle module 532 configured to control the actuation of the throttle of the vehicle 500. The steering system 528 further includes a brake module 534 configured to control the actuation of the brakes of the vehicle 500. The various modules of the steering system 528 can receive manual inputs from the driver of the vehicle 500 (i.e., from the steering wheel, accelerator pedal, and brake pedal, respectively). However, the steering system 528 can be communicatively connected to the ADS 510 of the vehicle to receive instructions on how the various modules should act. Therefore, the ADS 510 can control the steering of the vehicle 500.

[0108] As described above, the vehicle 500 includes an ADS 510. The ADS 510 may be part of the vehicle's control system 502. The ADS 510 is configured to perform functions and operations of automatic functions of the vehicle 500. The ADS 510 may include a plurality of modules, wherein each module is responsible for a different function of the ADS 510.

[0109] The ADS 510 may include a positioning module 512 or positioning block / system. The positioning module 512 is configured to determine and / or monitor the geographic location and heading of the vehicle 500 and may utilize data from a sensor system 520, such as data from a GNSS module 522. Alternatively or in combination, the positioning module 512 may utilize data from one or more sensors 524. The positioning system may alternatively be implemented as a real-time kinematic (RTK) GPS for improved accuracy.

[0110] The ADS 510 may further include a perception module 514 or perception block / system. The perception module 514 may refer to any known module and / or functionality, for example included in one or more electronic control modules and / or nodes of the vehicle 500, adapted and / or configured to interpret sensory data related to driving of the vehicle 500 to identify, for example, obstacles, lanes, relevant signs, appropriate navigation paths, etc. The perception module 514 may therefore be adapted to rely on and obtain input from multiple data sources such as automotive imaging, image processing, computer vision, and / or in-vehicle networking, in combination with, for example, perception data from the sensor system 520. An online perception model for performing perception tasks of the vehicle may be provided as part of the ADS 510, or more specifically, as part of the perception module 514.

[0111] The positioning module 512 and / or the perception module 514 may be communicatively connected to the sensor system 520 to receive the perception data from the sensor system 520. The positioning module 512 and / or the perception module 514 may further transmit control instructions to the sensor system 520.

[0112] The ADS may further include a path planning module 516. The path planning module 516 is configured to determine a planned path for the vehicle 500 based on the perception and position of the vehicle determined by the perception module 514 and the positioning module 512, respectively. The planned path determined by the path planning module 516 may be sent to the manipulation system 528 for execution.

[0113] The ADS may further include a decision and control module 518. The decision and control module 518 is configured to execute control of the ADS 510 and make decisions. For example, the decision and control module 518 may decide whether the planned path determined by the path planning module 516 should be executed. The decision and control module 518 may be further configured to detect any deviation of the vehicle such as deviation from the planned path or the expected trajectory of the path planning module 516. This includes evasive maneuvers performed by the ADS 510 and the vehicle driver.

[0114] It should be understood that parts of the described solutions may be implemented in the vehicle 500, in a system located external to the vehicle, or in a combination of internal and external to the vehicle; such as a so-called cloud solution implemented in a server that communicates with the vehicle. The different features and principles of the embodiments may be combined in other combinations than those described. Further, the elements (i.e., systems and modules) of the vehicle 500 may be implemented in different combinations than those described herein.

[0115] Figure 6The distributed system 600 according to some embodiments is illustrated by way of example. The system 600 should be considered as a non-limiting example of implementing aspects of the inventive concepts disclosed herein. For example, the system 600 is configured to perform the above-mentioned Figure 1 The system 600 may further be configured to perform the above-described method 100. Figure 2 Therefore, unless otherwise stated, any features or principles described above are also applicable to the system 600 described herein, and vice versa.

[0116] System 600 includes server 602 (or remote server, cloud server, central server, backend server, fleet server or backend server), which is referred to as remote server 602 or just server 602 in the following. Server 602 can be a combination of Figure 3 The described device 300 and / or as combined Figure 4 In other words, the server 602 may be configured to perform the functions of the above-described apparatus. Therefore, the server 602 may be configured to perform the following steps in conjunction with Figure 1 The method 100 described and / or as combined Figure 2 Described method 200. As illustrated, server 602 may be provided in the cloud, ie, as a cloud-implemented server.

[0117] The system 600 further includes one or more vehicles 604a to 604c, also referred to as a fleet. The one or more vehicles 604a to 604c may be a combination of the above Figure 5 The vehicle 500 described above. Therefore, one or more vehicles 604a to 604c can be used for sensor data collection to be used as training data. Moreover, one or more vehicles 604a to 604c can be provided with an online perception model as described above.

[0118] One or more vehicles 604a to 604c are communicatively connected to a remote server 602 for sending and / or receiving data 606 between the vehicle and the server. One or more vehicles 604a to 604c may be further communicatively connected to each other. Data 606 may be any type of data such as a communication signal or sensor data. Communication may be performed by any appropriate wireless communication protocol. The wireless communication protocol may be a long-range communication protocol such as a cellular communication technology (e.g., GSM, GPRS, EDGE, LTE, 5G, 5G NR, etc.) or a medium- and short-range communication protocol such as a solution based on a wireless local area network (LAN) (e.g., IEEE 802.11). The server 602 includes appropriate memory and control circuits (e.g., one or more processors or processing circuits) and one or more other components such as a data interface and a transceiver. The server 602 may also include a software module or other components so that the control circuit may be configured to execute machine-readable instructions loaded from the memory to implement the steps of the method to be performed.

[0119] Figure 6 The fleet illustrated in FIG. 6 includes three vehicles, for example, a first vehicle 604a, a second vehicle 604b, and a third vehicle 604c. However, the system 600 may include any number of vehicles 604a to 604c. In the following, the system 600 will be described primarily with reference to the first vehicle 604a. It should be understood that these principles apply to any vehicle in the fleet.

[0120] In the following, examples of how system 600 can perform techniques according to some embodiments will be described. For further details on the different steps, please refer to the above Figure 1 and Figure 2 , to avoid excessive repetition.

[0121] In the first case, the server 602 performs the process of providing an offline perceptual model for subsequent annotation of training data. In this case, the server 602 performs the above combined Figure 3 The functionality of the apparatus 300 is described.

[0122] In the second case, the server 602 performs the process of annotating data for subsequent training of the online perception model. As explained above, the vehicle 604a can collect sensor data of the physical surroundings of the vehicle. The vehicle 604a can then transmit the collected sensor data to the server 602. Upon receiving the sensor data, the server can determine the perception output by inputting the obtained sensor data into the offline perception model. The server can then store the sensor data together with the perception output as annotated data for subsequent training of the online perception model. This process can be repeated for the sensor data received from the fleet until a sufficiently large data set is formed. The online perception model can then be trained (or retrained) using the data set. The online perception model can be trained at the edge (i.e., at the vehicle 604a). The sensor data and the associated perception output can then be transmitted to the vehicle 604a. Alternatively, the online perception model can be trained at the server 602. An updated version of the trained online perception model can then be transmitted to the vehicles of the fleet.

[0123] The above-described process of system 600 should be understood as a non-limiting example of the presently disclosed technology for improving understanding. Further variations are obvious from this disclosure and are easily realized by those skilled in the art.

[0124] The present invention is presented above with reference to specific embodiments. However, other embodiments than those described above are possible and within the scope of the present invention. Within the scope of the present invention, method steps for executing the method by hardware or software different from those described above may be provided. Therefore, according to an exemplary embodiment, a non-computer transient readable storage medium is provided that stores one or more programs, the one or more programs being configured to be executed by one or more processors of a vehicle control system, the one or more programs including instructions for executing a method according to any one of the above-described embodiments. Alternatively, according to another exemplary embodiment, a cloud computing system may be configured to execute any of the methods presented herein. The cloud computing system may include distributed cloud computing resources that jointly execute the methods presented herein under the control of one or more computer program products.

[0125] It should be noted that any reference signs do not limit the scope of the claims, that the present invention may be implemented at least partially by both hardware and software, and that the same hardware item may represent several "means" or "units".

Claims

1. A computer-implemented method (100) for providing an offline perceptual model for subsequent annotation of training data, the training data being used for training an online perceptual model, the method (100) comprising: Using a first training data set, a base model is trained (S102) to predict sensor data related to a physical environment of a time instance in a sequence of time instances, the prediction being based on sensor data of remaining time instances in the sequence of time instances; forming (S104) the offline perception model by adding a task-specific layer to the trained base model, wherein the task-specific layer is configured to perform a perception task of the offline perception model; and The offline perception model is fine-tuned using a second training data set (S106) to perform the perception task, wherein the second training data set includes training data annotated for the perception task.

2. The method (100) according to claim 1, wherein: The remaining time instances in the sequence of time instances include past time instances and / or future time instances of the time instance for which sensor data is predicted.

3. The method (100) according to claim 1 or 2, wherein: The sensor data includes one or more types of sensor data from the group consisting of image data, LIDAR data, radar data, and ultrasonic data.

4. The method (100) according to claim 3, wherein: The base model is trained ( S102 ) to predict at least one type of the sensor data.

5. The method (100) according to claim 1 or 2, wherein: The sensor data includes at least two types of sensor data from the group consisting of image data, LIDAR data, radar data, and ultrasonic data, and Therein, the base model is trained (S102) to predict the type of sensor data among the two or more types of sensor data of the time instance in the time instance sequence, the prediction being based on the sensor data of the remaining time instances in the time instance sequence.

6. The method (100) according to claim 1 or 2, wherein: The sensor data includes at least two types of sensor data from the group consisting of image data, LIDAR data, radar data, and ultrasonic data, and Wherein, the basic model is trained (S102) to: predict the types of the two or more types of sensor data of the time instances in the time instance sequence based on the sensor data of the remaining time instances in the time instance sequence, and further based on the remaining types of the two or more types of sensor data of the predicted time instances.

7. The method (100) of claim 1, wherein: The first training data set is an implicitly labeled data set, and the second training data set is an explicitly labeled data set.

8. The method (100) of claim 1, wherein: The perception task is one of object detection, object classification, object tracking, lane estimation, free space estimation, trajectory prediction, obstacle avoidance, path planning, scene classification, and traffic sign classification.

9. The method (100) of claim 1, wherein: The training of the base model is performed by self-supervised learning (S102), and Wherein, fine-tuning of the offline perception model is performed through supervised learning (S106).

10. A computer program product comprising instructions which, when executed by a computing device, cause the computing device to perform the method (100) according to claim 1.

11. An apparatus (300) for providing an offline perceptual model for subsequent annotation of training data, the training data being used for training an online perceptual model, the apparatus (300) comprising a control circuit (302), the control circuit (302) being configured to: training a base model using a first training data set to predict sensor data related to a physical environment at a time instance in a sequence of time instances, the prediction being based on sensor data at remaining time instances in the sequence of time instances; The offline perception model is formed (S104) by adding task-specific layers to the trained base model, wherein: The task-specific layer is configured to perform the perception task of the offline perception model; as well as The offline perception model is fine-tuned using a second training dataset to perform the perception task, wherein the second training dataset includes training data labeled for the perception task.

12. A computer-implemented method (200) for annotating data for subsequent training of an online perception model, wherein: The online perception model is configured to perform a perception task of a vehicle equipped with an autonomous driving system, the method (200) comprising: Obtaining (S202) sensor data related to the physical environment; Determining (S204) a perception output by inputting the obtained sensor data into an offline perception model provided by the method (100) according to claim 1; and The sensor data is stored ( S206 ) together with the perception output as annotated data for subsequent training of the online perception model.

13. The method (200) according to claim 12, further comprising training (S210) the online perception model with respect to the stored sensor data and the perception output, thereby generating an updated online perception model.

14. A computer program product comprising instructions which, when executed by a computing device, cause the computing device to perform the method (200) according to claim 12 or 13.

15. An apparatus (400) for annotating data for subsequent training of an online perception model, wherein: The online perception model is configured to perform a perception task of a vehicle equipped with an autonomous driving system, and the device (400) includes a control circuit (402), wherein the control circuit (402) is configured to: Obtain sensor data related to the physical environment; determining a perception output by inputting the obtained sensor data into an offline perception model provided by the method (100) according to claim 1; as well as The sensor data is stored together with the perception output as annotated data for subsequent training of the online perception model.

Images