Method for the automated generation of annotations for sensor data for providing a data set for machine learning
The method uses a fine-tuned foundation model with text prompts to efficiently generate high-accuracy annotations for sensor data, addressing the cost and time issues of existing annotation methods, enhancing dataset creation for machine learning.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-09-01
- Publication Date
- 2026-03-12
AI Technical Summary
Current methods for creating annotated datasets for machine learning, especially using convolutional neural networks (CNNs), are costly and time-consuming, whether through manual annotation or multimodal model fusion, due to the need for large and diverse datasets.
A method utilizing a fine-tuned foundation model to generate annotations for sensor data based on textual descriptions, enabling efficient and flexible creation of training datasets through text prompts, with optional verification and optimization processes.
Enables rapid adaptation and high-accuracy annotation generation, reducing the need for extensive manual annotation and large datasets, and improving the precision and consistency of annotations.
Smart Images
Figure EP2025074806_12032026_PF_FP_ABST
Abstract
Description
[0001] R.413920
[0002] - 1 -
[0003] Description
[0004] title
[0005] Method for automatically generating annotations for sensor data to provide a dataset for machine learning.
[0006] The invention relates to a method for automatically generating annotations for sensor data to provide a dataset for machine learning. The invention further relates to a computer program, a device, and a storage medium for this purpose.
[0007] State of the art
[0008] Environmental perception utilizes various technologies, including cameras, radar, and lidar, and especially their data, to capture the environment. Furthermore, convolutional neural networks (CNNs) can be employed to perceive the environment. Foundation models, such as EVA-CLIP, are used to further enhance this perception.
[0009] (https: / / arxiv.org / abs / 2303.15389) New possibilities for perception are emerging. Foundation models are currently being trained, for example, to analyze images for various tasks, such as describing image content or even providing chatbot answers to specific questions. Training CNNs or other methods requires a large and diverse set of annotated data, but creating this data is very costly and time-consuming.
[0010] Current methods are mostly based on multimodal models, which are used for annotation and generate the most optimal annotation (label) possible through fusion. Alternatively, manual annotation can be performed, but this is very time-consuming. R.413920
[0011] - 2 -
[0012] Disclosure of the invention
[0013] The invention relates to a method with the features of claim 1, a computer program with the features of claim 9, a device with the features of claim 10, and a computer-readable storage medium with the features of claim 11. Further features and details of the invention will become apparent from the respective dependent claims, the description, and the drawings. Features and details described in connection with the method according to the invention naturally also apply in connection with the computer program, the device, and the computer-readable storage medium according to the invention, and vice versa, so that mutual reference is always possible with regard to the disclosure of the invention.
[0014] The invention relates in particular to a method for the automated generation of annotations (English: “labels”) for sensor data to provide a data set for machine learning, comprising:
[0015] Providing sensor data, wherein the sensor data results from the acquisition of data from at least one sensor,
[0016] Fine-tuning of a foundation model, wherein the foundation model is fine-tuned to generate annotations for the sensor data based on at least one text prompt, wherein the at least one text prompt represents a distinction between classes based on a textual description, where the classes can be categories such as "vehicle", "door" or "street".
[0017] Generating annotations for the sensor data using the finely tuned base model and classes; providing the dataset based on a combination of the sensor data with the generated annotations, for example, by assigning the generated annotations to the sensor data and / or individual objects within the sensor data. R.413920
[0018] - 3 -
[0019] The dataset can be a training dataset, a validation dataset, and / or a test dataset. At least one text prompt can, for example, include an instruction to generate bounding boxes. Furthermore, attributes of individual classes to be annotated can be described, for which example images can be used. Additionally, layers for the annotations can be defined, e.g., a vehicle layer and a passenger car layer below it. The sensor data can include image data, radar data, ultrasound data, and / or lidar data; this list is not exhaustive, and other environmental sensors can also be used, with the sensor data also including the data resulting from these additional environmental sensors.The at least one sensor can be configured as a camera, radar, ultrasonic, and / or lidar sensor, although this list is not exhaustive and other environmental sensors can also be used. The annotations for the sensor data generated by the finely tuned base model can, for example, include a detailed description of each object represented in the sensor data, e.g., "door open" in relation to a vehicle. The resulting dataset can thus include annotations for individual objects in the sensor data, for a set of objects, for a situation represented by objects, and / or for the entirety of the sensor data, in particular for an entire image within the sensor data. According to the invention, the training dataset can therefore be provided efficiently and flexibly.
[0020] Furthermore, the invention may provide that the method further comprises:
[0021] Creating an annotation guide (English: "label guide") using the finely tuned base model,
[0022] Using the created annotation instruction when generating the annotations.
[0023] This allows the annotations to be generated advantageously according to a structure specified by the annotation instruction. This can enable finer control and customization of the annotation generation and R.413920
[0024] - 4 - can lead to higher accuracy and consistency of the generated annotations.
[0025] Furthermore, within the scope of the invention, it is conceivable that the method also includes:
[0026] Verification of the generated annotations by the base model and / or a second base model based on at least one defined criterion.
[0027] As part of the verification process, at least one text prompt can be provided to the second base model as input to compare the output of the first and second base models. This allows for an assessment of the precision with which classes were identified in the annotations and the sensor data were correctly assigned to the classes, and, in particular, what accuracy requirements exist. Deviations can be identified, such as incorrectly set boundary frames as annotations. It is also conceivable that the base model performs a self-assessment with regard to at least one criterion, for example, through a suitably formulated text prompt. This provides an additional quality control mechanism for the generated annotations. By comparing the results with the at least one defined criterion, e.g.,Precise class assignments allow for the identification and, if necessary, correction of erroneous annotations. The use of a second base model enables an independent review process and can further improve annotation accuracy.
[0028] Preferably, the invention may provide that the method further comprises:
[0029] Optimizing at least one text prompt for generating annotations using the finely tuned base model.
[0030] An iterative process can be used to optimize the text prompts. By using the already finely tuned base model, the wording of the text prompts can be adapted to the specific requirements of the annotations, which can lead to improved accuracy and efficiency in annotation generation. It is also conceivable that the base model could be based on example R.413920.
[0031] - 5 -
[0032] Text prompts are further refined to learn an optimal structure for at least one text prompt, and the optimization can then be carried out accordingly with the further refined basic model.
[0033] Furthermore, it may be advantageous within the scope of the invention that the method also comprises:
[0034] Transforming the at least one text prompt into an embedding space, whereby the generation of annotations is carried out based on the transformed at least one text prompt.
[0035] In other words, the textual description of the text prompt is transformed into a numerical representation, a so-called embedding. This allows the base model to better understand the text prompt and generate more precise annotations for the sensor data. The transformation can be performed using methods such as word embeddings or transformer models.
[0036] Optionally, the generation of annotations may include: performing a segmentation based on the sensor data by the fine-tuned base model, performing a classification based on a result of the segmentation, where the result can be, for example, a bounding box, and where the fine-tuned base model adds at least one special attribute regarding the result of the segmentation, in particular the at least one bounding box, which depends on the at least one text prompt.
[0037] This allows for more precise and differentiated annotation of the sensor data. Segmentation enables the spatial mapping of objects within the sensor data, while classification using bounding boxes allows assignment to specific categories. By adding a specific attribute, depending on the text prompts, the base model can provide additional information about the objects, such as their size, shape, color, orientation, any actions performed, and whether the objects are R.413920.
[0038] - 6 - whether objects are static or dynamic, whether they are occluded, or whether they belong together. These detailed annotations can then be used to train machine learning models to achieve higher accuracy and performance. Segmentation and classification can be performed in separate steps or in a combined step. Furthermore, the annotation can also describe an action by road users to provide a basis for scene descriptions or trajectory planners.
[0039] Another possibility is that generating the annotations may also include:
[0040] Supplementing the annotations with metadata, where the metadata specifies properties that are specific to the sensor data, independent of the segmentation.
[0041] In other words, the base model can also add additional information, known as metadata, to the annotations. This metadata can describe properties that are not directly related to the segmentation of the sensor data, but rather are specific characteristics of the entire scene, such as weather conditions (heavy rain, snow, ground fog, and others), cloud cover, sun position, or similar factors. Furthermore, specific characteristics of the sensor data itself can be recorded. This additional information can include, for example, details about the time the sensor data was generated, the location of the data acquisition, or the type of sensor used.
[0042] Preferably, the invention may provide that the method further comprises:
[0043] Using the provided training dataset in training a machine learning model, wherein the machine learning model is trained for environmental perception, trajectory planning and / or scene description for a vehicle, in particular for classification based on an evaluation of sensor data based on pixels, wherein the sensor data are preferably image data.
[0044] This allows the machine learning model to be advantageously trained with the detailed training dataset according to the invention, resulting in an R.413920
[0045] - 7 - a more precisely trained machine learning model can be obtained. The scene description can supplement the environmental perception with a description of a current situation, i.e., in particular with a temporal component, whereby, for example, a given traffic scenario is described.
[0046] It is possible that the method according to the invention is used in a vehicle. The vehicle can be, for example, a motor vehicle and / or passenger vehicle and / or autonomous vehicle. The vehicle can have vehicle equipment, for example, for providing an autonomous driving function and / or a driver assistance system. The vehicle equipment can be designed to control the vehicle at least partially automatically and / or accelerate and / or brake and / or steer.
[0047] The machine learning model is trained on the training dataset, specifically for classification and / or object detection. Accordingly, the training can result in a trained machine learning model that can be used for classification and / or object detection, particularly in the context of environmental perception, trajectory planning, and / or scene description. Its use, and thus its inference capabilities, can be implemented, for example, in a vehicle. The input data points can be, for example, pixels from sensor data, which are primarily image data, or be based on these pixels, in order to perform the classification and / or object detection of the data points based on the pixels.The input data can include sensor and / or image data, which at least partially result from acquisition with a sensor, preferably a camera sensor, and / or which have been at least partially synthesized, thus in particular replicating the real data of a sensor. Specifically, it can be provided that the values of image points, preferably pixels, of the image data represent the environment of a sensor and / or a vehicle and / or a traffic scene. A classification, preferably image classification and / or object detection, based on these values can be provided. This enables, for example, the detection of objects in the traffic scene. The classification can also take the form of a semantic R.413920.
[0048] - 8 -
[0049] Segmentation (i.e., pixel- or area-based classification) and / or object detection may be provided. The image data can be, for example, images from a radar sensor, an ultrasonic sensor, a LiDAR sensor, and / or a thermal imaging camera. Accordingly, the images can also be presented as radar images, ultrasonic images, thermal images, and / or LiDAR images.
[0050] The invention also relates to a computer program, in particular a computer program product, comprising instructions which, when executed by a computer, cause the computer to execute the method according to the invention. Thus, the computer program according to the invention offers the same advantages as those described in detail with reference to a method according to the invention.
[0051] The invention also relates to a data processing device configured to execute the method according to the invention. The device can, for example, be a computer that executes the computer program according to the invention. The computer can have at least one processor for executing the computer program. Alternatively, a non-volatile data storage device can be provided in which the computer program is stored and from which the computer program can be read by the processor for execution.
[0052] The invention may also relate to a computer-readable storage medium which contains the computer program according to the invention and / or includes instructions which, when executed by a computer, cause the computer to execute the method according to the invention. The storage medium is, for example, designed as a data storage device such as a hard drive and / or non-volatile memory and / or a memory card. The storage medium can, for example, be integrated into the computer.
[0053] Furthermore, the method according to the invention can also be implemented as a computer-implemented method. Alternatively or additionally, at least one of the disclosed method steps can be computer-implemented and / or carried out automatically. R.413920
[0054] - 9 -
[0055] Further advantages, features, and details of the invention will become apparent from the following description, in which exemplary embodiments of the invention are described in detail with reference to the drawings. The features mentioned in the claims and in the description can each be essential to the invention individually or in any combination. The drawings show:
[0056] Fig. 1 shows a schematic visualization of a method, a device, a storage medium and a computer program according to exemplary embodiments of the invention.
[0057] Fig. 2 shows a schematic representation of a sensor in a vehicle, a basic model, a second basic model and a machine learning model according to exemplary embodiments of the invention.
[0058] Fig. 1 schematically shows a method 100, a device 10, a storage medium 15 and a computer program 20 according to exemplary embodiments of the invention.
[0059] Fig. 1 shows, in particular, an embodiment of a method 100 for the automated generation of annotations for sensor data to provide a training dataset for machine learning. In a first step 101, sensor data is provided, the sensor data resulting from the acquisition of at least one sensor 1. In a second step 102, a base model 50 is fine-tuned, wherein the base model 50 is fine-tuned to generate annotations for the sensor data based on at least one text prompt, the at least one text prompt representing a distinction between classes based on a textual description. In a third step 103, the annotations for the sensor data are generated by the fine-tuned base model using the classes. In a fourth step 104, the training dataset is provided based on a combination of the sensor data with the generated annotations.413920.
[0060] - 10 -
[0061] Fig. 2 schematically shows a sensor 1 in a vehicle 2, a basic model 50, a second basic model 51 and a machine learning model 52 according to embodiments of the invention.
[0062] The present invention describes in particular a method for the automated generation of annotations (“jabels”) for sensor data within the framework of machine learning, which in particular uses a language space of foundation models. In contrast to existing methods that rely on fine-tuning with already existing annotated data, this method uses the language space of the foundation model in particular to generate annotations directly from textual descriptions.
[0063] The base model is initially trained, preferably using textual descriptions in at least one text prompt, on a specific task, i.e., in particular, on generating annotations for the sensor data. These textual descriptions serve, in particular, as instructions for the base model on how annotations should be created and which attributes in the sensor data are relevant.
[0064] Advantages of this method include the following: Annotations can be quickly adapted and varied by changing at least one text prompt. The need for extensive annotated training datasets is eliminated. Using textual descriptions in the at least one text prompt can allow for a more precise definition of class features compared to methods based solely on similarity scores in an embedding space.
[0065] The method described in the exemplary embodiments makes it possible to modify and create annotations with very little effort. In the prior art, existing annotations are usually used when fine-tuning the base models. However, this requires that a large amount of annotated data already exists. In the method described in the exemplary embodiments, fine-tuning preferably relies solely on the language space of the base model. (See R.413920)
[0066] - 11 - Manual annotation may already be possible using annotation instructions (Jabel guides), which describe how to create annotations and also provide a distinction between different object classes or other special features, such as tires or open doors on the vehicle, should be handled.
[0067] In a first step, according to one implementation example, sensor data, such as a video image, is provided or read in. Subsequently, a basic model can be trained or fine-tuned to the corresponding task based on the provided sensor data; this involves, in particular, generating annotations for the sensor data based on a text prompt with a textual description. The annotations can then be combined with the sensor data to provide the training dataset.
[0068] According to the examples, the basic model can be used to create annotation instructions (label guides) so that the annotations are generated as precisely and consistently as possible.
[0069] According to the exemplary implementations, a second base model can be used to verify the annotations generated by the first base model. For example, the second base model can also generate annotations based on the same text prompt, and the two generated annotations can then be compared. If a discrepancy is found, the first base model can perform another iteration of generating the annotations.
[0070] According to the examples, the generation of annotations can be improved by using the basic model to optimize the text prompt, i.e., to define the textual description as accurately as possible.
[0071] Within the scope of the present invention, the following aspects are addressed in particular: Classes are preferably detailed based on a text prompt as input for the base model. R.413920
[0072] - 12 -
[0073] The text prompt, in particular, distinguishes classes based on a textual description and not solely within the embedding space using a similarity score based on similar images. This description preferably includes relevant attributes in the text to differentiate classes. This reduces the need for example images or allows for increased image variance without requiring a large number of example images. In an advantageous embodiment, the text prompt can also be transformed into an embedding space. In the method according to the exemplary embodiments, segmentation can first take place, whereby, for example, at least one bounding box or a comparable segmentation result can be created automatically or manually.Based on the created bounding box (at least one), a classification can then be performed. Specifically, attributes can be added by the base model, which can be described depending on the class, such as whether a vehicle door is open. Furthermore, the base model can also be used to add metadata, such as road conditions, which are preferably not dependent on the segmentation but are generally specific to the sensor data, i.e., they apply to an entire image within the sensor data.
[0074] The preceding explanation of the embodiments describes the present invention solely by way of examples. Naturally, individual features of the embodiments can be freely combined with one another, provided this is technically feasible, without departing from the scope of the present invention.
Claims
R.413920 - 13 - Claims 1. Method (100) for the automated generation of annotations for sensor data to provide a training dataset for machine learning, comprising: Providing (101) sensor data, wherein the sensor data result from the acquisition of at least one sensor (1), fine-tuning (102) a base model (50), wherein the base model (50) is fine-tuned to generate annotations for the sensor data on the basis of at least one text prompt, wherein the at least one text prompt represents a distinction of classes based on a textual description, Generating (103) the annotations for the sensor data by the finely tuned base model using the classes, providing (104) the training data set based on a combination of the sensor data with the generated annotations.
2. Method (100) according to claim 1, characterized in that the method (100) further comprises: Creating an annotation instruction using the fine-tuned base model (50), Using the created annotation instruction as part of generating (103) the annotations. R.413920 - 14 - 3. Method (100) according to one of the preceding claims, characterized in that the method (100) further comprises: Checking the generated annotations by the base model (50) and / or a second base model (51) on the basis of at least one defined criterion.
4. Method (100) according to one of the preceding claims, characterized in that the method (100) further comprises: Optimizing at least one text prompt for generating (103) the annotations using the finely tuned basic model (50).
5. Method (100) according to one of the preceding claims, characterized in that the method (100) further comprises: Transforming the at least one text prompt into an embedding space, wherein the generation (103) of the annotations is carried out on the basis of the transformed at least one text prompt.
6. Method (100) according to one of the preceding claims, characterized in that the generation (103) of the annotations comprises: Performing a segmentation based on the sensor data using the finely tuned base model (50), Performing a classification based on a result of the segmentation, wherein the fine-tuned basic model (50) adds at least one special attribute with respect to the result of the segmentation, which depends on the at least one text prompt. R.413920 - 15 - 7. Method (100) according to claim 6, characterized in that the generation (103) of the annotations further comprises: Supplementing the annotations with metadata, where the metadata specifies properties that are specific to the sensor data, independent of the segmentation.
8. Method (100) according to one of the preceding claims, characterized in that the method (100) further comprises: Using the provided training dataset in training a machine learning model (52), wherein the machine learning model (52) is trained for environmental perception, trajectory planning and / or scene description for a vehicle (2), in particular for classification based on an evaluation of the sensor data based on pixels.
9. Computer program (20), comprising instructions which, when the computer program (20) is executed by a computer (10), cause it to execute the method (100) according to one of the preceding claims.
10. Device (10) for data processing, which is configured to carry out the method (100) according to any one of claims 1 to 8.
11. Computer-readable storage medium (15) comprising instructions which, when executed by a computer (10), cause it to perform the steps of the method (100) according to any one of claims 1 to 8.