Automatic annotation of road models for autonomous driving systems

Abstract

Automatic annotation of road models of autonomous driving systems is disclosed. Specifically, methods and related aspects for generating road model annotations of autonomous driving systems are disclosed. The method includes obtaining a first data set comprising 3D polyline annotations of a set of road model objects in a specified environment. A second data set is obtained comprising one or more camera images depicting the specified environment over a period of time, and a subset of camera images is selected from the one or more camera images based on the first data set. A cue is generated to a trained base model, the cue comprising instructions for the trained base model to assign attributes to each road model object annotated in the obtained first data set. The selected subset of camera images of the second data set is fed to the trained base model, the generated cue is fed to the trained base model, and an output from the trained base model is resolved.

Description

Technical Field

[0001] The disclosed technology relates to methods and systems for generating road model annotations for autonomous driving systems. Specifically, but not exclusively, the disclosed technology relates to using a trained base model to automatically generate attribute annotations for road model objects based on 3D polyline annotations. Background Technology

[0002] In recent years, autonomous driving systems have attracted considerable attention, with increasingly higher levels of autonomy being deployed in consumer vehicles. These systems rely on a range of sensors, computing platforms, and control algorithms to perceive the vehicle's surroundings, plan driving maneuvers, and execute corresponding actions in real time. As the industry moves from driver assistance features (such as adaptive cruise control and lane keeping) to higher levels of autonomy (according to SAE international standards Level 3, 4, or 5), safety and reliability remain primary objectives.

[0003] Autonomous driving systems rely on various sensors (e.g., cameras, LiDAR, radar) to perceive and interpret the vehicle's surroundings. Lane detection is one of the key perception tasks necessary for autonomous driving functions such as lane keeping, adaptive cruise control, and highway automated driving assistance. Accurate lane detection requires a large amount of well-annotated training data, enabling machine learning models to robustly identify and track lane markings under diverse driving conditions.

[0004] Traditionally, annotating road model objects such as lane markings in raw sensor data is a labor-intensive, manual process. Annotators will examine images or sensor frames, delineate lane boundaries, and label each boundary segment for subsequent training of perception algorithms. Manual annotation presents significant challenges in terms of time consumption, cost, and managing inconsistencies and accuracy.

[0005] Therefore, there is still a need to improve systems and methods to effectively generate accurate annotations for road model objects such as lane dividers (also known as lane markings) in large datasets. Summary of the Invention

[0006] The techniques disclosed herein are intended to mitigate, alleviate, or eliminate one or more of the aforementioned defects and disadvantages in the prior art, in order to address various problems related to the time consumption and cost of providing annotations of the properties of various road model objects.

[0007] Various aspects and embodiments of the disclosed technology are defined in the following and accompanying independent and dependent claims.

[0008] A first aspect of the disclosed technology includes a computer-implemented method for generating road model annotations for an autonomous driving system. The computer-implemented method includes: obtaining a first dataset comprising 3D polyline annotations of a set of road model objects in a specified environment. The first dataset includes a 3D point cloud represented in 3D space, captured by a LiDAR device in the specified environment over a time period, indicating the presence of the set of road model objects. The method further includes: obtaining a second dataset comprising one or more camera images depicting the specified environment over the time period; and selecting a subset of the camera images from the one or more camera images based on the first dataset. The method further includes: generating a cue for a trained base model. The cue includes instructions for the trained base model to assign attributes to each road model object annotated in the obtained first dataset, and the instructions are automatically generated based on the obtained first dataset. The method further includes: feeding the selected subset of camera images from the second dataset to the trained base model, feeding the generated cue to the trained base model, and parsing the output from the trained base model. The output is parsed to generate: a file in a Structured Data Interchange format, wherein the file includes annotations indicating the assigned attributes of each road model object in the group of road model objects; and one or more annotated images, the one or more annotated images including annotations indicating the assigned attributes of each road model object in the group of road model objects in one or more camera images.

[0009] The second aspect of the disclosed technology includes a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform any of the methods according to the embodiments of the first aspect disclosed herein. This aspect of the disclosed technology has similar advantages and preferred features as the other aspects.

[0010] The third aspect of the disclosed technology includes a (non-transitory) computer-readable storage medium containing instructions that, when executed by a computing device, cause the computing device to perform any of the methods in the embodiments of the first aspect disclosed herein. This aspect of the disclosed technology has similar advantages and preferred features as the other aspects.

[0011] This aspect of the disclosed technology has similar advantages and preferred features as other aspects.

[0012] As used herein, the term "non-transitory" is intended to describe computer-readable storage media (or "memory") that do not propagate electromagnetic signals, but is not intended to otherwise limit the types of physical computer-readable storage devices covered by the phrase computer-readable media or memory. For example, the terms "non-transitory computer-readable media" or "tangible memory" are intended to cover types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in a non-transitory form on a tangible computer-accessible storage medium can be further transmitted via a transmission medium or a signal, such as an electrical signal, electromagnetic signal, or digital signal, that can be transmitted via a communication medium such as a network and / or a wireless link. Therefore, as used herein, the term "non-transitory" is a limitation on the medium itself (i.e., tangible, not a signal), rather than a limitation on the persistence of data storage (e.g., RAM vs. ROM).

[0013] A fourth aspect of the disclosed technology includes a system for generating road model annotations for an autonomous driving system. The system includes one or more processors and one or more memory storage areas including program code, wherein the one or more memory storage areas and program code are configured to use the one or more processors to cause the system to: obtain a first dataset comprising 3D polyline annotations of a set of road model objects in a specified environment. The first dataset includes a 3D point cloud represented in 3D space, captured by a LiDAR device in the specified environment over a time period, indicating the presence of the set of road model objects. The one or more memory storage areas and program code are further configured to use the one or more processors to cause the system to: obtain a second dataset comprising one or more camera images depicting the specified environment over the time period; and, based on the first dataset, select a subset of the camera images from the one or more camera images. The one or more memory storage areas and program code are further configured to use the one or more processors to cause the system to: generate cues for a trained base model. Here, the cues include instructions for the trained base model to assign attributes to each road model object annotated in the obtained first dataset, and these instructions are automatically generated based on the obtained first dataset. One or more memory storage areas and program code are further configured to enable the system, using one or more processors, to: feed a selected subset of camera images from a second dataset to a trained base model; feed generated cues to the trained base model; and parse the output from the trained base model. This output is parsed to generate: a file in a structured data exchange format, wherein the file includes annotations indicating the assigned attributes of each road model object in the set of road model objects; and one or more annotated images, the one or more annotated images including annotations indicating the assigned attributes of each road model object in the set of road model objects in one or more camera images. Similar advantages and preferred features exist for this aspect of the disclosed technique as for other aspects.

[0014] The fifth aspect of the disclosed technology includes a server comprising any of the systems described in the embodiments of the fourth aspect disclosed herein. This aspect of the disclosed technology shares similar advantages and preferred features with the other aspects.

[0015] The sixth aspect of the disclosed technology includes a cloud environment comprising one or more remote servers according to any of the embodiments of the fifth aspect disclosed herein. This aspect of the disclosed technology has similar advantages and preferred features as the other aspects.

[0016] The disclosed aspects and preferred embodiments may be appropriately combined with each other in any manner that is obvious to those skilled in the art, such that one or more features or embodiments disclosed with respect to one aspect may also be regarded as embodiments disclosed with respect to another aspect or another aspect.

[0017] One advantage of some implementations is that they can achieve significant time savings for generating large annotated datasets of road model objects, which can be used to train various models for autonomous driving systems.

[0018] One advantage of some embodiments is that large datasets in an accurately annotated dataset of road model objects can be obtained, thereby providing means to improve and / or validate machine learning algorithms, and especially machine learning algorithms for perceiving and understanding road geometry information.

[0019] One advantage of some embodiments is that, as large amounts of training data become available, the performance and robustness of models used to understand road geometry information in autonomous driving systems can be improved in a faster and more cost-effective manner.

[0020] Further embodiments are defined in the dependent claims. It should be emphasized that, when used in this specification, the term "comprising / including" is used to indicate the presence of the stated features, integers, steps, or components. It does not exclude the presence or addition of one or more other features, integers, steps, components, or groups thereof.

[0021] These and other features and advantages of the disclosed technology will be further illustrated below with reference to the embodiments described herein. Attached Figure Description

[0022] The foregoing aspects, features, and advantages of the disclosed technology will be more fully appreciated through the following illustrative and non-limiting detailed description of exemplary embodiments of the present disclosure, taken in conjunction with the accompanying drawings, in which:

[0023] Figure 1 This is a schematic flowchart representation of a method for generating road model annotations for an autonomous driving system according to some embodiments.

[0024] Figure 2 These are two 3D point cloud scans from a top-down perspective, one without 3D polyline annotations and the other with 3D polyline annotations.

[0025] Figure 3 This is a schematic block diagram representation of a system for generating road model annotations for an autonomous driving system, according to some embodiments.

[0026] Figure 4This is a schematic block diagram representation of a system for generating road model annotations for an autonomous driving system according to some embodiments, as well as a schematic illustration of a server including such a system and a cloud environment including multiple such servers. Detailed Implementation

[0027] This disclosure will now be described in detail with reference to the accompanying drawings, in which some exemplary embodiments of the disclosed technology are illustrated. However, the disclosed technology may be embodied in other forms and should not be construed as limited to the exemplary embodiments disclosed. The exemplary embodiments disclosed are provided to fully convey the scope of the disclosed technology to those skilled in the art. Those skilled in the art will appreciate that the steps, services, and functions described herein can be implemented using separate hardware circuitry, software that works in conjunction with a programmed microprocessor or general-purpose computer, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or one or more digital signal processors (DSPs).

[0028] It will also be appreciated that when this disclosure is described in terms of method, it can also be embodied as a device including one or more processors and one or more memories coupled to the one or more processors, wherein computer code is loaded to implement the method. For example, in some embodiments, the one or more memories may store one or more computer programs that, when executed by the one or more processors, cause the device to perform the steps, services, and functions disclosed herein.

[0029] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. It should be noted that, as used in the specification and appended claims, unless the context explicitly states otherwise, the words “a,” “the,” and “the” are intended to mean the presence of one or more elements. Thus, for example, in some contexts, references to “a unit” or “the unit” may refer to more than one unit, etc. Furthermore, the words “comprising,” “including,” and “containing” do not exclude other elements or steps. It should be emphasized that, when used in this specification, the term “comprising” is used to indicate the presence of the stated features, integers, steps, or components. It does not exclude the presence or addition of one or more other features, integers, steps, components, or groups thereof. The term “and / or” should be interpreted as meaning “both” and that each is optional.

[0030] It will also be understood that although the terms first, second, etc., may be used herein to describe various elements or features, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first signal may be referred to as a second signal, and similarly, a second signal may be referred to as a first signal, without departing from the scope of the embodiments. Both the first signal and the second signal are signals, but they are not the same signal.

[0031] As used herein, the term "in response to" can be understood, depending on the context, as meaning "when," "as soon as," or "if." Similarly, depending on the context, the phrases "in response to determining...", "if determining...", "when determining...", or "in the case of..." can be interpreted as meaning "as soon as determining...", "as soon as the occurrence of an event is detected and identified...", or "in response to detecting the occurrence of an event." Accordingly, depending on the context, the phrase "in response to X equaling Y" can be interpreted as "when X equals Y," "if X equals Y," "when determining X equals Y," or "in response to detecting / determining X equals Y."

[0032] The term "acquire" is to be interpreted broadly herein and encompasses the receiving, retrieving, collecting, acquiring, etc., directly and / or indirectly between two entities configured to communicate with each other or further with other external entities. However, in some embodiments, the term "acquire" will be interpreted as determining, deriving, forming, calculating, etc. In other words, acquiring the attitude of a vehicle can encompass determining or calculating the vehicle's attitude based on, for example, GNSS data and / or perception data and map data. Therefore, as used herein, "acquire" can indicate receiving parameters at a first entity / unit from a second entity / unit, or determining parameters at a first entity / unit, for example, based on data received from another entity / unit.

[0033] Overview

[0034] As mentioned above, training machine learning algorithms such as artificial neural networks used in autonomous driving systems typically requires a large amount of annotated or “labeled” data. For perception models whose task is to predict the properties of various road model objects, the required training data could, for example, include annotations on a large number of camera images defining the properties of lane dividers, pedestrian crossings, grass edges, curbs, snow, guardrails, walls, gravel, sand, soil, ditches, etc.

[0035] The performance a neural network can achieve in solving its objective depends on the quality and size of the dataset it is trained on. However, obtaining large amounts of high-quality annotated data is very time-consuming and / or expensive. This is especially true for annotating attributes such as color (e.g., white or yellow) and type (e.g., virtual, dashed, or solid) of road model objects (such as lane dividers or road markings). This is because annotating the geometry of road model objects (e.g., lane dividers) is relatively straightforward, for example, by drawing polylines in 3D coordinate space. However, selecting and inputting these attributes of road model objects requires the annotator to undergo rigorous training to minimize errors. Even with such training, errors still occur, necessitating quality assessment (QA) reviews, which further increase the cost and time required to provide the annotated dataset.

[0036] Therefore, the embodiments in this paper provide a solution for annotating the attributes of road model objects in large datasets. This solution leverages the latest advancements in the underlying model in a novel and improved manner to reduce the time and cost associated with annotating large datasets that can be used to train machine learning algorithms to be used in autonomous driving systems. More specifically, these annotated datasets can be used to train machine learning algorithms to predict or assign attributes of various road model objects, such as the color and type of lane dividers (also referred to as "lane markings") present in camera images.

[0037] Furthermore, with the recent and rapid development of base models, it's easy to assume that one can prompt base models to detect and classify all types of road model objects for training data to generate various perception models used in autonomous driving systems. However, while this may be true for many "everyday objects" such as people, cats, or dogs in photographs, these base models have proven inadequate in providing accurate annotations of the attributes of road model objects depicted in camera images. This unsatisfactory result persists regardless of the assigned task, meaning that whether the base model is prompted to generate bounding boxes (object detection), semantic segmentation, or keypoint extraction, the output from the base model is consistently unsatisfactory in assigning attributes to road model objects. Clearly, existing base models struggle to correctly identify and annotate road model objects such as lane dividers, and therefore, cannot (at least not in an obvious and straightforward way) be relied upon to assign attributes to road model objects such as lane dividers.

[0038] Therefore, these foundational models are not directly applicable to geometric understanding in complex real-world driving scenarios. They are primarily designed for general tasks such as general object detection, depth estimation, or segmentation. Similarly, while many foundational models can comprehensively interpret images and interactively respond to text prompts, they lack the ability to detect specific geometric information and perform in-depth reasoning about real-world driving scenarios.

[0039] This highlights the need to bridge the gap between the demands of understanding real-world driving scenarios and the current capabilities of the underlying models. Addressing this gap is crucial for improving the efficiency and accuracy of the annotation process in automotive applications.

[0040] To this end, embodiments in this paper propose using 3D polyline annotations of a set of road model objects represented by a 3D point cloud in 3D space to automatically generate cues for a trained base model, thereby assigning attributes to that set of road model objects in associated camera images. Accordingly, suitable camera images are selected by cross-referencing the 3D polyline annotations with a large number of camera images captured during the same time period as the 3D point cloud; these suitable camera images are provided as input to the trained base model along with the generated cues. The output from the trained base model is then parsed to obtain attribute annotations in a structured data exchange format (e.g., JSON) and to obtain an annotation image that visualizes the annotations of the base model. The annotation image can be used later for quality assurance.

[0041] definition

[0042] In the context of this disclosure, the term "annotation" refers to the process of labeling or annotating specific objects, features, or attributes within data (typically images or videos) to create a labeled dataset for training and evaluating machine learning models. Annotations provide truth values ​​or reference information that allows algorithms to learn objects, regions of interest, or features within the data and make predictions about them.

[0043] For example, in object detection tasks, annotation involves drawing bounding boxes around objects of interest in an image or video and / or indicating the object's category (e.g., car, pedestrian, traffic sign). This labeled data is used to train ML algorithms to identify and locate objects in new data. Furthermore, for semantic segmentation, each pixel in an image is labeled with a category identifier, thus assigning a category to each part of the image. This fine-grained labeling helps ML algorithms understand the layout and category of objects within an image. Additionally, instance segmentation combines object detection and semantic segmentation. It not only identifies object categories but also assigns unique identifiers to each instance of an object, enabling the model to distinguish individual objects of the same class. In image classification tasks, each image in the dataset is labeled with a single class or category (e.g., "cat" or "dog"). This allows the model to learn to classify new, unlabeled images into predefined categories.

[0044] In the current context, "machine learning algorithm" refers to a computational model or set of techniques used to enable computers to solve tasks, such as enabling a vehicle's perception system to interpret and understand its surrounding environment. Perception tasks in Automated Driving Systems (ADS) involve the vehicle's ability to detect and identify objects, obstacles, road signs, lane markings, pedestrians, other vehicles, and various environmental conditions. ADS uses machine learning algorithms to process sensor data, such as data from cameras, LiDAR, radar, and other sensors, to make informed decisions about how to navigate safely. These algorithms use data-driven techniques to analyze and classify objects, understand road geometry, predict the movement of other road users, and / or assess potential risks in real time. Common types of machine learning algorithms used in ADS perception tasks include deep neural networks, convolutional neural networks (CNNs) (e.g., for camera image processing, LiDAR output processing, etc.), recurrent neural networks (RNNs) (e.g., for sequential data), and various other techniques like support vector machines (SVMs) and decision trees.

[0045] In some embodiments, the machine learning algorithm (also referred to as the machine learning model) is implemented using suitable publicly available software development machine learning code elements (such as code elements available in PyTorch, TensorFlow / Keras, JAX, or any other suitable software development platform) in any suitable manner known to those skilled in the art.

[0046] In the current context, an Automated Driving System (ADS) refers to a complex combination of hardware and software components designed to control and operate a vehicle without direct human intervention. ADS technology aims to automate various aspects of driving, such as steering, acceleration, deceleration, and monitoring of the surrounding environment. The primary goal of ADS is to improve the safety, efficiency, and convenience of transportation. Classified by standards like SAE J3016, ADS can range from basic driver assistance systems to highly advanced autonomous driving systems, depending on their level of automation. These systems utilize various sensors, cameras, radar, lidar, and powerful computer algorithms to perceive the environment and make driving decisions. The specific capabilities and features / functions of ADS can vary greatly, from systems providing limited assistance to those capable of independently handling complex driving tasks under specific conditions.

[0047] While Advanced Driver Assistance Systems (ADAS) do not necessarily provide complete autonomy, they are technologies that assist the driver during driving. ADAS features often serve as building blocks for ADS. Examples include adaptive cruise control, lane keeping assist, automatic emergency braking, and parking assist. They improve safety and convenience but typically require some degree of human supervision and intervention. Autonomous Driving (AD), on the other hand, is a technology designed to control and navigate a vehicle without human supervision. Accordingly, it can be said that the difference between ADAS and AD lies in the level of autonomy and control. ADAS systems are designed to assist and support the driver, while AD systems currently aim for complete control of the vehicle without the need for continuous human supervision. Accordingly, AD aims for a higher level of autonomy (e.g., Levels 4 and 5 according to SAE International Standards), where the vehicle is able to operate independently in most or all driving scenarios without human intervention. For the sake of brevity, the term "ADS" is used herein as a general term encompassing both ADAS and AD. Furthermore, in the current context, ADS functions or ADS features can be understood as specific functions or features of the entire ADS stack, such as highway navigation features, traffic jam navigation features, route planning features, etc.

[0048] In the context of this disclosure, the term "3D point cloud" refers to a collection of data points in a three-dimensional coordinate space, where each individual data point is associated with at least three spatial coordinates. These data points can be generated or acquired by one or more sensors (such as LiDAR, radar, or stereo cameras) that capture positional and depth information about objects and their surrounding environment. In autonomous driving systems, 3D point clouds provide a spatial representation of the vehicle's environment, enabling functions such as object detection, localization, and path planning by depicting the shape, position, and relative distance of relevant features within the driving scene.

[0049] In the context of this disclosure, the term "3D polyline" can be understood as a piecewise linear geometric primitive approximating a road object (such as lane dividers or curb edges) within a 3D coordinate frame, derived from suitable sensor data such as LiDAR or stereo point cloud data. More specifically, a 3D polyline can be composed of an ordered list of vertices P=[(x0, y0, z0), (x1, y1, z1), ... (x n , y n , z n The geometry can be represented in a global coordinate frame or a sensor local coordinate frame, with line segment connections implicitly linking consecutive vertices within it. In practice, the geometry can be stored and exchanged as 3D line strings or equivalent vector primitives in formats such as OGC Simple Features, OpenDRIVE, Lanelet2, PCD / JSON, or proprietary HD-map modes. During annotation, humans or algorithms place vertices directly onto the point cloud where road model objects (e.g., physical lane paint) appear.

[0050] In the context of this disclosure, the term "base model" refers to a large-scale machine learning model trained on large and often extensive datasets to capture fundamental patterns, representations, or features relevant to a wide variety of potential downstream tasks. This type of model typically employs advanced deep neural network architectures, such as transformers, to learn from diverse sources of textual, visual, or multimodal data. Due to the breadth and depth of their training, base models can be easily adapted or fine-tuned for specialized applications without requiring the creation of entirely new model architectures for each use case. Examples of base models include, but are not limited to, (multimodal) large language models (LLMs) such as GPT and BERT, or visual models such as SAM or ViT.

[0051] In the context of this disclosure, the term "cue" refers to data input typically provided to an underlying model in the form of text, a query, or other structured instructions for the purpose of eliciting a specific output or behavior. Depending on the nature of the underlying model (e.g., language-based, vision-based, or multimodal), a cue may include textual descriptions, partial images, and / or other relevant data. The cue is used to guide or focus attention on the model's underlying general representations and capabilities, enabling it to produce customized responses, generate new content, or perform purposeful tasks according to user instructions.

[0052] In the context of this disclosure, the term "parsing" refers to the process of taking the raw, often loosely formatted text or tensor output of a machine learning model and analyzing it to recover fragments of explicit information that the model intends to convey, and then rewriting these fragments into a well-defined target format. For example, the target format could be a structured data exchange format file (e.g., JSON, XML, protocol buffers, or domain-specific HD-map files). Specifically, it can involve (1) identifying syntactic markers or positional conventions (notations, comma-separated fields, key-value pairs, shape indexes, or special sequence terminators) in the model's output; (2) extracting each value to convert it to the correct data type and unit; and (3) assembling those cleaned values ​​according to the field names, nesting, and order specified by the exchange specification. In short, parsing is a controlled transformation from the model's "writer-friendly" output to a "machine-to-machine" structure with guaranteed syntax and semantics, enabling downstream tools to use the data without guesswork or fragile string manipulation.

[0053] In the context of this disclosure, the term "road model object" refers to the geometric features used to represent and characterize roads within an autonomous driving system. These features include lane dividers (whether physical lane markings or virtual boundaries) that explain the drivable or impassable portions of the road, as well as road edges (e.g., curbs, grass, gravel, ditches). These road model objects capture fundamental spatial features such as position, shape, or curvature, and together form the infrastructure upon which autonomous driving functions (e.g., path planning, localization, or maneuvering) are based.

[0054] In the context of this disclosure, the term "attribute" as used with respect to a "road model object" refers to any feature or attribute of a road model object that can be inferred from a visual inspection of the road model object. Examples include, but are not limited to, the color (e.g., white, yellow, orange, red, blue, green) and type (e.g., solid line, short dashed line, long dashed line, dividing line) of lane dividers, and other similar visual identifiers of road edges or boundaries, or whether the road model object acts as a dividing line defining opposite lanes. These attributes provide additional descriptive information about the road model object, thereby informing the autonomous driving system of the nature, condition, or appearance of the road model object.

[0055] Example Implementation

[0056] Figure 1This is a schematic flowchart representation of a method S100 for generating road model annotations for an autonomous driving system according to some embodiments. Preferably, method S100 is a computer-implemented method S100 executed by one or more processors of a computer. More specifically, the computer may, for example, include one or more processors and one or more memories coupled to the one or more processors, wherein the one or more memories store one or more programs that, when executed by the one or more processors, perform the steps, services, and functions of the method S100 disclosed herein.

[0057] Method S100 includes: obtaining a first dataset (S101) comprising 3D polyline annotations of a set of road model objects in a specified environment. More specifically, the first dataset includes a 3D point cloud represented in 3D space, captured by a LiDAR device in the specified environment over a time period, indicating the presence of the set of road model objects. In other words, the obtained first dataset (S101) can be understood as a 3D representation of annotations of a traffic scene captured by a suitable sensor device (e.g., a LiDAR device), where the point cloud represents various road model objects such as lane dividers (or lane markings), pedestrian crossings, road boundaries, etc. The annotations delineate general geometric information of the road model objects, such as the location and orientation of lane dividers present in the traffic scene, in the form of 3D polyline annotations.

[0058] Now briefly switch to Figure 2 The leftmost image shows an example of a traffic scene captured by a LiDAR device, with point clouds indicating the location and shape / geometric information of lane dividers, crosswalks, and other road model objects in the traffic scene. The rightmost image then shows 3D polyline annotations of the lane dividers.

[0059] Return to Figure 1 Method S100 further includes: obtaining a second dataset (S102), which includes one or more camera images depicting the specified environment within the time period. In other words, method S100 further includes: obtaining a set of camera images (S102) depicting the same traffic scene during the same time period as the traffic scene captured by the LiDAR device. The camera images may be captured, for example, by an onboard monocular camera of the vehicle that collected the first dataset.

[0060] Furthermore, method S100 includes: selecting a subset of camera images S103 from one or more camera images based on a first dataset. More specifically, the purpose of selecting the subset of camera images S103 is to select road model objects represented by 3D polyline annotations in the first dataset as visible images, so that their attributes can be automatically labeled or annotated. Therefore, suitable reference images can be extracted from the second dataset by using the relative pose between the LiDAR device (capturing 3D point clouds) and the camera (capturing camera images) and the timestamps associated with each dataset.

[0061] More specifically, the relative pose between the LiDAR and the camera is known when the first and second datasets are created. Therefore, it is possible to calculate the camera pose of each image in one or more camera images within the 3D space of the 3D point cloud. Thus, for a given 3D scene (as captured by LiDAR), the position and orientation of each image are essentially known. Next, given the 3D point cloud and 3D polyline annotations, and using the camera's extrinsic and intrinsic parameters, the 3D polyline annotations can be projected onto each image in one or more camera images of the second dataset to see if the road model object marked by the 3D polyline annotation is captured within that image (i.e., within the field of view at the time point the camera captured the image). Next, S103 can be automatically selected from all or some images representing the road model object marked by the 3D polyline annotation. Furthermore, in addition to the "visibility criterion" (the marked road model object is represented in the selected images), another criterion can be used to select images with some reasonable interval distance / time in S103, such that other road users do not obstruct the same area of ​​the road.

[0062] Method S100 further includes generating S104 a hint for the trained base model (e.g., a multimodal LLM). This hint includes instructions for the trained base model to assign attributes to each road model object annotated in the obtained first dataset (by 3D polylines). Furthermore, these instructions are automatically generated based on the obtained first dataset. In other words, method S100 includes automatically generating a hint (e.g., a text hint) for the trained base model using the 3D polyline annotations of the first dataset.

[0063] More specifically, the inventors recognized that in order to provide sufficient information to the trained base model to infer the input images of road model objects whose attributes will be annotated, it is necessary to know the number of road model objects in the images. For example, regarding lane divider annotation, the number of lane dividers present in a given camera image can be automatically determined by using 3D polyline annotation, and this information can be used to automatically generate sufficiently detailed cues for the trained base model. For example, if there are three lane dividers in one or more camera images to be annotated, the generated cue S104 could include "There are 3 lane dividers in this image" or "There are 3 lane dividers in these images." The number 3 can be automatically retrieved from the first dataset obtained in S101. The cue generation S104 can be a rule-based functional and / or machine learning model trained to output appropriate cues given a first dataset as input. In some embodiments, generating S104 includes the number of road model object instances and a description of the attributes to be determined.

[0064] By leveraging these cues, the trained base model can generate structured responses that are easily parsed and seamlessly integrated into the annotation pipeline for road model annotation in autonomous driving systems. The proposed method reduces the required manual labor while ensuring consistency and scalability in the generated annotations. More specifically, the proposed architecture has the potential to identify many attributes crucial for autonomous vehicles' ability to reason about their environment. For example, dashed lane dividers indicate permission to change lanes, while solid lane dividers with two dashed lines on either side indicate restrictions on crossing lanes. Additional attributes that can be determined include, but are not limited to, cardinality (e.g., single or double lines) and whether the lane divider functions as a dividing line.

[0065] An example of a suitable cue for querying the trained base model to determine the color of a road model object in the form of lane dividers could be:

[0066] This image contains 3 lane dividers. What color is the leftmost lane divider? Please reply using a Python dictionary format such as {"leftmost_lane_delimiter":colour}. The color can be one of the following: "white", "yellow", "orange", "red", "blue", "green", "other", "virtual", or "uncertain". If you are unsure of the color, please use "other". If the lane divider is obscured or blurred, please use "uncertain".

[0067] Another example of a suitable hint for querying the trained base model to determine the type of a road model object in the form of lane dividers could be:

[0068] This image contains 3 lane dividers. What type is the leftmost lane divider? Please reply using a Python dictionary format such as {"leftmost_lane_delimiter":lane_delimiter_type}. Lane divider types can only be one of the following: "solid line", "short dashed line", or "long dashed line".

[0069] In some embodiments, generating S104 to provide a hint to the trained base model includes: retrieving a hint template in S105; and populating the retrieved hint template in S106 based on the first dataset obtained in S101 to form instructions for the trained base model to assign attributes to each road model object annotated in the first dataset obtained in S101.

[0070] Referring to the first example above, the template can essentially be the entire cue, except for the number "3" indicating the number of lane dividers in the image. Here, the number can be represented as a variable in the template determined using the first dataset obtained in S101.

[0071] Method S100 further includes: feeding a subset of camera images from the second dataset selected in S107 (S103); and feeding the generated cue in S104 to the trained base model in S108. Accordingly, the trained base model processes the cue and the selected subset of camera images, and method S100 further includes parsing the output from the trained base model in S109. More specifically, the output in S109 is parsed to generate a structured data exchange format file and one or more annotated images including annotations indicating the attributes assigned to each road model object in the set of road model objects in the one or more camera images. The structured data exchange format file includes annotations indicating the attributes assigned to each road model object in the set of road model objects. The structured data exchange format file can be, for example, a JSON file, a YAML file, an XML file, a CSV file, a TSV file, or a binary format file.

[0072] Furthermore, in some embodiments, method S100 further includes: computing the regions of interest (ROIs) of the trained base model within a subset of the selected camera images of S103. More specifically, to avoid “wasting” computational resources analyzing irrelevant regions in the camera images, and also to reduce the risk of errors, it is beneficial for the trained base model to know the ROIs in each image within the subset of images selected in S103. Alternatively, the first dataset obtained in S101 can be fed into the trained base model, allowing it to infer any ROIs on its own.

[0073] As mentioned above, the assigned attribute is either the color or the type of each road model object. For example, the color could be white, yellow, red, orange, or even "virtual." Regarding the latter, "virtual" lane dividers could be observed, for example, on rural roads where no visible lane dividers exist, but could be inferred based on the location of the vehicle (the primary vehicle) on the road, combined with the locations of other vehicles traveling on the same road. Furthermore, the type of road model object could be, for example, dotted dashed lane dividers, solid lane dividers, dashed lane dividers, contour / rumble-band lane dividers, road edges defined by gravel, road edges defined by curbs, or road edges defined by grass. In some embodiments, when the road object is in the form of lane dividers, the assigned attribute could further include the cardinality of the lane dividers, such as double lines. An annotation, along with the type, could indicate that it is a double solid lane divider, a double dashed lane divider, or a combination of solid and dashed lane dividers.

[0074] Furthermore, in some embodiments, the subset of camera images selected in S103 includes a continuous sequence of camera images. Some properties of road model objects (e.g., long dashed lane dividers) are difficult or sometimes even impossible to determine from a single camera image. Therefore, multiple consecutive images can be fed to the trained base model, and the prompt generation S104 can also automatically take this into account by including in the prompt the number of images / frames that the base model needs to consider (i.e., the number of images included in the subset of selected S103).

[0075] The exact number of camera images and the spacing between them can depend on the vehicle's speed and desired properties (while acquiring the first and second sets of sensor data). For example, if the road model object is lane dividers, a long dashed line could be 39 meters long to ensure gaps exist between lane dividers in the base model, allowing for proper evaluation between two images / frames. Approximately 20 meters would be needed between individual frames to determine if a lane divider is a long dashed line. It should be noted that the figures are examples and the distance between lane dividers depends on the regulations of the country where the images are captured. Optionally, the output (annotated images) can also be a continuous sequence of annotated images.

[0076] Furthermore, the advantage of using annotated 3D point clouds for selecting camera images for S103 cross-reference is that the selected camera images for S103 can depict the gaps between lane dividers by discarding images where the expected gaps are occluded by objects (which will be obvious from the 3D point cloud). For example, since the road objects to be annotated may be occluded, an object detection model can be used to find potential occluding objects in the 3D point cloud, and thus the associated camera images can be omitted.

[0077] Furthermore, another advantage of feeding a continuous sequence of camera images into the trained base model is that the trained base model can more easily identify virtual lane dividers, especially in rural areas where there are no visible lane markings. In such cases, a base model such as a multimodal LLM can still infer whether a virtual center lane divider functions as a divider by analyzing continuous camera images. For example, if we imagine a two-way street without visible lane markings and observe vehicles traveling in opposite directions relative to themselves, the base model can infer the presence of a virtual lane divider based on the motion and relative positioning of the two vehicles.

[0078] Furthermore, in some embodiments, the output from the trained base model further includes a confidence score associated with each assigned attribute. As mentioned, the output of the parsing S109 from the base model includes a "visualization" component in the form of one or more annotated images, which are used to visualize the final annotations made by the base model. These annotated images can be used for quality assurance purposes to verify the correctness of the annotations in the structured data exchange format file. By further including confidence scores in the annotated images, the quality assurance process can be facilitated, as it directly indicates which annotations should be checked.

[0079] Executable instructions for performing these functions may optionally be included in a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0080] Figure 3This is a schematic block diagram representation of a system 10 for generating road model annotations for an autonomous driving system according to some embodiments. As indicated in the figure, system 10 includes control circuitry (e.g., one or more processors) 11 configured to perform the functions of the method S100 disclosed herein, wherein these functions may be included in a non-transitory computer-readable storage medium 12 or other computer program product configured to be executed by the control circuitry 11. In other words, system 10 includes one or more memory storage areas 12 containing program code, and the one or more memory storage areas 12 and the program code are configured to cause system 10 to perform method S100 according to any of the embodiments disclosed herein using one or more processors 11. System 10 further has one or more input / output (I / O) interfaces 13 for receiving input data and for outputting output data.

[0081] To better illustrate the embodiments disclosed herein, Figure 3 In this context, the control circuit 11 is represented as various “modules” or blocks, each of which is linked to one or more specific functions of the control circuit 11.

[0082] Accordingly, one or more memory storage areas and program code are configured to enable system 10 to obtain a first dataset using one or more processors. This first dataset includes 3D polyline annotations 303 of a set of road model objects in a specified environment. The first dataset includes a 3D point cloud 301 represented in 3D space, captured by a LiDAR device in the specified environment over a time period, indicating the presence of the set of road model objects. In the depicted example, the 3D polyline annotations indicate a traffic scene (see...). Figure 2 The presence of lane dividers in the data. 3D polyline annotations (303) can be provided manually, and the 3D point cloud annotation process is handled by... Figure 3 Box 310 in the diagram is indicated schematically.

[0083] Furthermore, one or more memory storage areas and program code are configured to enable system 10 to acquire a second dataset using one or more processors. This second dataset includes one or more camera images 302 depicting a specified environment within the time period. In other words, system 10 receives 3D polyline annotations (in 3D coordinate space) of the traffic scene and corresponding camera images of the same traffic scene.

[0084] One or more memory storage areas and program code are further configured, using one or more processors, to enable system 10 to select a subset of camera images from one or more camera images based on a first dataset. Accordingly, system 10 has a cross-reference image search block 360 for selecting camera images showing corresponding road model objects annotated with 3D polylines. As previously described, the cross-reference image search block can be configured to select camera images in which the road model objects (whose attributes the system will annotate) are not occluded by other objects, for example, by selecting images that are sufficiently spaced or temporally separated.

[0085] Furthermore, the cross-reference image search block 360 can be configured to compute the region of interest (ROI) of the trained base model within a subset of selected camera images. Alternatively, the 3D polyline annotation 303 can be directly fed into the trained base model 330, which can compute the ROI within the selected subset of camera images.

[0086] Furthermore, one or more memory storage areas and program code are configured, using one or more processors, to enable system 10 to generate a cue 304 for the trained base model 330. As previously described, the cue includes instructions for the trained base model to assign attributes to each road model object annotated in the obtained first dataset. Furthermore, these instructions are automatically generated based on the obtained first dataset. This process is instructed by a cue generation block 320, in which 3D polyline annotations are provided as input and a cue (e.g., a text cue) is generated for the base model 330 as output. As previously described, cue generation 320 may be a rule-based functional and / or machine learning model trained to output appropriate cue given the first dataset as input. For example, cue generation function 321 may retrieve a cue template 321 and populate the template based on the first dataset to generate an appropriate cue 304.

[0087] Furthermore, one or more memory storage areas and program code are configured, using one or more processors, to enable system 10 to feed a selected subset of camera images from the second dataset and the generated cue 304 to the trained base model 330. Next, the output from the base model 330 (using a suitable parsing function) is parsed to generate a structured data exchange format file 305 and one or more annotated images 306. File 305 includes annotations indicating the assigned attributes of each road model object in the set of road model objects, and the annotated images include annotations indicating the assigned attributes of each road model object in the set of road model objects in one or more camera images. More specifically, once the output from the base model 330 has been parsed, system 10 can be configured to save the parsed output (as indicated by annotation save block 352) as a structured data exchange format file (e.g., a JSON file). Furthermore, as... Figure 3 As indicated in the document, the output from the base model 330 is added back to the 3D polyline annotation 303 in order to generate a file 305 in a structured data exchange format.

[0088] Furthermore, as indicated by visualization block 350, system 10 can be configured to visualize the parsed output by labeling a subset of selected camera images with corresponding attribute annotations.

[0089] Figure 4 Further illustrations include a server 20 comprising any of the embodiments disclosed herein, and a cloud environment 30 comprising one or more servers 20 comprising any of the embodiments disclosed herein.

[0090] Control circuitry 11 may include one or more processors, such as a central processing unit (CPU), graphics processing unit (GPU), microcontroller, or microprocessor. One or more processors may be configured to execute program code stored in memory 12 to perform various functions and operations of vehicle 1 other than those disclosed herein. Some processors may be or include any number of hardware components for performing data or signal processing or for executing computer code stored in memory 12. Optionally, memory 12 includes high-speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid-state storage devices; and optionally includes non-volatile memory such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 12 may include database components, object code components, script components, or any other type of information structure for supporting various operations of this specification.

[0091] The present invention has been presented above with reference to specific embodiments. However, other embodiments besides those described above are possible and within the scope of the invention. Within the scope of the invention, method steps different from those described above, performed by hardware or software, can be provided. Therefore, according to an exemplary embodiment, a non-transitory computer-readable storage medium is provided storing one or more programs configured to be executed by one or more processors of a vehicle control system, the programs including instructions for performing the methods according to any of the embodiments discussed above. Alternatively, according to another exemplary embodiment, a cloud computing system can be configured to perform any of the methods presented herein. The cloud computing system may include distributed cloud computing resources that collectively perform the methods presented herein under the control of one or more computer program products.

[0092] Generally, computer-accessible media can include any tangible or non-transitory storage medium or storage medium, such as electronic, magnetic, or optical media, like a disk or CD / DVD-ROM coupled to a computer system via a bus. The terms “tangible” and “non-transitory” as used herein are intended to describe computer-readable storage media (or “memory”) excluding those that transmit electromagnetic signals, but are not intended to otherwise limit the types of physical computer-readable storage devices covered by the phrases “computer-readable medium” or “memory.” For example, the terms “non-transitory computer-readable medium” or “tangible memory” are intended to cover types of storage devices that do not necessarily permanently store information, including, for example, random access memory (RAM). Program instructions and data stored in a non-transitory form on a tangible computer-accessible storage medium can be further transmitted via a transmission medium or a signal, such as an electrical signal, electromagnetic signal, or digital signal, that can be transmitted via a communication medium such as a network and / or a wireless link.

[0093] The processor 11 (associated with system 10) may be or include any number of hardware components for performing data or signal processing or for executing computer code stored in memory 12. Device 10 has an associated memory 12, and memory 12 may be one or more means for storing data and / or computer code for performing or facilitating the various methods described herein. Memory may include volatile or non-volatile memory. Memory 12 may include database components, object code components, script components, or any other type of information structure for supporting the various actions of this specification. According to exemplary embodiments, any distributed or local storage device may be used with the systems and methods of this specification. According to exemplary embodiments, memory 12 (e.g., via circuitry or any other wired, wireless, or network connection) may be communicatively connected to processor 11 and includes computer code for performing one or more processes described herein.

[0094] It should be noted that no reference numerals in the accompanying drawings limit the scope of the claims. The invention can be implemented at least in part by both hardware and software, and several “devices” or “units” can be represented by the same hardware item.

[0095] Although the accompanying drawings may show a specific order of method steps, the order of steps may differ from the depicted order. Furthermore, two or more steps may be performed simultaneously or partially concurrently. This variation will depend on the chosen software and hardware system and the designer's choices. All these variations are within the scope of this invention. Similarly, standard programming techniques with rule-based logic and other logic can be used to accomplish the various associated steps, alignment steps, annotation steps, and transformation steps. The embodiments mentioned and described above are given by way of example only and should not be construed as limiting the invention. Other solutions, uses, objectives, and functions within the scope of the invention claimed in the following patent claims will be apparent to those skilled in the art.

Claims

1. A computer-implemented method (S100) for generating road model annotations for an autonomous driving system, the computer-implemented method comprising: Obtain (S101) a first dataset, the first dataset comprising 3D polyline annotations of a set of road model objects in a specified environment, wherein the first dataset comprises 3D point clouds represented in 3D space, the 3D point clouds being captured by a lidar device in the specified environment over a time period, indicating the presence of the set of road model objects; Obtain (S102) a second dataset, the second dataset comprising one or more camera images depicting the specified environment within the time period; Based on the first dataset, a subset of camera images is selected from the one or more camera images (S103); (S104) Generate (S104) a hint for the trained base model, wherein the hint includes instructions for the trained base model to assign attributes of each road model object annotated in the obtained first dataset, wherein the instructions are automatically generated based on the obtained first dataset; The selected subset of the camera images from the second dataset is fed (S107) into the trained base model; The generated prompts are fed into the trained base model (S108); and Parse (S109) the output from the trained base model to generate: A structured data exchange format file, wherein the file includes annotations indicating the assigned attributes of each road model object in the set of road model objects; and One or more annotated images, the one or more annotated images including annotations indicating the assigned attributes of each of the set of road model objects in the one or more camera images.

2. The computer-implemented method (S100) according to claim 1, wherein, The structured data exchange format file includes at least one of the following: JSON file, YAML file, XML file, CSV file, TSV file, or binary format file.

3. The computer-implemented method (S100) according to claim 1, wherein, The prompts generated (S104) for the trained base model include: Search (S105) prompt template; and The retrieved prompt template is populated based on the obtained first dataset (S106) to form the instructions for the trained base model to assign the attributes of each road model object annotated in the obtained first dataset.

4. The computer-implemented method (S100) according to claim 1, wherein, The assigned attribute is either the color of each road model object or the type of the road model object.

5. The computer-implemented method (S100) according to claim 1, wherein, The selected (S103) subset of camera images includes a continuous sequence of camera images and the one or more annotation images include a continuous sequence of annotation images.

6. The computer-implemented method (S100) according to claim 1, wherein, The output from the trained base model further includes a confidence score associated with each assigned attribute.

7. A computer program product storing instructions, which, when executed by a computing device, cause the computing device to perform the method according to claim 1 (S100).

8. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing device, cause the computing device to perform the method (S100) according to claim 1.

9. A system (10) for generating road model annotations for an autonomous driving system, the system comprising one or more processors (11) and one or more memory storage areas (12) including program code, the one or more memory storage areas and the program code being configured to use the one or more processors to enable the system (10) to at least: Obtain a first dataset, which includes 3D polyline annotations (303) of a set of road model objects in a specified environment, wherein, The first dataset includes a 3D point cloud (301) represented in 3D space, which is captured by a lidar device in the specified environment over a time period, indicating the presence of the set of road model objects; Obtain a second dataset, which includes one or more camera images (302) depicting the specified environment during the time period. Based on the first dataset, a subset of camera images is selected from the one or more camera images; A hint (304) is generated for the trained base model (330), wherein the hint includes instructions for the trained base model to assign attributes of each road model object annotated in the obtained first dataset, wherein the instructions are automatically generated based on the obtained first dataset; A selected subset of the camera images from the second dataset is fed into the trained base model (330). The generated prompt (304) is fed into the trained base model (330); and Parse the output from the trained base model to generate: A structured data exchange format file (305), wherein the file includes annotations indicating the assigned attributes of each road model object in the set of road model objects; and One or more annotation images (306) include annotations indicating the assigned attributes of each of the set of road model objects in the one or more camera images.

10. The system (10) according to claim 9, wherein, The structured data exchange format file (305) includes at least one of JSON, YAML, XML, CSV, TSV, or binary format files.

11. The system (10) according to claim 9, wherein, The hints (304) generated for the trained base model (330) include: Search suggestion template (321); and The retrieved prompt template is populated based on the obtained first dataset to form the instructions for the trained base model, which are used to assign the attributes of each road model object annotated in the obtained first dataset.

12. The system (10) according to claim 9, wherein, The assigned attribute is either the color of each road model object or the type of the road model object.

13. The system (10) according to claim 9, wherein, The selected subset of camera images includes a continuous sequence of camera images and the one or more annotated images include a continuous sequence of annotated images.

14. A server (20) comprising the system (10) according to claim 9.

15. A cloud environment (30) comprising one or more servers (20) as claimed in claim 14.

Images