Road geometry estimation for vehicles

Abstract

The invention discloses road geometry estimation for a vehicle. Methods and aspects of determining a representation of a road object of a road for a vehicle are disclosed. The method comprises; encoding, for each of the plurality of consecutive time steps, an image output from a camera of the vehicle using a machine learning algorithm trained to output image features of the road object depicted in the image; converting a plurality of image features included in the encoded image into an aerial view BEV representation of the plurality of image features; decoding the bird's-eye view representation to extract an object embedding from the BEV representation using a converter-based machine learning algorithm trained to output the object embedding based on inputs including the BEV representation, the object query, and a transformed priori object embedding extracted at a previous time step; the location and category of each of the road objects are output by decoding the extracted object embedding.

Description

Technical Field

[0001] The disclosed technology relates to methods and systems for determining a representation of one or more road objects of a road for a vehicle traveling on the road. In particular, but not exclusively, the disclosed technology relates to real-time estimation of road geometry, where the road geometry is based on tracking of multiple road objects (e.g., lane markings, road edges, road obstacles, etc.). Background Art

[0002] Over the past few years, the amount of research and development activities related to autonomous vehicles has skyrocketed, and many different approaches are being explored. More and more modern vehicles have advanced driver assistance systems (ADAS) to improve vehicle safety and, more generally, road safety. ADAS (such as represented by adaptive cruise control (ACC) collision avoidance systems, forward collision warnings, etc.) are electronic systems that can assist vehicle drivers while driving. Today, research and development in many technical areas associated with both the ADAS and autonomous driving (AD) fields are ongoing. AD and ADAS can also be referred to as automated driving systems (ADS) with different levels, where AD is generally classified as Level 3 or higher ADS defined by SAE J3016, while ADAS is generally classified as Level 2 or lower ADS defined by SAE J3016.

[0003] As computer hardware in vehicles continues to advance, it allows more and more effort to be put into the development of autonomous driving systems, and many active safety systems such as ACC and lane support systems (LSS) rely on having some knowledge of the road around the vehicle. Therefore, improving accuracy and redundancy in road estimation systems may be critical to improving driver safety and comfort.

[0004] Therefore, accurate and reliable detection of the road geometry is a key function for all lane support systems or any other steering systems. In particular, such systems rely on accurate and reliable tracking of road objects, such as lane marking tracking or road edge tracking. Generally speaking, the lane marking tracking function detects and classifies lane markings from images captured by one or more cameras mounted on the vehicle. Then, based on this detection, a mathematical representation of the geometry or position of the lane markings or road edges is calculated and a road model (including lane geometry) for the traveled section can be determined. Summary of the invention

[0005] The techniques disclosed herein seek to mitigate, alleviate or eliminate the deficiencies and shortcomings in the prior art to address various issues associated with insufficiently accurate and / or computationally expensive and memory intensive road geometry tracking solutions.

[0006] Various aspects and embodiments of the disclosed invention are defined below and in the accompanying independent and dependent claims.

[0007] A first aspect of the disclosed technology includes a computer-implemented method for determining a representation of one or more road objects of a road for a vehicle traveling on a road, wherein the vehicle has one or more cameras. The method includes, for each of a plurality of consecutive time steps, encoding one or more images output from the one or more cameras using one or more machine learning algorithms, the one or more machine learning algorithms being trained to output image features of one or more road objects depicted in the image, the image being provided as input to the one or more machine learning algorithms. The method further includes (for each time step) converting a plurality of image features included in the encoded one or more images into a bird's eye view (BEV) representation of the plurality of image features. Moreover, the method includes (for each time step) decoding the BEV representation so as to extract a set of object embeddings from the BEV representation using one or more transformer-based machine learning algorithms, the one or more transformer-based machine learning algorithms being trained to output the set of object embeddings based on inputs including the BEV representation, a set of object queries, and a set of transformed prior object embeddings extracted at a previous time step. Further, the method includes (for each time step) outputting a position and category of each of the one or more road objects by decoding the extracted set of object embeddings.

[0008] A second aspect of the disclosed technology includes a computer program product including instructions that, when executed by a computing device of a vehicle, cause the computing device to perform a method according to any one of the embodiments disclosed herein. With this aspect of the disclosed technology, there are similar advantages and preferred features as with the other aspects.

[0009] A third aspect of the disclosed technology includes a (non-transitory) computer-readable storage medium containing instructions that, when executed by a computing device of a vehicle, cause the computing device to perform a method according to any one of the embodiments disclosed herein. With this aspect of the disclosed technology, there are similar advantages and preferred features as other aspects.

[0010] The term "non-transitory" as used herein is intended to describe a computer-readable storage medium (or "memory") that excludes propagating electromagnetic signals, but is not intended to otherwise limit the type of physical computer-readable storage device encompassed by the phrase computer-readable medium or memory. For example, the term "non-transitory computer-readable medium" or "tangible memory" is intended to encompass types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in non-transitory form on a tangible computer-accessible storage medium may be further transmitted via a transmission medium or signal such as an electrical signal, an electromagnetic signal, or a digital signal, which may be transmitted via a communication medium such as a network and / or a wireless link. Therefore, the term "non-transitory" as used herein is a limitation on the medium itself (i.e., tangible, not a signal), rather than a limitation on data storage persistence (e.g., RAM vs. ROM).

[0011] A fourth aspect of the disclosed technology includes a system for determining a representation of one or more road objects of a road for a vehicle traveling on a road, wherein the vehicle has one or more cameras. The system includes one or more memory storage areas (including program code), the one or more memory storage areas and the program code being configured to cause the system to, for each of a plurality of consecutive time steps, at least use one or more machine learning algorithms to encode one or more images output from one or more cameras, the one or more machine learning algorithms being trained to output image features of one or more road objects depicted in the image, the image being provided as input to the one or more machine learning algorithms. Further, the system is caused to convert a plurality of image features included in the encoded one or more images into a bird's eye view (BEV) representation of the plurality of image features, and decode the BEV representation so as to extract a set of object embeddings from the BEV representation using one or more transformer-based machine learning algorithms, the one or more transformer-based machine learning algorithms being trained to output the set of object embeddings based on inputs including the BEV representation, a set of object queries, and a set of transformed prior object embeddings extracted at a previous time step. Moreover, by decoding the extracted set of object embeddings, the system is caused to output a position and category of each of the one or more road objects. With this aspect of the disclosed technology, similar advantages and preferred features exist as with the other aspects.

[0012] A fifth aspect of the disclosed technology comprises a vehicle comprising a system according to any one of the embodiments of the fourth aspect disclosed herein. With this aspect of the disclosed technology, similar advantages and preferred features exist as with the other aspects.

[0013] The disclosed aspects and preferred embodiments may be suitably combined with one another in any manner apparent to a person skilled in the art, such that one or more features or embodiments relating to one aspect may also be considered to relate to another aspect or embodiments of another aspect.

[0014] An advantage of some embodiments is that a computationally efficient solution is provided for estimating road geometry for a vehicle, and in particular a vehicle equipped with an ADS.

[0015] An advantage of some embodiments is that the accuracy of road geometry estimation is improved without significantly increasing computing power and data storage requirements.

[0016] An advantage of some embodiments is that fusion of outputs from sensors of different modalities is facilitated to generate a road geometry estimate.

[0017] Further embodiments are defined in the dependent claims. It should be emphasized that when used in this specification, the term "comprise / comprising" is used to specify the presence of recited features, integers, steps or components. It does not exclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

[0018] These and other features and advantages of the disclosed technology will be further elucidated below with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above-described aspects, features, and advantages of the disclosed technology will be more fully understood by reference to the following illustrative and non-limiting detailed description of example embodiments of the present disclosure when taken in conjunction with the accompanying drawings, in which:

[0020] Figure 1 is a schematic flow chart representation of a method for determining a representation of one or more road objects of a road for a vehicle traveling on the road, according to some embodiments.

[0021] Figure 2 is a schematic block diagram representation of a system for determining a representation of one or more road objects of a road for a vehicle traveling on the road, according to some embodiments.

[0022] Figure 3 is a schematic illustration of a vehicle including a system for determining a representation of one or more road objects of a road for a vehicle traveling on the road, in accordance with some embodiments. DETAILED DESCRIPTION

[0023] The present disclosure will now be described in detail with reference to the accompanying drawings, in which some example embodiments of the disclosed technology are shown. However, the disclosed technology may be embodied in other forms and should not be construed as being limited to the disclosed example embodiments. The disclosed example embodiments are provided to fully convey the scope of the disclosed technology to those skilled in the art. Those skilled in the art will appreciate that the steps, services, and functions explained herein may be implemented using separate hardware circuits, using software that works in conjunction with a programmed microprocessor or general-purpose computer, using one or more application specific integrated circuits (ASICs), using one or more field programmable gate arrays (FPGAs), and / or using one or more digital signal processors (DSPs).

[0024] It will also be appreciated that when the present disclosure is described in the form of a method, it may also be embodied in an apparatus comprising one or more processors, one or more memories coupled to the one or more processors, wherein computer code is loaded to implement the method. For example, in some embodiments, the one or more memories may store one or more computer programs that, when executed by the one or more processors, cause the apparatus to perform the steps, services, and functions disclosed herein.

[0025] It should also be understood that the terms used herein are only for the purpose of describing particular embodiments and are not intended to be limiting. It should be noted that the articles "a", "an", "the" and "said" as used in the specification and the appended claims are intended to mean the presence of one or more elements, unless the context clearly indicates otherwise. Thus, for example, in some contexts, references to "unit" or "the unit" may refer to more than one unit, etc. In addition, the words "comprising", "including" and "containing" do not exclude other elements or steps. It should be emphasized that when used in this specification, the term "comprise / comprising" is used to specify the presence of the described features, integers, steps or components. It does not exclude the presence or addition of one or more other features, integers, steps, components or groups thereof. The term "and / or" should be interpreted as meaning "both" and each as optional.

[0026] It should also be understood that, although the terms first, second, etc. can be used to describe various elements or features in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the embodiment, the first signal can be referred to as the second signal, and similarly, the second signal can be referred to as the first signal. The first signal and the second signal are both signals, but they are not the same signal.

[0027] Overview

[0028] Generally speaking, autonomous vehicles (also referred to herein as ADS-equipped vehicles) rely on a variety of sensors (including cameras, radars, and lidars) to perceive the surrounding environment and make decisions about how to navigate the environment and manipulate the vehicle. Understanding road geometry can be considered an important aspect of this perception process because it provides key information about the layout of the road on which the vehicle is traveling. In short, some embodiments herein provide a way to estimate road geometry in real time, where the road geometry is formed based on tracked road objects. These road objects can be, for example, lane markings, road edges, crosswalks, etc. Each detected road object is composed of multiple points that span the geometry of the road object, and the road geometry (i.e., the geometry of the lanes and road edges) can be derived based on the geometry of the road object.

[0029] A popular approach for estimating road geometry involves a process of detecting and segmenting road objects in the image plane (i.e., the "camera view"), and projecting them into 3D (i.e., a bird's eye view (BEV) or top view representation) using either a single depth estimate or a geometric projection. Finally, some Bayesian filter architecture is employed to track the resulting road objects. An estimate of the road geometry is then made based on these tracked road objects. This means that such a tracking process discards a lot of available information about the road, since it only tracks the geometric representation of the detected road objects.

[0030] However, because the road is stationary while the vehicle is moving, it may be advantageous to accumulate more information (not just the tracked road objects) over time in order to improve the accuracy of the road model estimate. Therefore, while many road geometry estimation methods currently in use use statistical models such as Bayesian filters (such as Kalman filters) to track "lightweight" objects (such as detected lane markings), there are also some attempts to track a high-dimensional representation of the entire scene over time to accumulate information and make more accurate predictions. However, tracking the entire scene has proven to be computationally expensive and memory intensive, making it less suitable for online implementation (i.e., real-time tracking in a vehicle).

[0031] In summary, it can be said that there are two main approaches to tracking road geometry. That is, tracking "lightweight" detected objects or tracking "heavy" entire scene representations.

[0032] According to some embodiments herein, it is proposed to track high-level feature representations of road objects (i.e., object embeddings) instead of geometric representations of detected road objects or an entire bird's-eye view (BEV) representation of the entire scene. In other words, some embodiments herein propose a method that aims to include more information than "output objects" in the tracking process while avoiding the need to store the entire scene representation between time steps. Accordingly, by balancing computational efficiency and road geometry estimation accuracy, an advantage of some embodiments is that the perception of road geometry by an autonomous driving system for a vehicle can be enhanced or otherwise improved.

[0033] definition

[0034] The term "image features" can be understood as unique and informative characteristics or patterns extracted from a camera image. These features are typically generated by algorithms designed to process visual data, such as convolutional neural networks (CNNs) or other computer vision techniques. Image features enable machines to understand and interpret visual information from the environment. In more detail, image features can represent specific patterns or structures within an image that are relevant to the task at hand. These patterns can include edges, corners, textures, shapes, or more complex visual elements like objects or parts of objects. Image features are typically represented as numeric vectors, where each element of the vector corresponds to a specific aspect of the visual information. These vectors can then be used as input to other algorithms for various tasks, such as object detection, conversion, classification, or segmentation.

[0035] In the present context, "road objects" refer to specific elements or features on or associated with the road surface that are detected, identified, and tracked by the vehicle's perception system, and in particular the vehicle's road geometry estimation system. These road objects provide information about the road environment and enable safe and efficient autonomous driving. Examples of road objects include lane markings, road edges, curbs, road intersections, crosswalks, road geometry features such as curvature, slope, and elevation changes.

[0036] The term “machine learning algorithm” can be understood as a set of computational rules and procedures designed to enable a computer program or system to automatically learn patterns, relationships, and insights from data without being explicitly programmed. These algorithms form the basis of machine learning, which is a subfield of artificial intelligence (AI). Moreover, machine learning (ML) algorithms are data-driven. They operate by analyzing and learning from a data set that contains examples or instances with associated features or attributes. These algorithms are designed to discover patterns or correlations in the data that can be used for various tasks. Machine learning algorithms typically undergo a training process during which they are exposed to labeled data sets (supervised learning) or unlabeled data sets (unsupervised learning). During training, the algorithm adjusts its internal parameters to capture the underlying patterns in the data. Further, one of the main goals of a machine learning algorithm is to generalize its learning from the training data to make predictions or decisions on new, unseen data. The algorithm should be able to make accurate and reliable predictions outside of the training set. In some embodiments, machine learning algorithms (which may also be referred to as machine learning models, neural networks, etc.) are implemented in any suitable manner known to a person of ordinary skill in the art using appropriate software development machine learning code elements available to the public, such as, for example, code elements available in Python, Keras, and TensorFlow, or any other appropriate software development platform.

[0037] In this context, a "bird's eye view" representation, often referred to as a "top-down view" or "bird's eye view", is a graphical representation of the vehicle's surroundings or road scene from a perspective directly above the vehicle. This representation is typically generated using data from various sensors (such as cameras, lidar, radar, and / or GPS), and it provides a comprehensive and holistic view of the vehicle's surroundings. The BEV provides a top-down view of the road and its objects, typically represented as a grid, with each cell encoding information about the presence and characteristics of road objects.

[0038] A "Transformer-based machine learning algorithm" (or simply "Transformer") is a deep learning model designed primarily for processing sequential data such as natural language, time series data, etc. Transformers have gained significant popularity and success in various applications, especially in the fields of natural language processing (NLP) and computer vision. They were introduced by Vaswani et al. in their 2017 paper "Attention Is All You Need" and have since become a cornerstone of modern deep learning. Moreover, transformers are generally composed of multiple layers, each of which includes self-attention and feed-forward sublayers. Stacking these layers allows the model to learn increasingly abstract representations of the input data. Transformers can also be used as encoders, decoders, or both.

[0039] In more detail, a key aspect of the Transformer is the attention mechanism, which allows the model to focus on different parts of the input sequence as it processes it. This mechanism enables the model to capture long-range dependencies in the data, which was challenging for earlier recurrent neural networks (RNNs) and convolutional neural networks (CNNs). The Transformer uses a self-attention mechanism, where each element (or token) in the input sequence pays attention to all other elements in a weighted manner. This allows the model to dynamically weigh the importance of different elements in the sequence based on their context.

[0040] In general, the term "object embedding" can be understood as a technique for representing and encoding information about objects in a scene (such as vehicles, pedestrians, and obstacles) in a structured and meaningful way. In the context of the present disclosure, "object embedding" accordingly includes machine-readable information about detected road objects (such as lane markings, road edges, crosswalks, etc.). These embeddings are typically numerical vectors that capture various attributes and characteristics of objects, enabling computers (i.e., ADSs) to understand and reason about the environment around them.

[0041] In some examples, object embeddings encode high-dimensional features that describe the appearance, motion, and other relevant properties of an object. These features can include information about the object's position, speed, size, shape, color, and texture, among others. In some examples, object embeddings can also capture semantic information about an object, such as its category (e.g., car, bicycle, pedestrian) and its unique identifier. This semantic understanding can be important for object recognition and tracking. In some examples, object embeddings can include temporal information such as the trajectory of an object or its recent history to predict its future behavior. In some examples, object embeddings can represent spatial relationships between objects in a scene. For example, they can capture whether an object is in front of or behind another object, its relative distance, and its direction relative to other objects. In addition, object embeddings can also incorporate uncertainty estimates, indicating the system's confidence in its object detection and tracking. This is critical for safe decision-making in autonomous driving.

[0042] Object embeddings are therefore fundamental to a wide range of tasks in autonomous driving, such as object detection, object tracking, motion prediction, and path planning. They provide a compact and information-rich representation of the surrounding environment, ensuring that autonomous vehicles can make intelligent decisions and navigate safely in complex and dynamic traffic scenarios. Advanced machine learning techniques, such as deep neural networks, are often used to learn and generate these embeddings from sensor data like lidar, radar, and cameras.

[0043] The term "object query" can be understood as a mechanism or concept that ensures that object representations or embeddings can be retrieved in a larger set or database of such embeddings, or that a specific object representation or embedding can be interacted with. This concept is often associated with the way that attention mechanisms or similar techniques are used to focus on and retrieve relevant embeddings based on specific criteria or queries. Therefore, an object query can be understood as a request or query made to this database (e.g., BEV representations of multiple image features) to retrieve specific object embeddings that match certain criteria or attributes. These queries can be based on various factors, such as object category, similarity to a reference object, spatial proximity, or any other relevant attribute. In other words, an object query can be interpreted as an "initial guess" of an object embedding, and so can have a 1:1 ratio with the object embedding (i.e., the same number of object embeddings as the object query, and the same size vector).

[0044] Object queries often involve the use of attention mechanisms, which are a fundamental component of many modern machine learning models (including transformers). Attention mechanisms allow the model to focus on specific parts of the input data or embedding while de-emphasizing other parts, effectively acting as a mechanism for querying the data. In computer vision applications, object queries can be used to retrieve and track specific objects in a video stream or image.

[0045] Example

[0046] Figure 1 is a schematic flow chart representation of a method S100 for determining a (geometric) representation of one or more road objects of a road for a vehicle traveling on a road in accordance with some embodiments. In some embodiments, the geometric representation may be interpreted as a 3D representation. The method S100 is preferably a computer-implemented method S100 performed by a processing system of a vehicle equipped with an ADS. The processing system may, for example, include one or more processors and one or more memories coupled to the one or more processors, wherein the one or more memories store one or more programs that, when executed by the one or more processors, cause the processing system to perform the steps, services and functions of the method S100 disclosed herein.

[0047] Method S100 is a repetitive method S100, meaning that it repeats method steps at a certain set frequency or sampling rate. In other words, method S100 is executed at each time step or time period in a plurality of consecutive time steps or time periods. For example, the method may be executed once every 50 ms, meaning that the sampling rate is 20 Hz.

[0048] Method S100 includes encoding S101 one or more images 201 output from one or more cameras using one or more machine learning algorithms. The one or more machine learning algorithms are trained to output image features of one or more road objects depicted in the image 201, which is provided as input to the one or more machine learning algorithms. An "encoded image" generally refers to an image that has been converted or represented in a different format or space, typically using encoding techniques (in this case, using one or more trained machine learning algorithms). These encoding methods are used to extract and represent specific features or information from the original image to make it more suitable for further processing, analysis, or transmission. Therefore, in this context, the road objects depicted in the input image 201 are extracted and represented as "image features" to make them more suitable for further processing by a computer or machine.

[0049] For example, features are parts or patterns of an object in an image that help identify the object. For example, a square has 4 corners and 4 sides, which can be called features of a square, and they can help us humans recognize that it is a square. Features include properties like corners, sides, areas of interest, ridges, etc. Naturally, the features of an object that help a machine recognize the object may be completely different than the features that help a human recognize the same object.

[0050] The method S100 further includes converting S102 a plurality of image features included in the encoded one or more images into a bird's eye view (BEV) representation of the plurality of image features. In some embodiments, the plurality of image features are converted into the BEV representation using an inverse perspective mapping (IPM) algorithm and further based on the camera pose of each of the one or more cameras. In other words, the IPM algorithm ingests the image features (i.e., the encoded images) together with information related to the camera pose of the associated camera in order to output the BEV representation of the image features.

[0051] In more detail, the IPM algorithm requires information about the pose of the camera, including its position and orientation in 3D space. This pose information is needed to map the features of the camera's view to a bird's-eye view (i.e., a top-down view). The algorithm takes as input an image captured by the camera. This image contains various features that need to be converted to a BEV representation, such as lane markings, road edges, or other scene elements. To perform the conversion, IPM typically employs a mathematical technique called homography. Based on the camera pose information, a homography matrix is ​​calculated that describes the relationship between points in the camera's view and points in the BEV's view. The IPM algorithm uses the homography matrix to correct for perspective distortion in the image. This distortion is caused by the camera's viewpoint, which makes objects appear distorted or tilted in the image.

[0052] In some embodiments, method S100 further includes encoding S106 a lidar output data set 211 from one or more lidars of the vehicle onto the BEV representation using a machine learning algorithm trained to extract a plurality of features of one or more road objects indicated in the lidar output data set, the lidar output data set being provided as input to the machine learning algorithm. In other words, method S100 may include adding the lidar output data 211 to the BEV representation, thereby adding even more information to the BEV representation and ensuring that the accuracy of the resulting road geometry estimate can be improved.

[0053] Further, the method S100 includes decoding S103 the BEV representation so as to extract a set of object embeddings from the BEV representation using one or more transformer-based machine learning algorithms. Here, the one or more transformer-based machine learning algorithms are trained to output the set of object embeddings based on inputs including the BEV representation, a set of object queries 210, and a set of transformed prior object embeddings extracted at a previous time step. In other words, the one or more transformer-based machine learning algorithms are fed with the BEV representation, a set of object queries (i.e., one or more object queries) 210, and a set of transformed S105 prior object embeddings extracted at a previous time step.

[0054] In some embodiments, the transformer-based machine learning model is configured to output an object embedding for each object query in the set of object queries. Furthermore, the object embedding and the object query can be vectors of the same size.

[0055] Moreover, in some embodiments, method S100 includes using one or more multi-layer perceptron (MLP) algorithms and the vehicle's motion data 214 to transform S105 the object embeddings extracted at the previous time step. In other words, the extracted object embeddings are reused (or "tracked") in the next iteration. A multi-layer perceptron (MLP) is an artificial neural network used for machine learning. It is a feed-forward neural network with one or more hidden layers, which enables it to model complex relationships in the data. The term "perceptron" refers to a single-layer neural network, so the "multi-layer" aspect represents that the MLP is composed of multiple layers of interconnected neurons. In general, MLPs are trained by back-propagation, where an optimization algorithm is used to calculate the gradient of the loss function (i.e., the function that measures the difference between the predicted output and the ground truth) with respect to the network weights and biases, and they are updated accordingly using optimization techniques like gradient descent.

[0056] Further, the method S100 includes outputting S104 the position and category of each road object in one or more road objects by decoding the extracted set of object embeddings. In other words, the method S100 includes decoding S104 the extracted set of object embeddings so as to output a geometric representation and category of each road object. Optionally, the output may further include a geometric uncertainty or uncertainty value of the geometric representation of each road object. In some embodiments, the extracted set of object embeddings is decoded using one or more multi-layer perceptron (MLP) algorithms configured to output the position and category of each road object based on an input including an object embedding. In some embodiments, a set of MLPs (i.e., one or more MLPs) may be used to output the position of each road object, and another set of MLPs (i.e., one or more MLPs) may be used to output the category of each road object.

[0057] In some embodiments, the method S100 further includes forming S107 a geometric representation of one or more road objects based on the output position and category of each road object. The geometric representation of the one or more road objects can be, for example, in the form of a set of broken lines indicating the geometric shapes of the one or more road objects in the surrounding environment of the vehicle. Moreover, the formed S107 geometric representation can be output to an ADS function configured to control steering, acceleration and / or braking of the vehicle, wherein the steering, acceleration or braking is controlled according to the geometric representation.

[0058] Moreover, in some embodiments, the method S100 includes forming S108 a loss function based on the position and category of each road object output and a ground truth dataset. Further, the set of object queries can be updated S109 based on the formed loss function using an appropriate optimization algorithm. In more detail, for the "position" output, the loss function can be formed using Euclidean distance loss, while for the "category" output, the loss function can be formed using cross entropy loss (focal loss). The ground truth dataset can, for example, be in the form of manually annotated HD map data.

[0059] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0060] Figure 2is a schematic block diagram representation of a system 10 for determining a representation of one or more road objects of a road for a vehicle traveling on a road, wherein the vehicle has one or more cameras. The system 10 includes a control circuit (e.g., one or more processors) configured to perform the functions of the method S100 disclosed herein, wherein the functions may be included in a non-transitory computer-readable storage medium 12 or in other computer program products configured to be executed by the control circuit 11. In other words, the system 10 includes one or more memory storage areas 12 containing program code, and the one or more memory storage areas 12 and the program code are configured to cause the system 10 to perform the method S100 according to any one of the embodiments disclosed herein with one or more processors 11. However, in order to better illustrate the embodiments disclosed herein, the control circuit is not described in detail. Figure 2 1 is represented as various "modules" or blocks, each of which is linked to one or more specific functions of the control circuit. As previously mentioned, the process performed is a repetitive process, meaning that the system 10 repeats these steps at a certain set frequency or sampling rate. In other words, one or more memory storage areas and program code are configured to cause the system 10 to perform the process at each time step in a plurality of consecutive time steps using one or more processors.

[0061] Accordingly, one or more memory storage areas and program code are configured to cause the system 10, with one or more processors, to encode one or more images 201 output from one or more cameras of the vehicle using one or more machine learning algorithms 202, wherein the one or more machine learning algorithms are trained to output image features of one or more road objects depicted in the image, which is provided as input to the one or more machine learning algorithms 201.

[0062] Further, the one or more memory storage areas and program code are configured to cause the system 10, with the one or more processors, to convert the plurality of image features included in the encoded one or more images into a bird's eye view (BEV) representation of the plurality of image features 204. In some embodiments, the plurality of image features are converted into the BEV representation 204 using an inverse perspective mapping (IPM) algorithm 203 and based on a camera pose of each of the one or more cameras.

[0063] Further, in some embodiments, one or more memory storage areas and program code are configured to cause the system 10, with one or more processors, to encode a lidar output dataset 211 from one or more lidars of the vehicle onto a BEV representation using a machine learning algorithm (ML algorithm) 212, the machine learning algorithm being trained to extract multiple features of one or more road objects indicated in the lidar output dataset 211, the lidar output dataset 211 being provided as input to the machine learning algorithm.

[0064] The one or more memory storage areas and program code are further configured to cause the system 10, with one or more processors, to decode the BEV representation in order to extract a set of object embeddings from the BEV representation using one or more transformer-based machine learning algorithms 205. The one or more transformer-based machine learning algorithms 205 are trained to output the set of object embeddings based on inputs including the BEV representation 204, a set of object queries 210, and a set of transformed prior object embeddings 209 extracted at a previous time step. In some embodiments, the transformer-based machine learning model is configured to output an object embedding for each object query in the set of object queries. Moreover, the object embeddings and object queries may be vectors of the same size.

[0065] In this context, the transformer-based machine learning algorithm 205 for decoding a bird's eye view (BEV) representation of a set of image features into a set of object embeddings can be understood as a neural network architecture designed to understand and process spatial relationships and contextual information in the BEV representation. Here, the algorithm is specifically tailored for tasks like object detection and tracking in a road environment.

[0066] In more detail, the input to the transformer-based machine learning algorithm is the BEV representation of image features, a set of object queries, and prior object embeddings. Image features encode information about road objects such as their shape, size, position, and orientation. The transformer-based decoding stage takes these image features as input. It typically consists of multiple layers of transformer blocks. Each transformer block processes the image features using a self-attention mechanism to capture the relationship between different parts of the BEV representation of image features. As the algorithm processes the BEV representation of image features through the transformation layers, it learns to generate object embeddings for each detected road object. These embeddings can be understood as digital vectors that represent the characteristics and attributes of each object in a structured and machine-meaningful way. Moreover, for tracking purposes, transformer-based machine learning algorithms can also take into account temporal information such as the object's motion history or trajectory. This is achieved by feeding a set of prior object embeddings extracted after the transformation at previous time steps and helps predict the future position and behavior of the object. The output of the algorithm is a set of object embeddings, each of which corresponds to a road object detected in the BEV view. These embeddings capture rich information about the characteristics and context of each object in the scene. Depending on the specific application, post-processing steps can be applied to the object embeddings. This can include non-maximum suppression to remove duplicate detections or filtering based on confidence scores to ensure the highest quality object representation.

[0067] Further, in some embodiments, the one or more memory storage areas and program code are configured to cause the system 10, with one or more processors, to transform the object embedding extracted at a previous time step using one or more multi-layer perceptron (MLP) algorithms 208 and motion data 214 of the vehicle.

[0068] Further, the one or more memory storage areas and program code are configured to cause the system 10, with the one or more processors, to output the location and category of each of the one or more road objects by decoding the extracted set of object embeddings. Moreover, in some embodiments, the extracted set of object embeddings are decoded by using one or more multi-layer perceptron (MLP) algorithms 206a-206c configured to output the location and category of each road object based on inputs including the object embeddings.

[0069] Moreover, in some embodiments, one or more memory storage areas and program code are configured to cause the system 10, with one or more processors, to form / output a geometric representation 207 of one or more road objects based on the position and category of each road object. Thus, rather than outputting the position and category of each road object, the system 10 may simply output the geometric representation 207 based on the position and category of each road object indicated by the decoded object embedding. As mentioned, the geometric representation 207 may be consumed by various ADS functions (such as any autonomous driving functions) that rely on the road model representation as input.

[0070] Further, the one or more memory storage areas and program code may be configured to, with one or more processors, cause the system 10 to form a loss function based on the outputted location and category of each road object and the ground truth dataset 215. This is indicated in the illustration by the learning engine 213. Moreover, the one or more memory storage areas and program code may be configured to, with one or more processors, cause the system to update the set of object queries based on the formed loss function.

[0071] Figure 3 is a schematic illustration of an ADS-equipped vehicle 1 including such a system 10. As used herein, a "vehicle" is any form of motor vehicle. For example, the vehicle 1 may be any road vehicle such as (as illustrated herein) a car, a motorcycle, a (freight) truck, a bus, etc.

[0072] The system 10 includes a control circuit 11 and a memory 12. The control circuit 11 may physically include a single circuit device. Alternatively, the control circuit 11 may be distributed over several circuit devices. As an example, the device 10 may share its control circuit 11 with other components of the vehicle 1 (e.g., the ADS 310). Moreover, the control system 10 may form a part of the ADS 310, i.e., the control system 10 may be implemented as a module or feature of the ADS. The control circuit 11 may include one or more processors such as a central processing unit (CPU), a microcontroller, or a microprocessor. One or more processors may be configured to execute program codes stored in the memory 12 so as to perform various functions and operations of the vehicle 1 in addition to the methods disclosed herein. The processor may be or include any number of hardware components for performing data or signal processing or for executing computer codes stored in the memory 12. The memory 12 may optionally include high-speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and may optionally include non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 12 may include a database component, an object code component, a script component, or any other type of information structure for supporting the various activities of the present specification.

[0073] In the illustrated example, the memory 12 further stores map data 308. The map data 308 may be used, for example, by the ADS 310 of the vehicle 1 to perform the autonomous functions of the vehicle 1. The map data 308 may include high-definition (HD) map data. It is considered that even if the memory 12 is illustrated as a separate element from the ADS 310, it may be provided as an integrated element of the ADS 310. In other words, according to exemplary embodiments, any distributed or local memory device may be used for the implementation of the inventive concept. Similarly, the control circuit 11 may be distributed, for example, so that one or more processors of the control circuit 11 are provided as an integrated element of the ADS 310 (e.g., the perception module 314 of the ADS) or any other system of the vehicle 1. In other words, according to exemplary embodiments, any distributed or local control circuit device may be used for the implementation of the inventive concept. The ADS 310 is configured to perform the functions and operations of the autonomous or semi-autonomous functions of the vehicle 1. The ADS 310 may include a plurality of modules, each of which is responsible for a different function of the ADS 310.

[0074] The vehicle 1 includes a number of elements commonly found in autonomous or semi-autonomous vehicles. It should be understood that the vehicle 1 may have Figure 3 Any combination of the various elements shown in FIG. Moreover, the vehicle 1 may include Figure 3Further elements beyond those shown in . Although various elements are shown herein as being located inside the vehicle 1, one or more elements may be located outside the vehicle 1. For example, map data may be stored in a remote server and accessed by various components of the vehicle 1 via the communication system 326. Further, even though various elements are depicted herein in certain arrangements, as will be readily appreciated by those skilled in the art, the various elements may also be implemented in different arrangements. It should be further noted that the various elements may be communicatively connected to each other in any appropriate manner. Figure 3 The vehicle 1 should be regarded as an illustrative example only, as the elements of the vehicle 1 may be implemented in a number of different ways.

[0075] The vehicle 1 further comprises a sensor system 320. The sensor system 320 is configured to acquire sensory data about the vehicle itself or its surroundings. The sensor system 320 may, for example, comprise a global navigation satellite system (GNSS) module 322 (such as GPS) configured to collect geographic location data of the vehicle 1. The sensor system 320 may further comprise one or more sensors 324. The sensor 324 may be any type of on-board sensor such as a camera, a lidar and a radar, an ultrasonic sensor, a gyroscope, an accelerometer, an odometer, etc. It should be appreciated that the sensor system 320 may also provide the possibility of acquiring sensory data directly or via a dedicated sensor control circuit in the vehicle 1.

[0076] The vehicle 1 further includes a communication system 326. The communication system 326 is configured to communicate with external units such as other vehicles (i.e., via a vehicle-to-vehicle (V2V) communication protocol), a remote server (e.g., a cloud server), a database, or other external devices, i.e., a vehicle-to-infrastructure (V2I) or vehicle-to-everything (V2X) communication protocol. The communication system 318 can communicate using one or more communication technologies. The communication system 318 may include one or more antennas (not shown). Cellular communication technology can be used for remote communications such as to a remote server or a cloud computing system. In addition, if the cellular communication technology used has low latency, it can also be used for V2V, V2I, or V2X communications. Examples of cellular radio technologies include GSM, GPRS, EDGE, LTE, 5G, 5G NR, etc., including future cellular solutions. However, in some solutions, medium- and short-range communication technologies such as wireless local area networks (LANs) (e.g., solutions based on IEEE 802.11) can be used to communicate with other vehicles near the vehicle 1 or with local infrastructure elements. ETSI is working on cellular standards for vehicle communications, with 5G considered a suitable solution due to its low latency and efficient handling of high bandwidth and communication channels.

[0077] The communication system 326 may accordingly provide the possibility to send outputs to a remote location (e.g. a remote operator or a control center) and / or receive inputs from a remote location via one or more antennas. Moreover, the communication system 326 may further be configured to allow various elements of the vehicle 1 to communicate with each other. As an example, the communication system may provide a local network setup such as CAN bus, I2C, Ethernet, fiber optics, etc. The local communication within the vehicle may also be of wireless type with protocols such as WiFi, LoRa, Zigbee, Bluetooth or similar medium / short range technologies.

[0078] The vehicle 1 further includes a control system 320. The control system 328 is configured to control the control of the vehicle 1. The control system 328 includes a steering module 330 configured to control the heading of the vehicle 1. The control system 328 further includes a throttle module 332 configured to control the actuation of the throttle of the vehicle 1. The control system 328 further includes a brake module 334 configured to control the actuation of the brakes of the vehicle 1. The various modules of the control system 328 can also receive manual inputs from the driver of the vehicle 1 (i.e., from the steering wheel, accelerator pedal, and brake pedal, respectively). However, the control system 328 can be communicatively connected to the ADS 310 of the vehicle to receive instructions on how the various modules of the control system 328 should act. Therefore, the ADS 310 can control the control of the vehicle 1, for example, via the decision and control module 318.

[0079] The ADS 310 may include a positioning module 312 or positioning block / system. The positioning module 312 is configured to determine and / or monitor the geographic location and heading of the vehicle 1 and may utilize data from a sensor system 320, such as data from a GNSS module 322. Alternatively or in combination, the positioning module 312 may utilize data from one or more sensors 324. The positioning system may alternatively be implemented as a real-time kinematic (RTK) GPS for improved accuracy. Module

[0080] The ADS 310 may further include a perception module 314 or perception block / system 314. The perception module 314 may refer to any known module and / or functionality, for example included in one or more electronic control modules and / or nodes of the vehicle 1, adapted and / or configured to interpret sensory data related to driving of the vehicle 1 to identify, for example, obstacles, lanes, relevant signs, appropriate navigation paths, etc. The perception module 314 may therefore be adapted to rely on and obtain input from multiple data sources such as automotive imaging, image processing, computer vision, and / or in-vehicle networking, in combination with, for example, perception data from the sensor system 320. In some embodiments, the system 10 may be implemented as a submodule of the perception module 314.

[0081] The positioning module 312 and / or the perception module 314 may be communicatively connected to the sensor system 320 to receive the perception data from the sensor system 320. The positioning module 312 and / or the perception module 314 may further transmit control instructions to the sensor system 320.

[0082] The technology disclosed herein is presented above with reference to specific embodiments. However, other embodiments than those described above are possible and within the scope of the present invention. Within the scope of the technology disclosed herein, method steps for executing the method by hardware or software different from those described above may be provided. Therefore, according to some embodiments, a non-computer transient readable storage medium is provided that stores one or more programs, the one or more programs being configured to be executed by one or more processors of a computing device of a vehicle, the one or more programs including instructions for executing a method according to any one of the above-described embodiments. Alternatively, according to another exemplary embodiment, a cloud computing system may be configured to execute any of the methods presented herein. The cloud computing system may include distributed cloud computing resources that jointly execute the methods presented herein under the control of one or more computer program products.

[0083] In general, computer-accessible media may include any tangible or non-transient storage media or storage media such as electronic, magnetic, or optical media (e.g., disks or CD / DVD-ROMs connected to a computer system via a bus). The terms "tangible" and "non-transient" as used herein are intended to describe computer-readable storage media (or "memory") that exclude propagating electromagnetic signals, but are not intended to otherwise limit the types of physical computer-readable storage devices encompassed by the phrase computer-readable media or memory. For example, the terms "non-transient computer-readable media" or "tangible memory" are intended to encompass types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in non-transient form on tangible computer-accessible storage media may be further transmitted via a transmission medium or a signal such as an electrical signal, an electromagnetic signal, or a digital signal, which may be transmitted via a communication medium such as a network and / or a wireless link.

[0084] The processor 11 (associated with the system 10) may be or include any number of hardware components for performing data or signal processing or for executing computer code stored in the memory 12. The device 10 has an associated memory 12, and the memory 12 may be one or more devices for storing data and / or computer code to complete or facilitate the various methods described in this specification. The memory may include volatile memory or non-volatile storage. The memory 12 may include a database component, an object code component, a script component, or any other type of information structure for supporting the various activities of the present specification. According to an exemplary embodiment, any distributed or local storage device may be used with the system and method of the present specification. According to an exemplary embodiment, the memory 12 is communicatively connected to the processor 11 (e.g., via a circuit or any other wired, wireless or network connection) and includes computer code for executing one or more processes described herein.

[0085] Accordingly, it is understood that parts of the described solution may be implemented in the vehicle 1, in a system located outside the vehicle 1, or in a combination of inside and outside the vehicle; such as a so-called cloud solution implemented in a server communicating with the vehicle. For example, sensor data may be sent to an external system and the system performs the encoding, conversion and decoding steps. The different features and steps of the embodiments may be combined in other combinations than the steps described.

[0086] It should be noted that any reference signs do not limit the scope of the claims, that the present invention may be implemented at least partially by hardware and software, and that several "means" or "units" may be represented by the same item of hardware.

[0087] Although the accompanying drawings may show the specific order of method steps, the order of steps may be different from that described. In addition, two or more steps may be performed simultaneously or partially simultaneously. This variation will depend on the selected software and hardware system and the designer's choice. All these variations are within the scope of the present invention. Similarly, software implementation can be completed with standard programming techniques based on rule-based logic and other logics to complete various encoding steps, conversion steps, decoding steps and output steps. The above-mentioned and described embodiments are given as examples only and should not be limited to the present invention. Other solutions, uses, purposes and functions within the scope of the present invention claimed in the patent claims described below should be apparent to those skilled in the art.

Claims

1. A computer-implemented method for determining a representation of one or more road objects of a road for a vehicle traveling on the road, the vehicle having one or more cameras, the method comprising: For each of a number of consecutive time steps: encoding one or more images output from the one or more cameras using one or more machine learning algorithms trained to output image features of one or more road objects depicted in the images, the images being provided as input to the one or more machine learning algorithms; Converting a plurality of image features included in the encoded one or more images into a bird's eye view (BEV) representation of the plurality of image features; decoding the BEV representation to extract a set of object embeddings from the BEV representation using one or more transformer-based machine learning algorithms trained to output the set of object embeddings based on inputs including the BEV representation, a set of object queries, and a set of transformed prior object embeddings extracted at a previous time step; By decoding the extracted set of object embeddings, a position and a category of each road object in the one or more road objects are output.

2. The method according to claim 1, further comprising: For each time step in the plurality of consecutive time steps: The object embedding extracted at a previous time step is transformed using one or more multi-layer perceptron (MLP) algorithms and the motion data of the vehicle.

3. The method according to claim 1 or 2, further comprising: For each time step in the plurality of consecutive time steps: A lidar output dataset from one or more lidars of the vehicle is encoded onto the BEV representation using a machine learning algorithm trained to extract multiple features of one or more road objects indicated in the lidar output dataset, the lidar output dataset being provided as input to the machine learning algorithm. 4 . The method of claim 1 , wherein converting the plurality of image features into the BEV representation uses an inverse perspective mapping algorithm and is based on a camera pose of each of the one or more cameras.

5. The method of claim 1, wherein the transformer-based machine learning model is configured to output an object embedding for each object query in the set of object queries, and wherein the object embedding and the object query are vectors of the same size.

6. The method according to claim 1, further comprising: For each time step in the plurality of consecutive time steps: A geometric representation of the one or more road objects is formed based on the output position and category of each road object.

7. The method of claim 1 , wherein the extracted set of object embeddings are decoded by using one or more multi-layer perceptron (MLP) algorithms configured to output the position and category of each road object based on an input including the object embeddings.

8. The method according to any one of claims 1 to 2 and 4 to 7, further comprising: A loss function is formed based on the position and category of each road object output and a ground truth dataset.

9. The method according to claim 8, further comprising: The set of object queries is updated based on the formed loss function.

10. A computer program product comprising instructions which, when executed by a computing device of a vehicle, cause the computing device to perform the method of claim 1.

11. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing device of a vehicle, cause the computing device to perform the method of claim 1.

12. A system for determining a representation of one or more road objects of a road for a vehicle traveling on the road, the vehicle having one or more cameras, the system comprising one or more memory storage areas, the one or more memory storage areas including program code, the one or more memory storage areas and the program code being configured to cause the system, with the one or more processors, to at least: For each of a number of consecutive time steps: encoding one or more images output from the one or more cameras using one or more machine learning algorithms trained to output image features of one or more road objects depicted in the images, the images being provided as input to the one or more machine learning algorithms; Converting a plurality of image features included in the encoded one or more images into a bird's eye view (BEV) representation of the plurality of image features; decoding the BEV representation to extract a set of object embeddings from the BEV representation using one or more transformer-based machine learning algorithms trained to output the set of object embeddings based on inputs including the BEV representation, a set of object queries, and a set of transformed prior object embeddings extracted at a previous time step; By decoding the extracted set of object embeddings, a position and a category of each road object in the one or more road objects are output.

13. A vehicle comprising a system according to claim 12.

Images