Computer-implemented method for generating a control command for an autonomous vehicle

Abstract

The invention relates to a computer-implemented method for generating a control command for an autonomous vehicle, comprising the steps of: - Acquisition of sensor data (S10) by at least one sensor of the vehicle; - Generating an input text (S12) from the sensor data; - Interpreting the input text (S14) using a machine learning algorithm; and - Generating a control command (S16) for the vehicle from the interpreted input text.

Description

[0001] The invention relates to the prediction of the movement of other road users and the generation of control commands in the context of autonomous driving.

[0002] In a world where technology and mobility are developing at a rapid pace, autonomous driving is inexorably becoming the focus of research and industry. The vision of vehicles that can navigate independently through traffic has already made impressive progress and presents both automakers and technology companies with a number of complex challenges. One of the key challenges is the precise prediction of the movements of road users—be they other vehicles, pedestrians, or cyclists. These predictions form the foundation for safe and efficient vehicle control in an increasingly connected and dynamic traffic environment.

[0003] Autonomous driving promises not only increased road safety, but also optimized traffic flow management and reduced fuel consumption. However, to realize these benefits, autonomous vehicles must not only perceive their surroundings but also be able to accurately predict the intentions and actions of other road users. These predictions are essential for being able to react in a timely manner, whether by adjusting their own driving speed, changing lanes, or avoiding collisions.

[0004] Predicting the movements of other road users is a complex process that relies on a combination of advanced technologies and data processing. Autonomous vehicles are equipped with a wide range of sensors, including lidar, radar, cameras, and ultrasonic sensors. These sensors continuously collect information about the vehicle's surroundings, such as the position, speed, and direction of movement of other road users. The data from these sensors is collected in real time and serves as the basis for the prediction algorithms.

[0005] The collected sensor data is combined and analyzed in a process called data fusion. This process combines various data sources to create a comprehensive picture of the surroundings. Advanced object detection algorithms are used to identify road users and detect their positions and movement patterns.

[0006] After detecting road users, autonomous vehicles model their behavior. The models used are often based on machine learning and can leverage historical movement data to make predictions about how road users might behave in certain situations. Such models can be complex and consider various factors such as traffic regulations, road types, weather conditions, and individual behavior patterns.

[0007] Based on the collected data and behavioral models, the autonomous vehicle makes predictions about the future movements of the detected road users. These predictions are often represented as trajectories that describe the expected route and speed of the road users. This process is called trajectory prediction.

[0008] Because the movements of road users are often uncertain and variable, modern autonomous vehicles also consider uncertainties in their predictions. Probability distributions and uncertainty measures are used to estimate various possible scenarios and assess the risks of collisions or other undesirable events.

[0009] To control an autonomous vehicle, predictions are continuously updated as the environment changes. When new information is acquired or the behavior of road users changes, the autonomous vehicle adjusts its predictions accordingly.

[0010] Overall, the precise prediction of other road users' movements in the context of autonomous driving requires close integration of sensors, data processing, machine learning, and probabilistic modeling. The ability to accurately anticipate traffic situations is crucial to ensuring that autonomous vehicles can react appropriately and interact safely with their environment.

[0011] In this context, the object of the invention is to propose a method for generating a control command for an autonomous vehicle.

[0012] The problem is solved by the subject matter of the independent claims. Disclosure of the invention

[0013] According to a first aspect of the invention, this object is achieved by a computer-implemented method for generating a control command for an autonomous vehicle, comprising the steps: - Collecting sensor data by at least one sensor of the vehicle; - Generating an input text from the sensor data; - Interpreting the input text using a machine learning algorithm; and - Generate a control command for the vehicle from the interpreted input text.

[0014] The input text can, in particular, be a predefined text prompt into which the sensor data is inserted. For this purpose, a processing unit that processes the sensor data can, for example, encode it in such a way that the sensor data can be automatically inserted at the appropriate positions in the text prompt.

[0015] The computing unit uses a machine learning algorithm for interpretation.

[0016] A machine learning algorithm is an algorithm designed to automatically detect patterns and relationships in data and make predictions or decisions. It is created by training on existing data and can then be applied to new, unknown data to generate predictions or classifications.

[0017] A machine learning algorithm can take various forms, such as linear models, decision trees, support vector machines, neural networks, and many others. It is optimized by learning from the training data, identifying patterns and rules to make the best possible predictions or classifications for new data.

[0018] The effectiveness of a machine learning algorithm depends on several factors, including the quality and quantity of training data, the choice of algorithm, the model configuration, and the evaluation of the model using evaluation metrics. The model is continuously improved and optimized to maximize accuracy and performance.

[0019] The machine learning algorithm is configured to analyze and evaluate the text message upon receipt. To do this, the text message is used word by word as an input vector for the machine learning algorithm. A probability of correctness is determined for a variety of possible answers to the interpreted input text. The correctness has been encoded into the machine learning algorithm's model through prior training. The algorithm attempts to predict the most accurate trajectories possible, which can be used to generate an appropriate control command.

[0020] The control commands for the autonomous vehicle can, for example, concern the vehicle's steering, acceleration, and / or braking behavior. In modern vehicles, individual wheels can be braked or driven individually to improve vehicle control. The control command can therefore essentially be a vector with multiple instructions to the vehicle's various subcomponents.

[0021] Machine learning algorithms designed for text processing are structured differently than those that process numbers alone. Thus, generating a control command from the interpreted input text represents an alternative method to conventional control command generation. Typically, the raw sensor data is used for trajectory prediction, which is then processed to produce predicted trajectories.

[0022] The invention thus provides an alternative to methods in which the control commands are not generated from a text. This solves the problem of the invention.

[0023] In one embodiment, HD maps are used to generate the input text in addition to the sensor data. The term "HD map" refers to a high-resolution digital map created specifically for autonomous vehicles. An HD map is far more than a conventional road map, as it contains additional details and information that are crucial for the precise navigation and control of autonomous vehicles.

[0024] An HD map offers very high accuracy regarding road layouts, lane markings, traffic signs, buildings, and other objects. This is important for the autonomous vehicle to accurately understand its surroundings. It can also contain information about the topological structure of roads, including the number of lanes, lane directions, turning and reversing options, and connection points between different roads. HD maps can include traffic signs, traffic signals, and traffic light phases. This information is important for the autonomous vehicle to accurately interpret traffic rules and signals.

[0025] HD maps can also include information about obstacles, construction sites, accident sites, and other potential hazards. Because road and traffic conditions can change, it's important that HD maps are updated regularly. Some systems can also collect real-time data from other vehicles to provide accurate information about traffic flow and road conditions.

[0026] In one embodiment, the machine learning algorithm is a neural network.

[0027] Artificial neural networks are particularly well-suited for pattern recognition due to their structure, consisting of neurons arranged in layers. The movements of other road users are subject to physical laws such as inertia. Furthermore, it can be assumed that most road users follow predefined paths, such as a lane, sidewalk, or bike path. These laws and boundary conditions form patterns, albeit highly complex ones. These patterns can be recognized particularly effectively by an artificial neural network.

[0028] In one embodiment, the neural network comprises multiple layers and at least one layer is dedicated to interpreting the input text.

[0029] Neural networks typically consist of multiple layers. The number of layers can vary depending on the application of the neural network. Typically, neural networks have an input layer, one or more hidden layers, and an output layer.

[0030] The input layer is the starting point of the neural network. This is where the input data is fed into the network. This input data can be text or other numeric information, for example. For example, a text is cut into smaller pieces (tokens) by a so-called tokenizer. The tokens can then be converted into numeric values ​​using embedding.

[0031] The hidden layers lie between the input layer and the output layer. Each hidden layer consists of neurons, also called "nodes" or "units." These hidden layers perform computations to extract patterns and features from the input data. This is where the input text is interpreted.

[0032] The output layer provides the results of the neural network. Depending on the task, it can have a single output or multiple outputs. The output of the output layer can, for example, include one or more control commands for the autonomous vehicle, but also a trajectory prediction, which is then translated into control commands by another module.

[0033] Furthermore, the neural network can comprise multiple subnetworks, each with its own tasks. Interpreting the input text can be one such task, for which a separate subnetwork is set up.

[0034] The advantageous effect of setting up sub-networks is that the sub-network has its own cost function during training, so that the function of interpreting the text can be controlled and independently optimized.

[0035] In one embodiment, the neural network or a subnetwork thereof is trained with a large language model for interpreting the input text.

[0036] A large language model is a term that refers to language models that use massive amounts of text data to learn the ability to interpret and generate natural language. These models are often used in machine text processing and can perform various tasks such as machine translation, text generation, text classification, question-answering systems, and more. Large language models are specialized models of neural networks. The neural network is preferably trained with the large language model on a trajectory prediction dataset.

[0037] Due to their structure, neural networks, especially when trained with a large language model, are particularly well suited for text processing.

[0038] In another embodiment, the Large Language Model includes a transformer architecture. Transformer architectures, also referred to simply as transformers, are specific architectures for neural networks based on the mechanism of attention and used to process sequences, such as text. They have proven extremely powerful in machine text processing.

[0039] The main idea behind the Transformer is that it relies entirely on the attention mechanism to model relationships within a text. Traditional sequential models like LSTM or RNN struggle to efficiently capture long dependencies. The Transformer, on the other hand, can directly access the entire input text and model relationships between words in parallel, significantly improving training speed and model performance.

[0040] The Transformer architecture consists of a stack of identical layers. Each layer is divided into two main components: the multi-head self-attention mechanism and the position-wise feedforward network.

[0041] The multi-head self-attention mechanism is the heart of the Transformer. Here, the input text is converted into a query, a key, and a value. For each word, an attention weight is calculated, which indicates how relevant the other words in the text are to that word. Each word can then receive its representation through a weighted combination of the values ​​of the other words. This generates word embedding vectors. For example, each input word is broken down into tokens X = (x1, ..., x_N), which are then converted into a numerical representation using embedding. Here, N is the number of tokens and x_n is the features of a token.

[0042] The attention weights are calculated multiple times in so-called "heads" to generate different representations of the attention context. The results are then concatenated and recombined using a linear transformation.

[0043] The position-wise feedforward network comprises two linear transformations applied independently to each word. It is a non-sequential part of the architecture because each vector is processed separately.

[0044] Since a transformer architecture has no built-in information about the word order, positional information is added through position encodings. These position encodings are added to the word embedding vectors to account for the influence of positions.

[0045] The transformer architecture can be divided into an encoder and a decoder. The encoder processes the input sequence, and the decoder generates the output sequence. For some tasks, such as machine text processing, multiple transformer layers can be used for the encoder and decoder to increase the model's performance. Since no output text is generated in this invention, a decoder can be omitted from the architecture.

[0046] Overall, the Transformer architecture advantageously enables highly parallel and efficient text processing, leading to increased performance in text processing tasks. By using attention mechanisms, the model can specifically access relevant information and detect complex dependencies within the input text, which are then used to generate control commands.

[0047] In one embodiment, the sensor data includes camera images and / or radar sensor data.

[0048] The use of camera images and radar sensors as sensor data offers several advantages that enable comprehensive and reliable perception of the vehicle's surroundings. These sensor technologies complement each other and play a key role in enabling autonomous vehicles to obtain a precise interpretation of their surroundings and navigate safely.

[0049] Cameras provide high-resolution images that provide a detailed view of the surroundings. This is especially important for accurately identifying and classifying smaller objects such as traffic signs, pedestrians, or bicycles.

[0050] In addition, cameras can capture the environment in color, improving the autonomous vehicle's ability to detect visual features such as traffic light colors or color changes on the road. Modern image processing and deep learning algorithms make it possible to detect and classify these objects in the sensor data.

[0051] Furthermore, cameras can collect information about the behavior of other road users by analyzing their movement patterns and speeds. This is crucial for predicting their future actions.

[0052] Radar sensors are insensitive to lighting conditions such as darkness, fog, or heavy rain. They operate on the basis of radio waves and thus provide reliable detection in a variety of environmental conditions.

[0053] Radar sensors can measure distances to objects with extreme precision. This enables accurate detection of the relative position of vehicles and other obstacles. Radar sensors can also precisely detect the speeds of other vehicles and road users. This is essential for enabling collision-avoidance maneuvers and accurate movement prediction. Radar sensors can also provide metadata about objects, including size, speed, and direction, which helps distinguish between stationary and moving objects.

[0054] Camera images and radar sensor data complement each other advantageously, as they can each perceive different aspects of the environment. Merging data from these two sources enables more comprehensive and reliable information gathering. By integrating these technologies, autonomous vehicles can navigate more safely, handle complex traffic scenarios, and react appropriately despite changing conditions.

[0055] In one embodiment, the sensor data includes information about the position of the vehicle relative to its surroundings.

[0056] The environment may include immobile objects such as buildings, vegetation, signs, and traffic signals, as well as road markings. The environment can therefore be used as a reference to correlate one's own movement with the movement of other road users. For example, a camera can detect whether the vehicle is in the center of a lane by measuring the distance to the road markings to the left and right of the vehicle.

[0057] The inclusion of the environment in the generation of the control command advantageously results in an optimization of the generated control command.

[0058] In one embodiment, motion vectors are further determined from the sensor data and the control command is generated from the interpreted input text and the motion vectors.

[0059] In this embodiment, the generation of control commands is based on two different methods that complement each other and act as correctives for each other. This increases the reliability of the overall system, as well as the protection against incorrect outputs.

[0060] In a further aspect, the invention relates to a computer program with program code for executing a computer-implemented method for generating a control command for an autonomous vehicle as described above when the computer program is executed on a computer.

[0061] In a further aspect, the invention relates to a computer-readable data carrier with program code of a computer program for carrying out a computer-implemented method for generating a control command for an autonomous vehicle as described above when the computer program is executed on a computer.

[0062] In a further aspect, the invention relates to a system for generating a control command for an autonomous vehicle, wherein the system is designed to carry out a computer-implemented method as described above.

[0063] Overall, a method, a computer program, a computer-readable data carrier and a system are specified.

[0064] The described designs and further training courses can be combined as desired.

[0065] Further possible embodiments, developments and implementations of the invention also include combinations of features of the invention described previously or below with regard to the embodiments that are not explicitly mentioned. Short description of the drawings

[0066] The accompanying drawings are intended to provide a further understanding of embodiments of the invention. They illustrate embodiments and, in conjunction with the description, serve to explain principles and concepts of the invention.

[0067] Other embodiments and many of the aforementioned advantages will become apparent upon review of the drawings. The elements illustrated in the drawings are not necessarily drawn to scale.

[0068] They show: Fig. 1 a system for generating a control command for an autonomous vehicle; Fig. 2 an alternative system for generating a control command for an autonomous vehicle; Fig. 3 shows another alternative system for generating a control command for an autonomous vehicle; and Fig. 4 schematically shows the sequence of a method according to an embodiment.

[0069] In the figures of the drawings, the same reference symbols designate the same or functionally equivalent elements, parts or components, unless otherwise stated.

[0070] Fig. 1 schematically shows a system 10 comprising a machine learning algorithm, in particular a neural network for generating a control command.

[0071] The system 10 comprises a module 12 that provides the sensor data. The sensor data can preferably be in a form that can be further processed by the machine learning algorithm. The current speed of the vehicle can, for example, be specified as a numerical value in m / s.

[0072] Preferably, previous algorithms and modules have processed the sensor data in such a way that it can now be incorporated into a text as numerical values. For this purpose, system 10 includes a module 14. Module 14 creates an input text by feeding the sensor data from module 12 into a prompt.

[0073] In the context of text processing by a Large Language Model, the term "prompt" refers to a source text or the instruction given to such a model to perform a specific task. A prompt can be a question, a request, a description, or any form of input that requires the model to respond or generate further text.

[0074] The prompt acts as initial information, providing the model with the context and direction for the desired output. It determines what type of response or continuation the model should generate. The quality of the prompt directly influences the quality and relevance of the model's response. A precise, clearly worded prompt can help obtain more accurate and targeted responses.

[0075] When formulating prompts, it is important to be specific and provide clear instructions to achieve the desired results and minimize potential misunderstandings or unwanted outputs.

[0076] For example, but not limited to, a prompt might look like this: „Y ou are an expert self-driving-car model, that can predict the future trajectory for a given vehicle, while also incorporating its current and past states, its current and possible future lanes and also information about other vehicles, pedestrians, drivable areas and other important sets of features.

[0077] Task: Please predict the future trajectory for the given vehicle for the next 6 seconds, from a set of fixed trajectories.

[0078] Context Information: The 2D coordinate system (x,y) is from the prediction vehicle's own frame of view.

[0079] Lane information is encoded as the 4 control points of a cubic Bezier curve. The first and last control point match with the beginning and end of the lane.

[0080] Prediction Vehicle: Category: vehicle.car Current Speed: 7.12[m / s] Current Acceleration: 0.23[m / s 2 ] Current Yaw rate: 4.11[2π / s]

[0081] Past (x,y) positions in meters, sampled at 2 Hertz: Time[s] x[m] y[m] -2.0 0.83 -15.24 -1.5 0.27 -10.56 -1.0 0.03 -7.07 -0.5 0.01 -3.56

[0082] Current Lane Information (Bezier curve, as explained above): x y 0.88 -2.24 1.74 1.87 2.62 5.94 3.47 10.05

[0083] Possible Outgoing Lane Information (Bezier curve, as explained above): x y 3.47 10.05 4.59 15.21 5.33 20.46 5.68 25.72 x y 3.47 10.05 5.13 20.95 22.34 21.15 24.43 10.3 Predicted trajectory:"

[0084] The last line indicates that the system 10 is being asked to generate a trajectory for the autonomous vehicle. For this purpose, the input text is fed into a neural network 16. In the illustrated embodiment, the neural network comprises three layers 18, 20, and 22. In other embodiments, the neural network may comprise many more layers. However, the system 10 would also function without the last line, "Predicted Trajectory," because the LLM is specialized for this task. This means that the LLM's weights are retrained so that the predicted trajectory is as close to the ground truth as possible.

[0085] Layer 18 is the input layer, which receives the input text. Layer 20 is the layer that interprets the input text, and Layer 22 is the layer that generates a trajectory for the autonomous vehicle from the interpreted input text. Each layer comprises several neurons 24, or nodes, that are interconnected. The interconnection shown is to be understood schematically. Real neural networks 16 can each have a different number of neurons 24 in each layer, which are interconnected differently than shown.

[0086] In a module 26, a trajectory is formed from the output of neurons 24 of output layer 22. The trajectory is then passed to another module 28, which generates a control command for the vehicle from the trajectory. The trajectory itself is not a quantity with which the vehicle can operate. Instead, module 28 translates the trajectory into concrete instructions, for example, braking with braking force x for a period of time t and steering to the right by an angle α.

[0087] Fig. Figure 2 shows an alternative system 10 comprising two neural networks 16 and 30. The input for the neural network 30 is generated from the sensor data in a module 32. This input can include, for example, a two-dimensional map with position information of the surroundings, any lane markings, and / or other road users.

[0088] Like neural network 16, neural network 30 consists of three layers 34, 36, and 38. Layer 34 is an input layer that receives the input from module 32. Layer 36 processes the input, and layer 38 generates an output that is converted into a trajectory for the autonomous vehicle in another module 40.

[0089] Now there are two trajectories generated by modules 26 and 40. The two trajectories must be evaluated so that only one trajectory is converted into a control command, since the autonomous vehicle can only follow one trajectory. The comparison and evaluation of the trajectories are performed in another module 42, so that the remaining trajectory can then be passed on to module 28.

[0090] In this embodiment, the input text and its interpretation are used as a corrective for another data processing method or vice versa.

[0091] Fig. Figure 3 shows another embodiment of a system 10. The illustrated system 10 includes a neural network 16 consisting of three layers of neurons 24. A portion of the neurons of the input layer 18 receives the input data from module 32. Another portion of the neurons 24 of the input layer 18 receives the input text from module 14.

[0092] The neural network 20 is designed to process the input from module 32 together with the input text from module 14 in a layer 20. In the output layer 22, an output is then generated, which is processed into a trajectory by module 26. This trajectory is then processed by module 28 into a control command for the autonomous vehicle.

[0093] In this embodiment, the input text and its interpretation are processed in parallel and in addition to the input data from module 32.

[0094] Fig.Figure 4 schematically shows the flow of the method according to one embodiment. In step S10, sensor data is first acquired from a sensor of the autonomous vehicle. The sensor data can, in particular, be camera recordings or radar sensor data.

[0095] In step S12, an input text is generated from the sensor data. Generating the input text may, for example, involve supplementing a prompt with the sensor data.

[0096] In step S14, the input text is interpreted. A machine learning algorithm, specifically a neural network pre-trained with a large language model, is used for this purpose. The input text is then converted from a trajectory prediction into a control command for the autonomous vehicle in step S16.

[0097] Steps S10 to S16 are repeated at regular intervals to ensure continuous control of the autonomous vehicle. For example, steps S10 to S16 can be executed every millisecond.

Claims

[1] Computer-implemented method for generating a control command for an autonomous vehicle comprising the steps: - Acquiring sensor data (S10) by at least one sensor of the vehicle; - generating an input text (S12) from the sensor data; - interpreting the input text (S14) using a machine learning algorithm; and - Generating a control command (S16) for the vehicle from the interpreted input text. [2] The computer-implemented method of claim 1, wherein the machine learning algorithm is a neural network (16). [3] Computer-implemented method according to claim 2, wherein the neural network (16) comprises a plurality of layers (18, 20, 22) and wherein at least one layer is provided for the interpretation of the input text. [4] A computer-implemented method according to any one of claims 2 or 3, wherein the neural network (16) or a sub-network thereof is trained with a large language model for interpreting the input text. [5] Computer-implemented method according to one of the preceding claims, wherein the sensor data comprises camera recordings and / or radar sensor data. [6] A computer-implemented method according to any one of the preceding claims, wherein the sensor data comprises information on the position of the vehicle relative to its surroundings. [7] Computer-implemented method according to one of the preceding claims, wherein motion vectors are further determined from the sensor data and wherein the control command is generated from the interpreted input text and the motion vectors. [8] A computer program comprising program code for carrying out a computer-implemented method for generating a control command for an autonomous vehicle according to any one of the preceding claims when the computer program is executed on a computer. [9] Computer-readable data carrier with program code of a computer program for executing a computer-implemented method for generating a control command for an autonomous vehicle according to one of claims 1 to 7 when the computer program is executed on a computer. [10] System (10) for generating a control command for an autonomous vehicle, wherein the system is designed to carry out a computer-implemented method according to one of claims 1 to 7.

Images