Computer-implemented method for generating control instructions for autonomous vehicle

Abstract

The invention relates to a computer-implemented method for generating control instructions for an autonomous vehicle, comprising the following steps: detecting sensor data (S10) by at least one sensor of the vehicle; -generating an input text from the sensor data (S12); -interpreting the input text using a machine learning algorithm (S14); and generating a control command (S16) of the vehicle as a function of the interpreted input text.

Description

Technical Field

[0001] The invention relates to the prediction of the movements of other traffic participants and the generation of control commands in the context of autonomous driving. Background Art

[0002] In a world where technology and mobility are developing at a rapid pace, autonomous driving is inexorably becoming a research and industry focus. Vision for vehicles that can autonomously navigate through traffic has made impressive progress and presents a complex set of challenges not only to car manufacturers but also to technology companies. One of the core challenges lies in the precise prediction of the movements of traffic participants, whether other vehicles, pedestrians or cyclists. These predictions form the basis for safe and efficient vehicle control in increasingly networked and dynamic traffic environments.

[0003] Autonomous driving not only promises an increase in traffic safety, but also an optimized traffic flow regulation and a reduction in fuel consumption. However, in order to realize these advantages, autonomous vehicles must not only perceive their environment, but must also be able to accurately predict the intentions and actions of other traffic participants. These predictions are necessary in order to be able to react in time, whether by adjusting their own driving speed, changing lanes or avoiding collisions.

[0004] Predicting the movements of other traffic participants is a complex process based on a combination of advanced technologies and data processing. Autonomous vehicles are equipped with a wide range of sensors, including lidar, radar, cameras and ultrasonic sensors. These sensors continuously detect information about the vehicle's environment, such as the position, speed and direction of movement of other traffic participants. The data from these sensors is detected in real time and used as the basis for prediction algorithms.

[0005] The collected sensor data is combined and analyzed in a process called data fusion. Here, different data sources are combined to create a comprehensive image of the environment. With the help of advanced algorithms for object recognition, traffic participants are identified and their positions and movement patterns are recognized.

[0006] After identifying traffic participants, the autonomous vehicle models the behavior of these participants. The models used here are often based on machine learning and can utilize historical motion data in order to make predictions about how traffic participants will likely behave in determined situations. Such models can be complex and take into account various factors, such as traffic regulations, road types, weather conditions, and individual behavior patterns.

[0007] Based on the detected data and the behavior model, the autonomous vehicle creates predictions about the future movements of the identified traffic participants. These predictions are often represented as trajectories, which describe the expected routes and speeds of the traffic participants. This process is called trajectory prediction.

[0008] Since the movements of traffic participants are often uncertain and variable, modern autonomous vehicles also take into account the uncertainty of their predictions. Probability distributions and uncertainty measures are used to estimate various possible scenarios and assess the risk of a collision or other undesirable events.

[0009] To control the autonomous vehicle, the predictions are continuously updated during changes in the environment. When new information is detected or the behavior of traffic participants changes, the autonomous vehicle adjusts its predictions accordingly.

[0010] Overall, accurately predicting the movements of other traffic participants in the context of autonomous driving requires a tight integration of sensor systems, data processing, machine learning, and probabilistic modeling. The ability to accurately foresee traffic situations is critical in order to ensure that: autonomous vehicles can react appropriately and interact safely with their environment. Summary of the invention

[0011] Against this background, the object of the present invention is to specify a method for generating control commands for an autonomous vehicle.

[0012] This object is achieved by the subject matter of the independent claims.

[0013] According to a first aspect of the invention, the object is achieved by a computer-implemented method for generating control commands for an autonomous vehicle, the method comprising the following steps:

[0014] - detecting sensor data by at least one sensor of the vehicle;

[0015] - generating input text based on the sensor data;

[0016] - interpreting the input text using a machine learning algorithm; and

[0017] - generating control instructions for the vehicle based on the interpreted input text.

[0018] The input text may in particular be a predefined text prompt into which the sensor data is inserted. For this purpose, a computing unit processing the sensor data may for example encode the sensor data so that the sensor data can be automatically inserted into a suitable position in the text prompt.

[0019] The computing unit uses machine learning algorithms for interpretation.

[0020] A machine learning algorithm is an algorithm that is developed to automatically recognize patterns and relationships in data and make predictions or decisions. The algorithm is created by training with existing data and can then be applied to new, unknown data to produce predictions or classifications.

[0021] Machine learning algorithms can take various forms such as linear models, decision trees, support vector machines, neural networks, etc. The algorithm is optimized by learning from the training data in such a way that it identifies patterns and rules in order to make the best predictions or classifications for new data.

[0022] The effectiveness of a machine learning algorithm depends on various factors, including the quality and quantity of training data, the choice of algorithm, model configuration, and the evaluation of the model based on evaluation metrics. Models are continuously improved and optimized to maximize accuracy and performance.

[0023] The machine learning algorithm is configured so that the algorithm analyzes and evaluates the text message after it is obtained. To this end, the text message is used word by word as an input vector for the machine learning algorithm. For the various response possibilities to the interpreted input text, the probability of correctness is determined. This correctness is encoded into the model of the machine learning algorithm through previous training. Here, the algorithm attempts to predict trajectories that are as accurate as possible, which can be used to generate suitable control instructions.

[0024] The control instructions of an autonomous vehicle may relate, for example, to the steering, acceleration and / or braking behavior of the vehicle. In modern vehicles, individual wheels can be braked or driven individually in order to improve the control of the vehicle. Therefore, the control instructions can in principle be a vector with multiple instructions to different subcomponents of the vehicle.

[0025] Machine learning algorithms designed for text processing are constructed differently from algorithms that process only numbers. Generating control instructions based on interpreted input text is therefore an alternative to conventional generation of control instructions. Typically, sensor data are used in their raw form for trajectory prediction, which is then calculated into a predicted trajectory.

[0026] The present invention therefore forms an alternative to methods which do not generate control commands from a text. This achieves the object of the present invention.

[0027] In one embodiment, in addition to the sensor data, HD-Maps are used to generate the input text. The term "HD-Map" refers to high-resolution digital maps created specifically for autonomous vehicles. HD-Maps go far beyond conventional road maps because they contain additional details and information that are critical for accurate navigation and control of autonomous vehicles.

[0028] HD maps provide a very high degree of accuracy, which concerns the course of the road, lane markings, traffic signs, buildings and other objects. This is important so that autonomous vehicles can accurately understand their environment. In addition, the HD map can contain information about the topology of the road, including the number of lanes, directional lanes, turning and turning possibilities, and connection points between different roads. HD maps can contain traffic signs, traffic signals and the light cycles of traffic lights. This information is important so that autonomous vehicles can accurately interpret traffic regulations and signals.

[0029] HD maps can also contain information about obstacles, construction sites, accident sites, and other potential hazards. Since road conditions and traffic conditions may change, it is important to update HD maps regularly. Some systems can also collect real-time data from other vehicles to provide accurate information about traffic flow and road conditions.

[0030] In one embodiment, the machine learning algorithm is a neural network.

[0031] Artificial neural networks are particularly suitable for recognizing patterns due to their structure consisting of neurons arranged in layers. The movements of other traffic participants are influenced by physical laws, such as inertia. In addition, it can be assumed that most traffic participants follow predefined paths, such as lanes or pedestrian or bicycle paths. These laws and boundary conditions form (albeit highly complex) patterns. These patterns can be recognized particularly effectively by artificial neural networks.

[0032] In one embodiment, the neural network includes a plurality of layers, and at least one layer is configured to interpret the input text.

[0033] Typically, a neural network includes multiple layers. Depending on the purpose of the neural network, the number of layers can be different. In general, a neural network has an input layer, one or more hidden layers, and an output layer.

[0034] The input layer is the starting point of the neural network. Here, input data is fed into the network. These fed data can be, for example, text or other digital information. For example, the text is cut into smaller parts (= tokens) by so-called tokenizers. These tokens can then be converted into numerical values ​​by embedding.

[0035] Hidden layers are located between the input layer and the output layer. Each hidden layer consists of neurons also called "nodes" or "units". Calculations are performed in these hidden layers in order to extract patterns and features from the input data. Interpretation of the input text takes place here.

[0036] The output layer provides the results of the neural network. Depending on the task, this output layer can have a single output or multiple outputs. For example, the output of the output layer can include one or more control instructions for the autonomous vehicle, but it can also include trajectory predictions, which are then translated into control instructions by another module.

[0037] Furthermore, the neural network can contain multiple subnetworks, each of which has its own task. Interpretation of the input text can be such a task, for which its own subnetwork is set up.

[0038] The provision of the subnetworks advantageously results in that the subnetworks have their own cost function during training, so that the function, ie the interpretation of the text, can be controlled and optimized independently.

[0039] In one embodiment, the neural network or a sub-network thereof is trained using a large language model for interpreting the input text.

[0040] Large language models are terms related to language models that use huge amounts of text data in order to learn the ability to interpret and generate natural language. These models are often used in machine text processing and can perform various tasks such as machine translation, text generation, text classification, question answering systems, etc. Large language models are special models of neural networks. Preferably, the neural network is trained using a large language model in view of the "trajectory prediction dataset".

[0041] Due to their structure, neural networks can be used particularly well for text processing, especially when they are trained using large language models.

[0042] In another embodiment, the large language model includes a transformer architecture. The transformer architecture (also referred to as transformer for short) is a specific architecture of a neural network based on an attention mechanism and used to process sequences, such as text. These transformer architectures have been shown to be very powerful in machine text processing.

[0043] The main idea behind the transformer is that it is completely based on the attention mechanism to model the relationships within the text. Traditional sequential models such as LSTM or RNN have difficulty detecting long dependencies effectively. The transformer can directly access the entire input text and model the relationships between words in a parallel manner, which significantly improves training speed and model performance.

[0044] The Transformer architecture consists of a stack of identical layers. Each layer is divided into two main components: a Multi-Head Self-Attention Mechanism and a Position-wise Feedforward Network.

[0045] The multi-head self-attention mechanism is the core of the transformer. Here, the input text is converted into a query, a key, and a value. For each word, a so-called attention weight is calculated, which describes how other words in the text are related to the word. Each word can then obtain its representation through a weighted combination of the values ​​of other words. Here, a so-called word embedding vector is generated. For example, each word of the input is decomposed into word units X = (x1,...,x_N), and these word units are then converted into numerical representations through embedding. Here, N is the number of word units, and x_n is the feature of the word unit.

[0046] The attention weights are computed multiple times in so-called “heads” in order to generate different representations of the attention context. These results are then concatenated and combined again via a linear transformation.

[0047] The position-wise feed-forward network consists of two linear transformations, which are applied to each word independently. This involves a non-sequential part of the architecture, as each vector is processed individually.

[0048] Since the Transformer architecture has no built-in information about the order of words, position information is added through positional encodings. These positional encodings are added to the word embedding vectors in order to take into account the effect of position.

[0049] The transformer architecture can be divided into an encoder and a decoder. The input sequence is processed in the encoder and the output sequence is generated in the decoder. For some tasks, such as machine text processing, multiple transformer layers can be used for the encoder and decoder to improve the performance of the model. Since no output text is generated within the scope of the present invention, the decoder can be discarded in the architecture.

[0050] Overall, the Transformer architecture enables highly parallel and efficient processing of text in an advantageous manner, which leads to improved performance in text processing tasks. By using an attention mechanism, the model can access relevant information in a targeted manner and identify complex dependencies within the input text, which are then used to generate control commands.

[0051] In one specific embodiment, the sensor data include camera recordings and / or radar sensor data.

[0052] The use of camera recordings and radar sensors as sensor data results in a number of advantages that enable a comprehensive and reliable perception of the vehicle's surroundings. These sensor technologies complement each other and contribute decisively to the autonomous vehicle's accurate interpretation of its surroundings and the ability to navigate safely.

[0053] The cameras provide high-resolution images, which enable a detailed view of the environment. This is particularly important in order to accurately recognize and classify smaller objects such as road signs, pedestrians or bicycles.

[0054] In addition, cameras can detect the environment in color, which improves the ability of autonomous vehicles to recognize visual features, such as the color of traffic lights or color changes on the road. Modern image processing and deep learning algorithms enable the recognition and classification of these objects in sensor data.

[0055] Furthermore, cameras can detect information about the behavior of other road users by analyzing their movement patterns and speeds. This is crucial for predicting the future actions of other road users.

[0056] Radar sensors are not sensitive to lighting conditions such as darkness, fog or heavy rain. They operate on the basis of radio waves and therefore provide reliable perception in a wide range of environmental conditions.

[0057] Radar sensors can measure the distance to objects very accurately. This enables accurate detection of the relative position of the vehicle and other obstacles. In addition, radar sensors can precisely detect the speed of other vehicles and traffic participants. This is necessary in order to enable accurate predictions of maneuvers and movements to avoid collisions. In addition, radar sensors can provide metadata about objects, including size, speed and direction, which helps to distinguish between stationary and moving objects.

[0058] Camera recordings and radar sensor data complement each other in an advantageous way, as they can each perceive different aspects of the environment. The fusion of data from both sources enables more comprehensive and more reliable detection of information. By integrating these technologies, autonomous vehicles can navigate more safely, master complex traffic scenarios, and react appropriately despite changing conditions.

[0059] In one specific embodiment, the sensor data include information about the position of the vehicle relative to its surroundings.

[0060] The environment may contain, for example, immovable objects such as buildings, vegetation, signs and traffic signs, but also lane markings. The environment can thus be used as a reference in order to correlate one's own movements and the movements of other road users with one another. For example, the camera can detect whether the vehicle is in the middle of the lane by measuring the distances to the lane markings to the left and right of the vehicle.

[0061] Including the environment in the generation of the control commands advantageously leads to an optimization of the generated control commands.

[0062] In one specific embodiment, a movement vector is also determined as a function of the sensor data and a control command is generated as a function of the interpreted input text and the movement vector.

[0063] In this embodiment, the generation of the control command is based on two different methods, which complement each other and serve as a correction to the other method. This can increase the failsafety of the entire system, but also increase the protection against erroneous outputs.

[0064] In another aspect, the invention relates to a computer program having a program code in order to perform the computer-implemented method for generating control instructions for an autonomous vehicle as described above, when the computer program is executed on a computer.

[0065] In another aspect, the invention relates to a computer-readable data carrier having a program code of a computer program in order to perform a computer-implemented method for generating control instructions for an autonomous vehicle as described above when the computer program is executed on a computer.

[0066] In another aspect, the invention relates to a system for generating control instructions for an autonomous vehicle, wherein the system is configured to perform the computer-implemented method as described above.

[0067] Overall, therefore, a method, a computer program, a computer-readable data carrier and a system are described.

[0068] The described embodiments and developments can be combined with one another as desired.

[0069] Further possible embodiments, improvements and implementations of the invention also include combinations of features of the invention described above or below with respect to the exemplary embodiments that are not explicitly mentioned. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings should facilitate a further understanding of the embodiments of the present invention. These drawings illustrate the embodiments and, together with the description, are used to explain the principles and concepts of the present invention.

[0071] Further embodiments and many of the advantages mentioned ensue in view of the drawings.The illustrated elements of the drawings are not necessarily shown to scale relative to one another.

[0072] Figure 1 A system for generating control instructions for an autonomous vehicle is shown;

[0073] Figure 2 An alternative system for generating control commands for an autonomous vehicle is shown;

[0074] Figure 3 Another alternative system for generating control commands for an autonomous vehicle is shown; and

[0075] Figure 4 The sequence of a method according to one embodiment is schematically shown.

[0076] In the various figures of the drawings, unless otherwise indicated, the same reference numerals indicate the same or functionally identical elements, components or parts. DETAILED DESCRIPTION

[0077] Figure 1 A system 10 is schematically shown which includes a machine learning algorithm, in particular a neural network, for generating control commands.

[0078] The system 10 comprises a module 12 for providing sensor data. The sensor data may preferably be in a form in which it can be further processed by a machine learning algorithm. For example, the current speed of the vehicle may be specified as a numerical value in m / s.

[0079] Preferably, the sensor data has been processed by previous algorithms and modules so that it can now be incorporated into the text as a value. To this end, the system 10 comprises a module 14. The module 14 creates an input text by feeding the sensor data of the module 12 into the prompt.

[0080] In the context of text processing by large language models, the term "prompt" refers to the initial text or instructions given to such a model in order to perform a determined task. A prompt can be a question, request, description, or any form of input that requests the model to react to it or produce additional text.

[0081] The prompt acts as initial information that provides the model with context and direction for the desired output. The model determines what type of response or continuation the model should generate. The quality of the prompt directly affects the quality and relevance of the model's response. Precise, clearly expressed prompts can help obtain more accurate and targeted responses.

[0082] It is important in the expression of prompts to be specific and give clear instructions in order to achieve the desired results and minimize possible misunderstandings or undesired outputs.

[0083] The prompt may appear, for example (but not limited to), as follows:

[0084] “You are an expert self-driving car model that is able to predict the future trajectory of a given vehicle while incorporating its current and past states, its current and likely future lanes, and also information about other vehicles, pedestrians, drivable areas, and other important feature sets.

[0085] Task: Please predict the future trajectory of a given vehicle in the next 6 seconds from a set of fixed trajectories.

[0086] Background information:

[0087] The 2D coordinate system (x, y) comes from the prediction vehicle's own field of view frame.

[0088] The lane information is encoded as four control points of a cubic Bezier curve. The first and last control points match the start and end of the lane.

[0089] Predicted Vehicles:

[0090] Category: Vehicle.Automobile

[0091] Current speed: 7.12 [m / s]

[0092] Current acceleration: 0.23[m / s 2 ]

[0093] Current yaw rate: 4.11[2π / s]

[0094] Past (x,y) positions in meters, sampled at 2 Hz:

[0095]

[0096] Current lane information (Bezier curve, as explained above):

[0097]

[0098]

[0099] Possible exiting lane information (Bezier curve, as explained above):

[0100]

[0101] Predicted trajectory:

[0102] The last line states: the system 10 is requested to generate a trajectory for the autonomous vehicle. To this end, the input text is input into the neural network 16. In the embodiment shown, the neural network includes three layers 18, 20 and 22. In other embodiments, the neural network may include more layers. However, the system 10 will work properly even without the last line "predicted trajectory" because the LLM is specialized for this task. That is, the weights of the LLM are retrained so that the predicted trajectory is as close to the ground truth as possible.

[0103] Layer 18 is an input layer that receives the input text. Layer 20 is a layer that interprets the input text, and layer 22 is a layer that generates the trajectory of the autonomous vehicle based on the interpreted input text. Each layer includes a plurality of neurons 24 or nodes that are interconnected. The links shown are to be understood as schematic. The actual neural network 16 may have a different number of neurons 24 in each of these layers, which are interconnected differently than shown.

[0104] In module 26, a trajectory is formed from the outputs of the neurons 24 of the output layer 22. The trajectory is then submitted to another module 28, which generates control instructions for the vehicle based on the trajectory. The trajectory itself is not yet a variable that can be used by the vehicle. Instead, module 28 translates the trajectory into specific instructions, such as braking with a braking force x for a duration t and turning right at an angle α.

[0105] Figure 2 An alternative system 10 is shown, which comprises two neural networks 16 and 30. The input for the neural network 30 is generated from the sensor data in a module 32. The input can be, for example, a two-dimensional map with position information of the environment, possible lane markings and / or other road users.

[0106] Neural network 30, like neural network 16, consists of three layers 34, 36 and 38. Layer 34 is an input layer that receives input from module 32. Layer 36 processes the input and layer 38 generates an output that is converted in another module 40 into a trajectory for the autonomous vehicle.

[0107] There are now two trajectories generated by modules 26 and 40. These two trajectories must be evaluated so that only one trajectory is converted into control instructions, because the autonomous vehicle can only follow one trajectory. The comparison and evaluation of the trajectories takes place in another module 42 so that the remaining trajectory can then be submitted to module 28.

[0108] In such an embodiment, the input text and its interpretation are used as a correction to another data processing method or vice versa.

[0109] Figure 3Another embodiment of the system 10 is shown. The system 10 shown includes a neural network 16 composed of three layers of neurons 24. A portion of the neurons of the input layer 18 receive input data from the module 32. Another portion of the neurons 24 of the input layer 18 receive input text from the module 14.

[0110] Neural network 20 is constructed so that it processes the input from module 32 together with the input text from module 14 in layer 20. An output is then generated in output layer 22, which is processed into a trajectory by module 26. The trajectory is processed by module 28 into control instructions for the autonomous vehicle.

[0111] In this embodiment, the input text and its interpretation are processed in parallel and complementary to the input data from module 32 .

[0112] Figure 4 The flow of the method according to one embodiment is schematically shown. In step S10, sensor data are first detected by sensors of the autonomous vehicle. The sensor data may be, in particular, camera recordings or radar sensor data.

[0113] In step S12, input text is generated based on the sensor data. Generating the input text may, for example, include supplementing a prompt with the sensor data.

[0114] In step S14, the input text is interpreted. For this purpose, a machine learning algorithm is used, in particular a neural network pre-trained with a large language model. Then in step S16, the input text is converted from the trajectory prediction into control instructions for the autonomous vehicle.

[0115] Steps S10 to S16 are repeated at regular intervals so that continuous control of the autonomous vehicle is ensured. For example, steps S10 to S16 may be performed every millisecond.

Claims

1. A computer-implemented method for generating control instructions for an autonomous vehicle, the method comprising the steps of: - detecting sensor data by at least one sensor of the vehicle (S10); - generating input text according to the sensor data (S12); - interpreting the input text using a machine learning algorithm (S14); as well as - Generating control instructions for the vehicle according to the interpreted input text (S16).

2. The computer-implemented method of claim 1, wherein the machine learning algorithm is a neural network (16).

3. The computer-implemented method of claim 2, wherein the neural network (16) comprises a plurality of layers (18, 20, 22), and wherein at least one layer is configured to interpret the input text.

4. A computer-implemented method according to claim 2 or 3, wherein the neural network (16) or a sub-network thereof is trained using a large language model for interpreting the input text.

5. The computer-implemented method of any of the preceding claims, wherein the sensor data comprises camera images and / or radar sensor data.

6. A computer-implemented method according to any one of the preceding claims, wherein the sensor data comprises a description of the position of the vehicle relative to its environment. 7 . The computer-implemented method according to claim 1 , wherein a motion vector is further determined from the sensor data, and wherein the control instruction is generated from the interpreted input text and the motion vector.

8. A computer program having a program code for executing the computer-implemented method for generating control commands for an autonomous vehicle according to any one of the preceding claims, when the computer program is executed on a computer.

9. A computer-readable data carrier having a program code of a computer program for executing the computer-implemented method for generating control commands for an autonomous vehicle according to any one of claims 1 to 7 when the computer program is executed on a computer.

10. A system (10) for generating control instructions for an autonomous vehicle, wherein the system is configured to perform the computer-implemented method according to any one of claims 1 to 7.

Images