Computer-implemented method for generating control command for automatic driving vehicle

Abstract

To provide a computer-implemented method for generating a control command for an automatic driving vehicle.SOLUTION: The method includes: a step (S10) for detecting sensor data using at least one sensor of a vehicle; a step (S12) for generating an input text from the sensor data; a step (S14) for interpreting the input text by using an algorithm of mechanical learning; and a step (S16) for generating a control command for the vehicle, from the interpreted input text.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to the prediction of the movements of other road users and the generation of control commands in the context of automated driving. [Background technology]

[0002] In a world where technology and mobility are rapidly developing, autonomous driving is inevitably becoming a focus of research and industry. The concept of vehicles able to navigate autonomously through traffic networks has already made significant progress, and vehicle manufacturers and technology companies alike are faced with a series of complex challenges. One of the central challenges is the accurate prediction of the movements of road users, whether they are other vehicles, pedestrians or cyclists. These predictions form the basis for reliable and efficient vehicle control in a highly networked and dynamic traffic environment.

[0003] Automated driving promises not only to improve road safety, but also optimized traffic flow coordination and reduced fuel consumption. To realize these benefits, however, an automated vehicle must not only perceive its environment, but also be able to accurately predict the intentions and actions of other road users. These predictions are crucial to be able to react in a timely manner, whether that be by adjusting its own driving speed, by changing lanes, or by avoiding a collision.

[0004] Predicting the movements of other road users is a complex process based on a combination of advanced technology and data processing. Autonomous vehicles are equipped with a wide palette of sensors, including lidar, radar, camera and ultrasonic sensors. These sensors continuously detect information about the vehicle's environment, such as the position, speed and direction of movement of other road users. The data from these sensors is collected in real time and is used as the basis for predictive algorithms.

[0005] The collected sensor data is combined and analyzed in a process called data fusion, where different data sources are combined to generate a comprehensive picture of the environment. Using advanced algorithms for object recognition, road users are identified and their locations and movement patterns are recognized.

[0006] After recognizing road users, the autonomous vehicle models the behavior of these users. The models used here are often based on machine learning and can use past movement data to predict how road users will behave in certain situations. Such models can be complex and can take into account various factors such as traffic regulations, road types, weather conditions and individual behavior patterns.

[0007] Based on the detected data and the behavioral model, the autonomous vehicle makes predictions about the future movements of the recognized road users. These predictions are often represented as trajectories that describe the expected path and speed of the road users. This process is called trajectory prediction.

[0008] Because road user movements are often uncertain and variable, modern automated vehicles also consider the uncertainty in their predictions: Probability distributions and uncertainty measures are used to estimate different possible scenarios and assess the risk of a collision or other undesirable events.

[0009] For the control of the automated vehicle, the predictions are continuously updated while the environment is changing: if new information is detected or the behavior of road users changes, the automated vehicle adapts its predictions accordingly.

[0010] Overall, accurate prediction of the behavior of other road users in an autonomous driving context requires a tight integration of sensor systems, data processing, machine learning, and probabilistic modeling. The ability to accurately predict traffic situations is crucial to ensure that autonomous vehicles can react appropriately and interact reliably with their environment. Summary of the Invention [Problem to be solved by the invention]

[0011] In this context, the problem underlying the invention is to propose a method for generating control commands for an autonomous vehicle.

[0012] The above mentioned problem is solved by the subject matter of the independent claims. [Means for solving the problem]

[0013] Disclosure of the Invention According to a first aspect of the present invention, the above mentioned problem is solved by a computer implemented method for generating control instructions for an autonomous vehicle, the method comprising: - detecting sensor data by at least one sensor of the vehicle; generating an input text from the sensor data; - interpreting the input text using a machine learning algorithm; generating control instructions for the vehicle from the interpreted input text; Includes.

[0014] The input text may in particular be a predefined text prompt into which the sensor data is inserted. To this end, the computation unit processing the sensor data may for example encode the sensor data in such a way that the sensor data can be automatically inserted at the appropriate position in the text prompt.

[0015] The computational unit uses machine learning algorithms for the interpretation.

[0016] Machine learning algorithms are algorithms developed to automatically recognize patterns and relationships in data and make predictions or take decisions, i.e., they are created by training on existing data and can then be applied to new, unknown data to generate predictions or classifications.

[0017] Machine learning algorithms can take a variety of forms, such as linear models, decision trees, support vector machines, neural networks, and many others. Algorithms are optimized by learning from training data by recognizing patterns and rules to make the best predictions or classifications for new data.

[0018] The efficiency of machine learning algorithms depends on a variety of factors, including the quality and quantity of training data, the choice of algorithm, model configuration, and model evaluation based on evaluation metrics. Models are continuously improved and optimized to maximize accuracy and performance.

[0019] The machine learning algorithm is configured such that the algorithm analyzes and evaluates the text message after it is received. For this purpose, the text message is used word by word as an input vector for the machine learning algorithm. For a large number of possible responses to the interpreted input text, a probability of accuracy is determined. The accuracy is encoded in the model of the machine learning algorithm by a previous training. In doing so, the algorithm tries to predict as accurate a trajectory as possible, which can be used to generate appropriate control commands.

[0020] Control commands for an autonomous vehicle may relate, for example, to the steering, acceleration and / or braking behavior of the vehicle. In modern vehicles, individual wheels may be braked or driven individually to improve control of the vehicle. Thus, a control command may essentially be a vector that includes multiple instructions for various subcomponents of the vehicle.

[0021] Machine learning algorithms provided for text processing are configured differently than those that simply process numbers. Thus, the generation of control instructions from interpreted input text is an alternative approach to the traditional generation of control instructions. Typically, sensor data is used raw for trajectory prediction and then converted into a predicted trajectory.

[0022] The invention therefore constitutes an alternative to methods in which control instructions are not generated from the text, which solves the problem of the invention.

[0023] In one embodiment, in addition to the sensor data, HD maps are also used to generate the input text. The term "HD maps" refers to high-resolution digital maps created specifically for autonomous vehicles. HD maps go far beyond traditional road maps because they contain additional details and information that are critical for accurate navigation and control of autonomous vehicles.

[0024] HD maps provide a very high degree of accuracy regarding the course of the road, road markings, traffic signs, buildings and other objects. This is important to enable an automated vehicle to accurately understand its environment. Furthermore, HD maps may contain information about the road geometry, including the number of lanes, the direction of the road, the possibilities for turning and reversing, and the connection points between different roads. HD maps may include traffic signs, traffic signals and the lighting phases of traffic lights. This information is important to enable an automated vehicle to accurately interpret traffic rules and traffic signals.

[0025] HD maps may also include information about obstacles, construction sites, accident sites, and other potential hazards. Because road and traffic conditions may change, it is important that HD maps are updated regularly. Some systems can also collect real-time data from other vehicles to provide accurate information about traffic flow and road conditions.

[0026] In one embodiment, the machine learning algorithm is a neural network.

[0027] Artificial neural networks are particularly suitable for recognizing patterns, based on their structure, which consists of neurons arranged in layers. The movement of other road users is governed by physical laws such as inertia. Furthermore, it can be assumed that most road users follow predefined paths, which are lanes or footpaths or cycle paths. These laws and boundary conditions form patterns, which are highly complex. These patterns can be recognized particularly effectively by artificial neural networks.

[0028] In one embodiment, the neural network includes multiple layers, with at least one layer dedicated to interpreting the input text.

[0029] Typically, a neural network includes multiple layers. The number of layers may vary depending on the use of the neural network. Typically, a neural network has an input layer, one or more hidden layers, and an output layer.

[0030] The input layer is the starting point of the neural network. Here, input data is fed into the network. These fed data can be, for example, text or other numerical information. For example, text is cut into smaller parts (=tokens) by a so-called tokenizer. These tokens can then be converted into numerical values ​​by embedding.

[0031] The hidden layers are located between the input layer and the output layer. Each hidden layer is made up of neurons, also called "nodes" or "units". In these hidden layers, calculations are performed to extract patterns and features from the input data. This is where the interpretation of the input text is performed.

[0032] The output layer provides the results of the neural network. Depending on the task, these results may have one output or multiple outputs. The output of the output layer may for example include one or multiple control commands for the autonomous vehicle, but may also include a trajectory prediction, which is then converted into a control command by another module.

[0033] Furthermore, a neural network may include multiple sub-networks, where each sub-network has a unique task: interpreting the input text may be the task for which a unique sub-network is configured.

[0034] The sub-network setup advantageously allows the sub-network to have its own cost function during training, which allows the functionality, text interpretation, to be controlled and optimized independently.

[0035] In one embodiment, the neural network or sub-networks of the neural network are trained with a large language model to interpret the input text.

[0036] Large-scale language models are a term for language models that use large amounts of text data to learn natural language interpretation and generation capabilities. These models are frequently used in mechanical text processing and can perform a variety of tasks, such as mechanical translation, text generation, text classification, and question answering systems. Large-scale language models are special models of neural networks. Preferably, the neural network is trained on the "Trajectory Prediction Dataset" using the large-scale language model.

[0037] Due to their structure, neural networks can be particularly well used for text processing, especially when they are trained with large language models.

[0038] In another embodiment, the large-scale language model comprises a Transformer architecture. The Transformer architecture, also called simply Transformer for short, is a special architecture for neural networks, which is based on the mechanism of attention and is used to process sequences such as text. The Transformer architecture has proven to be very powerful in mechanical text processing.

[0039] The main idea behind the Transformer is that it is fully based on the mechanism of attention to model relationships in the text. Traditional sequential models such as LSTM or RNN have difficulty capturing long-term dependencies efficiently. In contrast, the Transformer has direct access to the entire input text and can model relationships between words in parallel, which significantly improves training speed and modeling performance.

[0040] The Transformer architecture consists of a stack of similar layers, where each layer is divided into two main components: a Multi-Head Self-Attention mechanism and a Positional Forward Propagation network.

[0041] The multi-headed self-attention mechanism is the core of the Transformer. Here, the input text is transformed into a query, keys and values. For each word, a so-called attention weight is calculated, which indicates how relevant other words in the text are to this word. Each word can now get its own representation by a weighted combination of the values ​​of these other words. Here, a so-called word embedding vector is generated. For example, each word in the input is decomposed into tokens X=(x1,...,x_N), and then these tokens are transformed into a numerical representation by embedding, where N is the number of tokens and x_n are the features of the token.

[0042] The attention weights are computed multiple times in the so-called "head" to generate different representations of the attention context. These results are then concatenated and again combined by a linear transformation.

[0043] The position-wise forward propagation network contains two linear transformations that are applied independently to each word. This is the non-sequential part of the architecture because each vector is processed separately.

[0044] Since the transformer architecture does not incorporate information about the order of words, positional information is added through positional encodings. These positional encodings are appended to the word embedding vectors to take into account the effect of position.

[0045] The transformer architecture can be split into an encoder and a decoder, where an input sequence is processed and an output sequence is generated. For many tasks, such as mechanical text processing, multiple transformer layers can be used for the encoder and the decoder to improve the performance of the model. Within the framework of the present invention, the decoder can be omitted in the architecture, since no output text is generated.

[0046] Overall, the Transformer architecture advantageously enables highly parallel and efficient processing of text, which leads to improved performance in text processing tasks. By using the attention mechanism, the model is able to access relevant information in an expected way and recognize complex dependencies in the input text, which are then used to generate control instructions.

[0047] In one embodiment, the sensor data includes camera capture data and / or radar sensor data.

[0048] The use of camera and radar sensor data as sensor data offers several advantages that allow a comprehensive and reliable perception of the vehicle environment. These sensor technologies complement each other and make an important contribution to enabling an autonomous vehicle to accurately interpret its environment and navigate reliably.

[0049] Cameras provide high-resolution images that allow a detailed view of the environment, which is particularly important for accurately identifying and classifying relatively small objects such as traffic signs, pedestrians or cyclists.

[0050] Furthermore, cameras can detect the environment in color, improving the ability of autonomous vehicles to recognize visual features, such as the color of traffic lights or color changes on the road. Modern image processing and deep learning algorithms can recognize and classify these objects in the sensor data.

[0051] Furthermore, the cameras can detect information about the behavior of other road users by analysing their movement patterns and speeds, which is important for predicting their future actions.

[0052] Radar sensors are not vulnerable to light conditions such as darkness, fog or heavy rain. Radar sensors operate based on radiation waves and therefore provide reliable perception in a variety of environmental conditions.

[0053] Radar sensors can measure the distance to an object very accurately, which allows the relative position of the vehicle and other obstacles to be accurately detected. Additionally, radar sensors can accurately detect the speed of other vehicles and road users, which is important to allow accurate prediction of collision avoidance maneuvers and movements. Additionally, radar sensors can provide metadata about the object, including size, speed and direction, which helps distinguish between stationary and moving objects.

[0054] Advantageously, camera capture data and radar sensor data are complementary since they are able to perceive different aspects of the environment. Fusing data from these two sources allows for more comprehensive and more reliable detection of information. By integrating these technologies, autonomous vehicles can navigate more reliably, handle complex traffic scenarios, and react appropriately despite changing conditions.

[0055] In one embodiment, the sensor data includes information regarding the position of the vehicle relative to its environment.

[0056] For example, the environment may contain buildings, vegetation, signs, traffic signs, as well as immovable objects such as road markings. The environment can therefore be used as a reference to correlate its own motion with that of other road users. For example, a camera can detect whether the vehicle is in the center of a lane by measuring the distance to road markings on the left and right of the vehicle.

[0057] Including the environment in the generation of the control instructions advantageously optimizes the generated control instructions.

[0058] In one embodiment, movement vectors are further determined from the sensor data, and control commands are generated from the interpreted input text and the movement vectors.

[0059] In this embodiment, the generation of control instructions is aided by two different approaches that complement each other and act as correctives for the other, which may increase the reliability of the overall system as well as provide better protection against erroneous outputs.

[0060] In another aspect, the invention relates to a computer program having program code which, when run on a computer, performs a computer-implemented method for generating control instructions for an autonomous vehicle as described above.

[0061] In another aspect, the invention relates to a computer readable data carrier having a program code of a computer program which, when executed on a computer, performs a computer implemented method for generating control instructions for an autonomous vehicle as described above.

[0062] In another aspect, the present invention relates to a system for generating control instructions for an autonomous vehicle, configured to perform a computer-implemented method as described above.

[0063] Thus, in general, a method, a computer program, a computer readable data carrier and a system are presented.

[0064] The described configurations and developments can be combined with one another in any combination.

[0065] Other possible configurations, developments and implementations of the invention also include not explicitly mentioned combinations of the features of the invention described above or explained below with respect to the embodiments.

[0066] The accompanying drawings are provided to provide a further understanding of the embodiments of the present invention. These drawings illustrate embodiments and are used in conjunction with the description to explain the principles and concepts of the present invention.

[0067] Other embodiments and many of the advantages discussed above will become apparent with reference to the drawings, in which elements illustrated in the drawings are not necessarily drawn to scale relative to each other. [Brief description of the drawings]

[0068] [Figure 1] FIG. 1 illustrates a system for generating control instructions for an autonomous vehicle. [Diagram 2] FIG. 1 illustrates an alternative system for generating control instructions for an autonomous vehicle. [Diagram 3] FIG. 1 illustrates another alternative system for generating control instructions for an autonomous vehicle. [Figure 4] FIG. 2 illustrates a schematic flow chart of a method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0069] In the various figures of the drawings, identical or functionally equivalent elements, components or parts are labeled with the same reference numerals unless otherwise stated.

[0070] FIG. 1 shows a schematic diagram of a system 10 including a machine learning algorithm, in particular a neural network, for generating control instructions.

[0071] The system 10 includes a module 12 for providing sensor data. The sensor data may preferably be present in a form that allows further processing by machine learning algorithms. The current speed of the vehicle may for example be indicated as a numerical value (units m / s).

[0072] Preferably, the previous algorithms and modules process the sensor data so that it can now be incorporated into the text as a numerical value. For this purpose, the system 10 includes a module 14. The module 14 creates the input text by feeding the sensor data of the module 12 into a prompt.

[0073] In the context of text processing with large-scale language models, the term "prompt" refers to output text or instructions given to such a model to perform a particular task. A prompt can be a question, a request, a statement, or any form of input that requests the model to respond to it or to generate more text.

[0074] Prompts serve as initial information that provides the model with context and direction for the desired output. This determines what type of response or continuation the model should generate. The quality of the prompts directly impacts the quality and relevance of the model's response. More precise and clearly crafted prompts can contribute to more accurate and tailored responses.

[0075] When creating a prompt, it is important to be specific and provide clear instructions to achieve the desired outcome and minimize potential misunderstandings or unwanted output.

[0076] For example, the prompt may be, but is not limited to, the following: "You are an expert autonomous vehicle model that can predict future trajectories for a given vehicle, taking into account the vehicle's current and past states, its current lane and potential future lanes, as well as information about other vehicles, pedestrians, drivable areas, and a set of other important features. Task: From a fixed set of trajectories, predict the future trajectory for the next 6 seconds for a given vehicle. Context information: The 2D coordinate system (x,y) is from the predicted vehicle's own field of view. The lane information is encoded as four control points of a cubic Bezier curve. The first and last control points correspond to the start and end of the lane. Predicted vehicle: Category: Vehicles, cars Current speed: 7.12[m / s] Current acceleration: 0.23 [m / s 2 ] Current yaw rate: 4.11 [2π / s] Past (x,y) position in meters, sampled at 2 Hertz: Time [s] x[m] y[m] -2.0 0.83 -15.24 -1.5 0.27 -10.56 -1.0 0.03 -7.07 -0.5 0.01 -3.56 Current lane information (Bezier curves as explained above): xy 0.88 -2.24 1.74 1.87 2.62 5.94 3.47 10.05 Potential departure lane information (Bézier curves as explained above): xy 3.47 10.05 4.59 15.21 5.33 20.46 5.68 25.72 xy 3.47 10.05 5.13 20.95 22.34 21.15 24.43 10.3 Predicted trajectory:

[0077] The last line indicates that the system 10 is requested to generate a trajectory for an autonomous vehicle. To this end, an input text is fed into the neural network 16. In the illustrated embodiment, the neural network includes three layers 18, 20 and 22. In other embodiments, the neural network could include significantly more layers. However, the system 10 can function without the last line "Predicted trajectory" since the LLM is specialized for this task. This means that the weights of the LLM are retrained so that the predicted trajectory is as close as possible to the ground truth.

[0078] Layer 18 is an input layer that receives input text. Layer 20 is a layer that interprets the input message, and layer 22 is a layer that generates a trajectory for the autonomous vehicle from the interpreted input text. Each layer includes a number of neurons 24 or nodes that are connected to each other. It should be understood that the connections shown are schematic. An actual neural network 16 may have a different number of neurons 24 in each of these layers that are connected to each other differently than shown.

[0079] In module 26, a trajectory is formed from the outputs of neurons 24 of output layer 22. The trajectory is then transmitted to a further module 28, which generates from it control commands for the vehicle. The trajectory is not itself a variable that the vehicle can handle. Instead, module 28 converts the trajectory into concrete instructions, for example braking with braking force x for duration t and steering to the right by an angle α.

[0080] 2 shows an alternative system 10 including two neural networks 16 and 30. The input to the neural network 30 is generated from sensor data in module 32. This input may include, for example, a two-dimensional map including the location of the environment, possibly present lane markings and / or other road users.

[0081] Neural network 30, like neural network 16, consists of three layers 34, 36 and 38. Layer 34 is an input layer that receives the input of module 32. Layer 36 processes the input and layer 38 produces an output that is converted in a further module 40 into a trajectory for the autonomous vehicle.

[0082] There are now two trajectories: one generated by module 26 and one generated by module 40. Since an autonomous vehicle can only follow one trajectory, the two trajectories have to be evaluated, whereby only one trajectory is converted into a control command. The comparison and evaluation of these trajectories takes place in a further module 42, so that the trajectory that remains can then be transmitted to module 28.

[0083] In this embodiment, the input text and interpretations of the input text are used as corrective measures for other data processing techniques, or vice versa.

[0084] 3 illustrates another embodiment of system 10. The illustrated system 10 includes a neural network 16 that is composed of three layers of neurons 24. One portion of the neurons in the input layer 18 receive input data from module 32. The other portion of the neurons 24 in the input layer 18 receive input text from module 14.

[0085] Neural network 16 is configured to process input from module 32 in layer 20 along with input text from module 14. Output is then generated in output layer 22, which is processed by module 26 into a trajectory. The trajectory is processed by module 28 into control commands for the autonomous vehicle.

[0086] In this embodiment, the input text and the interpretation of the input text are processed in parallel and complementary to the input data from module 32 .

[0087] 4 shows a schematic flow diagram of a method according to an embodiment. In step S10, sensor data is first detected by a sensor of the autonomous vehicle. The sensor data may in particular be camera capture data or radar sensor data.

[0088] In step S12, an input text is generated from the sensor data, which may include, for example, supplementing a prompt with the sensor data.

[0089] In step S14, the input text is interpreted using machine learning algorithms, in particular neural networks pre-trained with large-scale language models, and then in step S16, the input text is converted from trajectory predictions into control commands for the autonomous vehicle.

[0090] Steps S10 to S16 are repeated at regular intervals to ensure continuous control of the autonomous vehicle. For example, steps S10 to S16 may be performed every millisecond.

Claims

1. 1. A computer-implemented method for generating control instructions for an autonomous vehicle, comprising: - detecting sensor data by at least one sensor of the vehicle (S10); - generating an input text from said sensor data (S12); - interpreting the input text using a machine learning algorithm (S14); - generating control instructions for the vehicle from the interpreted input text (S16); A computer-implemented method comprising:

2. The machine learning algorithm is a neural network (16).

10. The computer-implemented method of claim 1.

3. the neural network (16) includes a number of layers (18, 20, 22), at least one layer being dedicated to the interpretation of the input text; 3. The computer-implemented method of claim 2.

4. the neural network (16) or a sub-network of the neural network is trained with a large-scale language model to interpret the input text; A computer-implemented method according to claim 2 or 3.

5. The sensor data includes camera capture data and / or radar sensor data. A computer-implemented method according to any one of claims 1 to 4.

6. The sensor data includes information regarding the position of the vehicle relative to its environment. A computer-implemented method according to any one of claims 1 to 5.

7. A movement vector is further determined from the sensor data, and the control command is generated from the interpreted input text and the movement vector. A computer-implemented method according to any preceding claim.

8. 1. A computer program having a program code, comprising:

8. A computer program product, which when executed on a computer, performs a computer-implemented method for generating control instructions for an autonomous vehicle according to any one of claims 1 to 7.

9. A computer readable data carrier carrying a program code of a computer program, 8. A computer readable data carrier, which, when the computer program is executed on a computer, performs a computer implemented method for generating control instructions for an autonomous vehicle according to any one of claims 1 to 7.

10. A system (10) for generating control instructions for an autonomous vehicle, comprising: A system (10) configured to perform the computer-implemented method of any one of claims 1 to 7.

Images