Automated movement control of a vehicle

Abstract

The invention relates to an automated movement control system (10) for a vehicle, comprising the steps of inputting sensor data (12) from at least one vehicle sensor (14) about the vehicle environment to a data processing model (20) comprising at least one single module (26) with at least one trained artificial neural network as input data, and calculating control data (22) depending on the input data by the data processing model (20), inputting user information (38) of a vehicle user to a user interface model (36) comprising at least one trained artificial neural network, and calculating target data (40) depending on the user information (38) by the user interface model (36).Providing the target data (40) to the at least one individual module (26) of the data processing model (20) and calculating the control data (22) for controlling at least one vehicle function (24) that influences the movement of the vehicle, by means of the data processing model (20) depending on the input data and the target data (40).

Description

[0001] The invention relates to an automated locomotion control system according to claim 1. State of the art

[0002] Vehicles with autonomous driving functions use end-to-end trained models that process sensor data, such as from the vehicle's cameras or LiDAR sensors, to calculate control data for the vehicle's movement. These models include a perception module that detects objects in the vehicle's environment, a prediction module that calculates the subsequent movement of these objects, and a trajectory planning module that uses this information to calculate a safe driving strategy and the optimal trajectory for the vehicle. Disclosure of the invention

[0003] According to the present invention, an automated motion control system with the features of claim 1 is proposed. This improves the personal driving experience when moving around in the vehicle. The vehicle's movement can be more precisely tailored to the needs and preferences of the vehicle user and to the current vehicle environment. The vehicle user's confidence in the automated motion control system can be increased. The vehicle can be operated more energy-efficiently and safely.

[0004] The vehicle can be a motor vehicle, truck, two-wheeled vehicle or a mobile robot.

[0005] Automated motion control can enable automated and / or autonomous driving of the vehicle.

[0006] The term "vehicle user" refers to the person responsible for the movement of the vehicle, who, as someone present in the vehicle, has the authority to change the vehicle's movement.

[0007] The vehicle sensor can be a LiDAR sensor, a radar sensor, an ultrasonic sensor and / or a camera.

[0008] The sensor data can be measurement data from the vehicle sensor or data processed from measurement data from the vehicle sensor, for example, through signal processing. The data processing model calculates the output data from the input data by processing it with at least one artificial neural network. The sensor data can include map data of the vehicle's surroundings.

[0009] The user interface model can be integrated in parallel within the processing chain between the input data of the data processing model and the output of the control data. This means that providing the input data to the data processing model can be done on demand and optionally. The user interface model is implemented separately from the data processing model.

[0010] The sensor data, as input data, is processed by the data processing model using at least one artificial neural network in at least one individual module, and the control data is output as output data. For example, the sensor data is inputted to an input layer of the artificial neural network, and the control data is output by an output layer of the artificial neural network.

[0011] The user information is processed by the user interface model using at least one artificial neural network, and the target data is output. For example, the user information is input as user data to an input layer of the artificial neural network, and the target data is output by an output layer of the artificial neural network.

[0012] In a preferred embodiment of the invention, it is advantageous if the user information includes at least stored route information for a commute. The commute can be a route between the user's residence and workplace, particularly that of the vehicle user.

[0013] The stored route information can be located in a memory that can be accessed for processing by the user interface model. This memory can be located within the vehicle or implemented as cloud storage accessible by the vehicle.

[0014] The stored route information may include details of recorded hazards, closures, or similar issues.

[0015] The stored route information may be located in a database, which is continuously expanded, particularly as the commuter route is used more frequently. This stored route information can be retrieved from the database to generate user information.

[0016] The user interface model can be fine-tuned at least once, and preferably increasingly, using information gathered from one or more trips along the commuter route. This fine-tuning can be performed in the vehicle itself or via a cloud-based computer.

[0017] A preferred embodiment of the invention is advantageous in which position information is assigned to the stored route information. The position information can be obtained by at least one position sensor of the vehicle. The position sensor can be a GPS sensor. The position information can include the current position of the vehicle and / or a planned route of the vehicle.

[0018] User information may also include identifying information of the vehicle user. This identifying information can be obtained through analysis, particularly image, video, and / or audio analysis, of the vehicle user. The analysis may consider the vehicle user's seating position in the vehicle to identify the driver and exclude statements from other occupants. A user profile can be assigned to the identified vehicle user, which influences, modifies, or specifies the user information. The identifying information can be used to assign the vehicle user to a group, for example, through processing on a cloud server. Each group may be assigned its own user interface model. In particular, each vehicle user is assigned its own user interface model.This can further improve the personal driving experience of the vehicle user.

[0019] User information can include individual information about the vehicle user, such as information about their behavior, expectations, abilities, or condition. For example, this individual information can include the user's preferred driving speed at at least one location, particularly a section of road. It can also include the user's driving experience.

[0020] In an advantageous embodiment of the invention, the user information is generated at least from real-time communication data of the vehicle user, which is communicated to the user interface model in real time. The real-time communication data can include information about the immediate or the vehicle's surroundings or the route ahead. The user information can comprise unaltered, processed, or augmented real-time communication data.

[0021] The real-time communication data may include instructions for the movement of the vehicle, information on possible hazards in the vehicle environment, information on events or possible events, conditions or possible conditions, information on conditions or possible conditions in the vehicle environment and / or corrections of intermediate output data from individual modules of the data processing model and / or the output control data of the data processing model, in each case by the vehicle user.

[0022] Instructions for the vehicle's movement can include commands such as "Slow down," "Increase speed," "Child is traveling with us," "Select scenic route," and the like. Warnings can include information such as "Pothole ahead," "Increased risk of aquaplaning," "Road closure ahead," "Lost cargo ahead," or the like. Corrections can include information such as "Vehicle ahead is not a car but a cart," or the like. Corrections, in particular, require an output interface between the data processing model and the vehicle user, for example, a visible display, especially a head-up display.

[0023] In a specific embodiment of the invention, it is advantageous if the real-time communication data is generated from text and / or voice input from the vehicle user. The voice input can be recorded by a microphone in the vehicle. The voice input can be converted into text. The real-time communication data can be calculated from the voice and / or text input. For example, in the case of voice input, the real-time communication data can be in the form of transcoded text. The real-time communication data can comprise unaltered, processed, or prepared text and / or voice input.

[0024] In a specific embodiment of the invention, it is advantageous if the data processing model comprises several individual modules, namely a perception module, a prediction module, a trajectory planning module, and a control module. The individual modules can be arranged sequentially in the processing chain.

[0025] The individual modules can exchange data with each other in a latent space. The output of one module, for example the perception module, can form the input of a subsequent module, for example the prediction module.

[0026] In an advantageous embodiment of the invention, the provision of the target data comprises input to the perception module, the prediction module, and / or the trajectory planning module. For example, the target data can indicate that a road closure exists at a position along the planned commute route, and if this information is entered into the trajectory planning module, an alternative route can be planned and executed to bypass the road closure in a timely manner.

[0027] A preferred embodiment of the invention is advantageous in which the data processing model, including all individual modules, is trained as a whole. The data processing model can be trained uniformly and without targeted training of the individual modules, in particular by end-to-end learning.

[0028] In a specific embodiment of the invention, it is advantageous if the input data are data in a latent feature space. The data present in the latent feature space can be input into the data processing model in such a way. For this purpose, the data processing model can have at least one interface. The interface can be arranged parallel to the input interfaces in the processing chain, starting from the sensor data and leading to the control data of the data processing model. At least one individual module can have such an interface. At least two individual modules can each have such an interface. At least two individual modules can have a single common interface.

[0029] In a specific embodiment of the invention, it is advantageous if the user interface model is a pre-trained multimodal model. The multimodal model can process multiple modalities, i.e., different types of input data. A modality can be, for example, text, image, audio, or video. The user interface model can be trained on very large and diverse datasets. The user interface model can be a deep neural network, in particular with the Transformer, Variational Autoencoder (VAE), and / or Generative Adversarial Network (GAN) structure. The user interface model can be a generative model. Through self-attention mechanisms, the user interface model can efficiently grasp the context of the user information and model relationships between different elements within the user information.The user interface model can be a massive language model (LLM).

[0030] The user interface model may have been trained through unsupervised or self-supervised learning. It may also have been trained using labeled data. The labeled training data may be structured according to the specific module of the data processing model to which the input of the target data can be applied. The labeled training data may include user information, such as the instruction "Make a lane change," and, as a label, the control data—in this example, for performing the lane change. Alternatively, the user interface model may have been trained using backpropagation and gradient descent to incrementally optimize the weights of the network layers, minimizing error (loss) by comparing the model predictions with the actual values.

[0031] The user interface model can consist of multiple layers of encoders and / or decoders. The user interface model can have more than one hundred million, preferably more than one billion, preferably more than 100 billion parameters, which are weights learned during training.

[0032] The invention further relates to a processing device for carrying out the previously described automated motion control. The processing device can be arranged inside the vehicle. Some calculation steps of the automated motion control can also be carried out outside the vehicle, for example on a cloud computer.

[0033] Furthermore, the invention relates to a computer program comprising machine-readable instructions executable on at least one computer, the execution of which involves the method described above.

[0034] Furthermore, the invention relates to a storage unit that is machine-readable and accessible by at least one computer and on which the said computer program is stored.

[0035] Further advantages and advantageous embodiments of the invention will become apparent from the description of the figures and the illustrations. Character description

[0036] The invention is described in detail below with reference to the illustrations. These show, in detail: Fig. 1: An automated locomotion control system in a specific embodiment of the invention. Fig. 2: An automated locomotion control in a further special embodiment of the invention.

[0037] Fig. Figure 1 shows an automated motion control system in a specific embodiment of the invention. The automated motion control system 10 of a vehicle comprises inputting sensor data 12 from at least one vehicle sensor 14 relating to the vehicle's environment, for example, a radar sensor 16 or a camera 18, to a data processing model 20 as input data, and calculating control data 22 based on the input data by the data processing model 20. The control data 22 are used to control at least one vehicle function 24, for example, to control lateral velocity, steering angle, braking function, or the like.

[0038] The data processing model 20 comprises several individual modules 26 with a trained artificial neural network, in particular a perception module 28 that processes the sensor data 12 input, a downstream prediction module 30, a trajectory planning module 32, and a final control module 34 for outputting the calculated control data 22. The data processing model 20 was preferably trained as a whole, including all individual modules 26. This allows even complex relationships and patterns to be captured and taken into account.

[0039] A user interface model 36 operates in parallel with the data processing model 20. User information 38 of a vehicle user is input into the user interface model 36, which has at least one trained artificial neural network, and the user interface model 36 calculates target data 40 based on this user information 38. The user interface model 36 can be a pre-trained multimodal model that has been trained with very extensive and diverse datasets.

[0040] Furthermore, the input data 40 is provided, in particular, to the perception module 28, the prediction module 30, and the trajectory planning module 32 of the data processing model 20. Providing the input data 40 involves inputting it to the respective individual module 26. The input data 40 consists, in particular, of data in a latent feature space, which is made available to each of the individual modules 26 individually.

[0041] The user information 38 is formed at least from real-time communication data 42 of the vehicle user communicated to the user interface model 36 in real time. The real-time communication data 42 can be formed from text input 44 or voice input 46 of the vehicle user and may include, for example, instructions for moving the vehicle, warnings about possible hazards in the vehicle's environment, environmental information about events or possible events, conditions or possible conditions in the vehicle's environment, and / or corrections of intermediate output data 47, here for example, from the perception module as a display in a head-up display, and / or the output control data 22 of the data processing model 20, each by the vehicle user.

[0042] The calculation of the control data 22 for controlling at least one vehicle function 24 that influences the movement of the vehicle by the data processing model 20 is thus dependent on the sensor data 12 as input data and the target data 40.

[0043] For example, the camera can provide sensor data 12 of the vehicle's surroundings, in which an environmental object, such as a pane of glass broken by lost cargo, is located on the road ahead of the vehicle. When calculating the control data 22 by the data processing model 20, this environmental object could be interpreted as belonging to the roadway.However, the vehicle user can, by means of voice input 46, use the real-time communication data 42 as user information 38 to indicate the upcoming hazard to the user interface model 36 in the form of a warning, and the user interface model 36 calculates target data 40 based on this user information 38, which is provided in particular to the perception module 28, whereby the data processing model 20 calculates control data 22 based on the sensor data 12 of the camera 18 and the target data 40, taking into account the environmental object announced by the vehicle user.

[0044] Fig. Figure 2 shows an automated locomotion control in a further special embodiment of the invention. The automated locomotion control 10 compensates for the Fig.1 except for the following differences. The user information 38 includes stored route information 48 of a commute, for example, a route between the vehicle user's residence and workplace. The stored route information 48 can include details of recorded hazards, road closures, or the like. The stored route information 48 is, for example, stored in a database 50 that is continuously expanded as the commute is traveled and is stored in a memory 52 accessible for processing by the user interface model 36, for example, in the vehicle or as cloud storage outside the vehicle.

[0045] The stored route information 48 is assigned position information 54, which is provided, for example, by a position sensor 56 of the vehicle, in order to be able to assign the route information 48 assigned to the position information 54 to a specific location and to take the route information 48 into account depending on the position of the vehicle or the planned route of the vehicle.

[0046] The user interface model 36 calculates the target data 40, for example for the perception module 28, the prediction module 30 and the trajectory planning module 32, from the user information 38 formed by the stored route information 48 and the position information 54. The calculation of the control data 22 for controlling at least one vehicle function 24 that influences the movement of the vehicle by the data processing model 20 is thus dependent on the sensor data 12 as input data and the target data 40.

[0047] For example, the stored route information 48 can indicate an existing construction site on the commuter route. The user interface model 36 can use this information to calculate target data 40, for example for the trajectory planning module 32, and calculate the control data 22 for controlling at least one vehicle function 24 that influences the vehicle's movement, depending on this, for example to react to the construction site in a timely manner during automated vehicle movement.

Claims

[1] Automated movement control (10) of a vehicle, comprising the steps Input of sensor data (12) from at least one vehicle sensor (14) relating to the vehicle environment to at least one single module (26) comprising a data processing model (20) with at least one trained artificial neural network as input data and calculation of control data (22) depending on the input data by the data processing model (20), Input of user information (38) of a vehicle user of the vehicle to a user interface model (36) having at least one trained artificial neural network and calculation of target data (40) depending on the user information (38) by the user interface model (36), Providing the target data (40) to at least one individual module (26) of the data processing model (20) and Calculation of the control data (22) for controlling at least one vehicle function (24) that influences the movement of the vehicle by the data processing model (20) depending on the input data and the target data (40). [2] Automated movement control (10) according to claim 1, characterized by , that the user information (38) includes at least stored route information (48) of a commuter route. [3] Automated movement control (10) according to claim 2, characterized by , that the stored route information (48) is assigned position information (54). [4] Automated movement control (10) according to any one of the preceding claims, characterized by , that the user information (38) is formed at least from real-time communication data (42) of the vehicle user communicated to the user interface model (36) in real time. [5] Automated movement control (10) according to claim 4, characterized by , that the real-time communication data (42) are formed from text inputs (44) and / or voice inputs (46) from the vehicle user. [6] Automated movement control (10) according to any one of the preceding claims, characterized by , that the data processing model (20) has several individual modules (26) namely a perception module (28), a prediction module (30), a trajectory planning module (32) and a control module (34). [7] Automated movement control (10) according to claim 6, characterized by , that providing the target data (40) includes an input to the perception module (28), the prediction module (30) and / or the trajectory planning module (32). [8] Automated movement control (10) according to claim 6 or 7, characterized by , that the data processing model (20) including all individual modules (26) was trained as a whole. [9] Automated movement control (10) according to any one of the preceding claims, characterized by , that the target data (40) are data in a latent feature space. [10] Automated movement control (10) according to any one of the preceding claims, characterized by , that the user interface model (36) is a pre-trained multimodal model.

Images