Method for at least semi-automatically controlling motor vehicle and motor vehicle

Abstract

In order to at least semi-automatically control the motor vehicle (1), a second large language model is connected to the artificial intelligence unit (first large language model) used. In this way, the user can actively change motor vehicle control through the second large language model when asking for the user. It is proposed that the artificial intelligence unit (the first large language model) is particularly not retrained.

Description

Technical Field

[0001] The present invention relates to methods for at least semi-automatic, preferably fully automatic, control of motor vehicles according to two different aspects of the present invention, and the corresponding motor vehicles. Background Technology

[0002] The use of artificial intelligence units to control motor vehicles is known. In this regard, the application of "Large Language Models" is becoming increasingly important. The term "Large Language Model" refers to an AI-based program capable of recognizing and generating text, preferably using so-called deep learning capabilities (especially trained with training data). This large language model is used in vehicle control to execute control commands input via voice and to provide feedback textual explanations of the current control method. The model can also use an environmental map provided by sensor data as input and output the trajectory to be traveled, thereby generating control commands for lower-level control units of the vehicle's individual actuators or actuator groups.

[0003] Therefore, occupants of vehicles with at least semi-automatic control, and preferably fully automatic control, may expect corrective intervention. This is discussed in the paper "Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles" by Can Cui et al., *Journal of Latex Class Films*, Vol. 14, No. 8, August 2015, available online at https: / / arxiv.org / pdf / 2310.08034.pdf on October 12, 2023. According to this paper, occupants can use language models to issue commands such as "Drive more aggressively!" or conversely, "Drive more conservatively!" to the vehicle controlled by the AI ​​unit to change their driving behavior. This involves adjusting the control method in real time based on voice input. The goal is for the AI ​​unit to learn user preferences over a long period, thereby building user profiles and executing driving accordingly, thus eliminating the need for constant re-input by the user.

[0004] The paper "Drive Like a Human: Rethinking Autonomous Driving with Large Language Models" by Daocheng Fu et al. (preprint, available online July 14, 2023 at https: / / arxiv.org / pdf / 2307.07162.pdf) points out that language models can be used to evaluate how well an AI unit describes a system. To improve model performance, experts can provide suggestions on how a human driver would respond to specific scenarios. The model can then learn over a long period based on these suggestions.

[0005] The current approach is based on the premise that the AI ​​units may not be trained well enough, and that this can be improved through training.

[0006] However, AI unit training cannot always take into account all scenarios. Summary of the Invention

[0007] Therefore, the object of the present invention is to improve the method of controlling a motor vehicle at least semi-automatically and to provide a corresponding motor vehicle, enabling the user to give personalized instructions, thereby enabling the system for driving the motor vehicle to respond better to the situation.

[0008] This objective is achieved, on the one hand, by a method having the features of claim 1 and a motor vehicle having the features of claim 5, and on the other hand, by a method having the features of claim 9 and a motor vehicle having the features of claim 10.

[0009] According to the first aspect, a method for controlling a motor vehicle at least semi-automatically includes the following steps:

[0010] - Provide the first major language model in motor vehicles and provide a second major language model that is also preferred in motor vehicles;

[0011] -Use the first major language model to control motor vehicles;

[0012] - The first speech input is received by the second language model; and

[0013] - At least during the follow-up questioning process, the second language model issues the follow-up question and receives input for that follow-up question, preferably another voice input;

[0014] -After the final questioning process, the second language model transmits the output instruction to the first language model;

[0015] - The motor vehicle is controlled by the first language model taking into account the output command.

[0016] This invention proposes a task partitioning scheme in which a second language model allows users to perform voice input without directly using the first language model. By posing follow-up questions, the second language model can more accurately understand the user's needs, while also considering the actual execution capabilities of the first language model (and, depending on the situation, the feedback from the initial inquiry or the preprocessing of sensor data), thereby optimizing the final output instructions transmitted to the first language model.

[0017] This is not merely about dividing tasks, but about providing new operational options by probing the solutions, ensuring the necessary flexibility during execution.

[0018] According to a preferred embodiment, (in the next, or if final, step) the following input, preferably manually, is received: This input confirms that the control of the motor vehicle via the first language model, according to the inputter's view, is sufficiently consistent with the previous input. In other words, a confirmation button (or a "button" on a user interface such as a touchscreen) can be pressed here. In this way, the first language model is informed that the initiated motor vehicle control can continue without the need to use a second language model.

[0019] This button or key can also allow confirmation to be input to the second language model during the follow-up process (especially at the end of the final follow-up process), and then the second language model transmits the output instruction to the first language model.

[0020] Alternatively, the following input, preferably manual (which can also be input via buttons or keys on the user interface), can be received: this input cancels the previous input (including the first voice input and subsequent input in response to follow-up questions) and prompts a return to the control mode implemented by the first language model before the first voice input was received. (For this purpose, the first language model can always simultaneously calculate two alternative trajectories: one that conforms to its "normal" execution and the other that conforms to the input previously executed by the user; thereby enabling rapid switching between trajectories or at least simplifying the switching process).

[0021] In another preferred embodiment of the invention, automatic monitoring is performed based on sensor data from the vehicle's sensors to determine whether the execution of output commands by the first language model adequately meets expectations (via the first voice input and subsequent input in response to follow-up questions). While monitoring is conducted via a "Policy Supervisor," corrective commands can be transmitted to the first language model as needed (through this unit) to better meet user expectations. Alternatively or additionally, it may be proposed to monitor whether the output commands executed by the first language model conform to other conditions, such as prescribed rules: for example, if the user expects the vehicle to travel at 60 km / h, but the vehicle is in a 30 km / h speed limit area, correction should be made through automatic monitoring, and a command (preferably in the form of a voice command) should be issued to the first language model to correct this. Similarly, it may also be checked whether the planned trajectory conflicts with other objectives that may not have been considered in the user's command. In this case, braking measures should be taken, and an alternative trajectory that is as close as possible to the user's expected trajectory should be planned.

[0022] According to a first aspect of the invention, a motor vehicle has a control unit for performing at least semi-autonomous control of the motor vehicle, wherein the control unit includes a first large language model designed to receive voice commands and issue control commands dependent on those commands to other units of the motor vehicle. The motor vehicle also has a first interface capable of receiving voice input and a second interface connected to the control unit, wherein these interfaces are connected to a second large language model provided with the motor vehicle or are capable of connecting to a second large language model external to the motor vehicle, such that voice input received through the first interface can be converted into commands transmitted to the control unit through the second interface.

[0023] As described in the method according to the invention, the vehicle according to this aspect allows for the insertion of a second language model, thereby enabling a more flexible input scheme. In a variant where the second language model is located outside the vehicle, the connection can be wireless (e.g., connected to an external server via the Internet or also to a so-called edge node, a locally deployed unit for external vehicle data processing).

[0024] However, in a preferred embodiment of the motor vehicle, the second language model is an integral part of the motor vehicle and is designed to issue a question through the first interface and receive input (preferably another voice input) through the first interface in at least one questioning process, and then (before transmitting the instruction to the control unit through the second interface) form an instruction to be sent to the control unit.

[0025] In this respect, the motor vehicle specifically performs the method according to the invention as described in the first aspect.

[0026] According to another preferred embodiment, the motor vehicle has a manual input device for: performing input confirmation during a follow-up process, and / or for confirming that the control of the motor vehicle via the first language model is sufficiently consistent with the inputter's view according to the inputter's view, and / or for canceling the previous input to return to the control mode implemented via the first language model before the first language input was received.

[0027] According to another preferred embodiment, the motor vehicle has a monitoring unit (as described above, a "strategy monitor") to which sensor data of the motor vehicle can be transmitted. This monitoring unit is designed to check whether the instructions transmitted to the control unit can be executed (based on traffic rules such as speed limits and overtaking prohibitions, or taking into account other objects on the road), and is preferably also designed to transmit corrective instructions to the first language model. These corrective instructions may even include completely ignoring user commands.

[0028] According to a second aspect of the invention, a method for at least semi-automatically controlling a motor vehicle using a first large language model is provided, wherein the first large language model is the result of training using training data. This second aspect is preferably associated with the first aspect, and the method according to the second aspect is preferably also designed to be based on the method of the first aspect. In the method according to the second aspect, the motor vehicle is controlled using the first large language model, and input of a transmitted command is received, which requests a change in the current control mode based on the first large language model, wherein the command is executed. According to the invention, after a period of time (within predetermined time conditions and / or based on the received input) and / or after the end of the driving situation and / or based on user input, the change made according to the command is revoked, while the training data used within the scope of the method steps (“in this case”) remains unchanged.

[0029] Therefore, according to the second aspect of the invention, the learning process is not continued within the first major language model. This—preferably in conjunction with the insertion of a second major language model—has the advantage that not every potentially unreasonable user input will cause the finely trained results of the first major language model to be at least partially discarded or altered. In this way, the first language model can be reliably and persistently run in a motor vehicle and can be run in a way that is always up-to-date from the outside, but its application in the motor vehicle will not have a permanent impact on the training results.

[0030] According to the second aspect, the corresponding motor vehicle is preferably designed in the same way as the motor vehicle according to the first aspect, having a control unit for performing at least semi-automatic control of the motor vehicle, wherein the control unit has a first large language model, which is designed as a training result of training data to receive (preferably in language form) instructions that cause a change in the current driving control method via the (first) large language model and execute the instructions, while, according to the invention, although the instructions are executed, all the training data of the (first) large language model ("at the same time", see above) remain unchanged.

[0031] For application scenarios or situations that may occur in this method but are not explicitly described, it may be stipulated that, according to this method, error reports and / or invitations to input user feedback may be output, and / or default settings or preset initial states may be set.

[0032] The present invention also includes a control device for a motor vehicle, which may have a data processing device or a processor device designed to perform embodiments of the method according to the invention. The processor device may therefore have at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (Field Programmable Gate Array) and / or at least one DSP (Digital Signal Processor). In particular, a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or NPU (Neural Processing Unit) can be used as the microprocessor. Furthermore, the processor device may include program code configured to execute embodiments of the method according to the invention when run by the processor device. The program code may be stored in the data memory of the processor device. The processor device may, for example, be based on at least one circuit board and / or at least one SoC (System-on-Chip).

[0033] The motor vehicle according to the present invention is preferably an automobile, especially a passenger car or commercial vehicle, or a bus or motorcycle.

[0034] As another embodiment, the invention also includes a computer-readable storage medium comprising program code that, when run by a computer or computer network, causes it to perform an embodiment of the method of the invention. The storage medium may be provided at least partially as non-volatile data storage (e.g., flash memory and / or SSD – solid-state drive) and / or at least partially as volatile data storage (e.g., RAM – random access memory). The storage medium may reside in a computer or computer network. However, the storage medium may also operate, for example, as a so-called internet application store server and / or cloud server. A processor circuit having, for example, at least one microprocessor may be provided via a computer or computer network. The program code may be provided in the form of binary code and / or assembly code and / or source code in a programming language (e.g., C) and / or program scripts (e.g., Python).

[0035] The present invention also includes combinations of features of the described embodiments. Therefore, the present invention also includes implementations having combinations of features of multiple embodiments described in the embodiments, provided that the multiple embodiments are not described as mutually exclusive. Attached Figure Description

[0036] Embodiments of the present invention are described below. Wherein:

[0037] Figure 1 The schematic diagram illustrates components used for controlling a motor vehicle in the method of the present invention;

[0038] Figure 2 The flowchart of steps in the first embodiment of the method of the present invention at the first response of the artificial intelligence unit is shown;

[0039] Figure 3 The flowchart of the steps in the secondary response of the artificial intelligence unit in the second embodiment of the method of the present invention is shown. Detailed Implementation

[0040] The embodiments described below are preferred embodiments of the present invention. In these embodiments, the components described in the embodiments represent individual, independently viewable features of the invention, which also independently improve the invention. Therefore, this disclosure should also cover combinations of features other than those shown in the embodiments. Furthermore, the embodiments can be supplemented by other features of the invention already described.

[0041] In the accompanying drawings, the same reference numerals denote elements that have the same function.

[0042] like Figure 1As shown, the motor vehicle, generally labeled 1, includes a trained artificial intelligence unit as the actual control unit 10, which may include, for example, one or more neural networks. This can be referred to as an AI unit for "Learned Black-Box Driving" (LBBD), which uses the environment of the motor vehicle 1 presented by sensors (represented by sensors S1 and S2 in the figure) and executes commands in the form of voice input. It can output a description of the current control mode in text form. The control unit 10 can simultaneously calculate two driving trajectories: one is the currently driven trajectory that conforms to the voice input, and the other is the trajectory that would be automatically selected if there is no voice input. In this case, the control unit is implemented by a first large language model, hereinafter referred to as 10; other components of the control unit are designed in a manner known in the art. Furthermore, a user interface (UI) is provided, through which the user can perform voice input by providing a voice recognition system. Preferably, a haptic interface is also provided for confirming the voice input after follow-up questioning (as described below) and, if necessary, returning to the normal mode.

[0043] It is worth noting that the user interface (UI) is not directly connected to the first language model 10 as currently used. Instead, a second language model 12 is inserted in the middle. This second language model is connected to the user interface (UI) through a first interface (I1, "interface 1") and to the first language model 10 through a second interface (I2, "interface 2"). The second language model 12 is indicated by dashed lines because it does not necessarily have to be part of the vehicle 1. Instead, it can be located outside the vehicle 1 and perform external data processing, provided that interfaces I1 and I2 are designed to also support wireless communication.

[0044] In addition, there is a so-called "Policy Monitor" (PS), which is a monitoring unit that ensures that commands entered through the user interface (UI) are actually executed and comply with rules (such as speed limits, prohibition of overtaking, etc.).

[0045] Figure 2 The sequence of steps of one embodiment of the method of the present invention is shown, wherein a first large language model responds in a more or less desirable manner.

[0046] In step S10, the user performs voice input, for example, "Keep a greater distance from trucks!" In step S12, the language model receives this input and in step S14 issues a follow-up question: "All trucks, or only trucks in adjacent lanes? Is it okay to increase the distance by 10%?"

[0047] In step S16, the user answers via the user interface (UI): "Yes, only trucks in adjacent lanes. 10% is possible."

[0048] During this period, according to step S22, the first language model 10 receives the instruction (see instruction W column) as before: "Drive in normal mode!", and then executes the instruction according to step S22. This process continues until the user completes the final instruction confirmation through the user interface (UI). This confirmation may have already been completed in step S16, in which case the command can be executed immediately. Figure 2 A variant is shown where, according to step S18, the second language model 12 receives the instruction "Maintain a 10% distance from the truck in the adjacent lane!", at which point, in step S24, it asks, "Should the instruction be executed immediately?", while still maintaining normal driving mode according to command S20 and step S22. The user can then simply say "Yes!" during voice input. However, in this preferred variant, the confirmation is completed by the user pressing a button in step S26 or by pressing a key on a display such as a touchscreen. In response to this operation (via voice input or... Figure 2 As shown in step S26 (confirmed by pressing a button), in step S28, the second language model 12 prompts the first language model to issue an actual instruction: "Maintain a 10% distance from the truck in the adjacent lane!" Then, in step S30, the first large language model executes the command; however, due to reasons present during training, it is possible that an additional 50% (instead of 10%) of the current distance is actually maintained. In other words, the instruction W according to step S28 is not perfectly executed. However, the policy monitoring of the policy supervisor PS in steps S22 and S30 can be considered to have fully taken into account the input command, as shown by the checkmark in the figure. After a period of time or in response to maintaining an excessive distance according to step S30, the user can input in step S32 to return to the normal mode, where the command can be sent directly to the first large language model 10, i.e., without going through the second large language model 12. This directness can preferably be achieved by using an alternative input method instead of voice input, such as a button or a key on a touchscreen as shown. In response, step S20' is executed, i.e., the instruction to drive in the normal mode is issued again, and the vehicle is driven in the normal mode in step S22'. This is then also confirmed by the policy supervisor PS.

[0049] The situation at this point might be that the AI ​​unit hasn't been trained well enough to execute such commands. According to... Figure 3To elaborate: If the steps S10, S12, S14, S16, S18, S24, and S26 follow the same flow as above, then according to the instruction given in step S28: "Maintain a 10% distance from the truck in the adjacent lane!", in step S30, the first large language model 10 might now want to maintain a distance of more than 100% from the truck in the adjacent lane (doubling the distance instead of increasing it by 10%), resulting in a deviation. However, at this point, the strategy monitor PS will intervene and issue an instruction in step S34, indicating that the lane width may be exceeded, and therefore it is no longer possible to drive in the adjacent lane. Therefore, the strategy monitor PS then prompts a return to control in the normal mode, thus... Figure 2 The steps S20' and S22' are initiated in the same way as the return based on the command in step S32.

[0050] This invention introduces a second language model 12 between the user interface and the first language model 10, enabling the user to input commands that the first language model may not currently execute perfectly. The second language model 12 ensures that such commands are expressed as accurately as possible by prompting the user, i.e., translated into instruction W, so that the first language model 10 can ultimately generally execute the command. It is particularly reasonable to use a policy monitor PS to revert to normal control if the first language model 10 ultimately fails to execute the command correctly.

[0051] Overall, these embodiments demonstrate how to implement strategy adjustments in at least semi-automatic control (preferably fully automatic control).

Claims

1. A method for at least semi-automatically controlling a motor vehicle (1), comprising the steps of: - Provide a first major language model (10) in the motor vehicle (1), and preferably also provide a second major language model (12) in the motor vehicle (1); -Use the first major language model (10) to control motor vehicles (1); - The first speech input is received by the second language model (12); -At least during the follow-up questioning process, the second language model (12) issues the follow-up question and receives input for the follow-up question, preferably another speech input; -After the final questioning process, the second language model (12) transmits the output instruction to the first language model (10); - The motor vehicle (1) is controlled by the first language model (10) taking into account the output instructions.

2. The method according to claim 1, characterized in that, The input received is preferably manual and is as follows: This input confirms that the control of the motor vehicle (1) by the first language model (10) is in full accordance with the inputter's opinion and the previous input.

3. The method according to claim 1, characterized in that, The system receives the following input, preferably manually: this input cancels the previous input and returns to the control mode implemented by the first language model before receiving the first voice input.

4. The method according to any one of the preceding claims, characterized in that, Automatic monitoring based on sensor data from the sensors of the motor vehicle (1): whether the execution of the output instructions by the first language model fully meets specific conditions, preferably whether it fully matches the input of the inputter and / or matches the actual situation.

5. A motor vehicle (1) having a control unit for performing at least semi-automatic control of the motor vehicle (1), wherein, The control unit includes: a first large language model (10) designed to receive instructions in the form of speech and issue control commands related to the instructions to other units of the vehicle (1); the vehicle (1) also has a first interface (I1) capable of receiving voice input and a second interface (I2) connected to the control unit (10); these interfaces (I1, I2) are connected to the second large language model (12) of the vehicle (1) or can be connected to a second large language model outside the vehicle (1), so that the voice input received by the first interface (I1) can be converted into instructions transmitted to the control unit by the second interface (I2).

6. The motor vehicle (1) according to claim 5, characterized in that, The second language model (I2) is part of the vehicle (1) and is designed to issue a question through the first interface (I1) and receive input through the first interface (I1), preferably another voice input, in at least one questioning process, and then transmit instructions to the control unit through the second interface (I2).

7. The motor vehicle (1) according to claim 5 or 6, characterized in that, The motor vehicle has a manual input device for making a confirmation or negation input during the questioning process, and / or for confirming that the control of the motor vehicle (1) by the first language model (10) according to the inputter's opinion is fully consistent with the previous input, and / or for canceling the previous input in order to return to the control mode implemented by the first language model before receiving the first voice input.

8. The motor vehicle (1) according to any one of claims 5 to 7, characterized in that, The vehicle has a monitoring unit (PS) to which sensor data of the vehicle (1) can be transmitted. The monitoring unit is designed to check whether the instructions transmitted to the control unit can be executed. The monitoring unit is also preferably designed to transmit correction instructions to the first language model (10).

9. A method for at least semi-automatically controlling a motor vehicle (1) using a first large language model (10), the first large language model being obtained by training with training data, the method preferably being designed as claimed in any one of claims 1 to 4, wherein, In this method, a first major language model is used to control the motor vehicle (1), and an input is received, the input transmitting an instruction requesting a change in the current control mode executed according to the first major language model, wherein the instruction is executed. Its features are, After a period of time and / or after the driving situation ends and / or based on user input, changes made according to instructions are reversed, and the overall training data used remains unchanged.

10. A motor vehicle (1), particularly a motor vehicle according to any one of claims 5 to 8, the motor vehicle having a control unit for performing at least semi-automatic control of the motor vehicle (1), wherein, The control unit has a first major language model (10), which is obtained by training with training data and is designed to receive instructions in a preferred language form that require changing the current control mode performed according to the first major language model, and to execute the instructions. Its features are, Despite the existence and execution of the instruction, the training data of the first large language model remains unchanged overall.

Images