Method for operating a (partially) autonomously operating control unit of a vehicle, vehicle control system and vehicle

Abstract

The invention relates to a method (1) for operating an autonomously or partially autonomously operating control unit (190) of a vehicle (2). The method (1) comprises the following steps: assigning (S1) permission for control of the vehicle (2) to a co-driver (400) of the vehicle (2); detecting (S2) a voice input of the co-driver (400); identifying (S3) a driving manoeuvre command on the basis of the voice input of the co-driver (400); generating and outputting (S4) an audio signal corresponding to the identified driving manoeuvre command; detecting (S5) a signal output; determining (S6), on the basis of the detected signal output, whether the audio signal has been securely output; determining (S7) whether confirmation has been provided by the co-driver (400); and if it has been determined that the audio signal was securely output and confirmation has been provided by the co-driver (400), forwarding (S8) the driving manoeuvre command to the autonomously or partially autonomously operating control unit (190) of the vehicle (2). The invention further relates to a corresponding vehicle control system (3), vehicle (2) and storage medium.

Description

[0001] Description

[0002] METHOD FOR OPERATING A (PARTIALLY) AUTONOMOUSLY OPERATING CONTROL UNIT OF A VEHICLE, VEHICLE CONTROL SYSTEM AND VEHICLE

[0003] Field of invention

[0004] The invention relates to autonomous or semi-autonomous driving of a vehicle. In particular, the invention relates to a method for operating an autonomously or semi-autonomously operating control unit of a vehicle. Furthermore, the invention relates to a vehicle control system and a vehicle with such a vehicle control system.

[0005] background

[0006] Voice control systems in vehicles allow a driver to control a variety of vehicle functions. To authenticate the driver, there are numerous methods available that allow for personnel verification of safety-critical operations or instructions.

[0007] These user control options are reserved for the driver. However, if the driver is tired, distracted, or wishes to focus their attention elsewhere, the passenger should also be able to control driving instructions via voice input. Driving safety must still be guaranteed in this regard.

[0008] Summary

[0009] It is therefore an object of the invention to provide a method for a passenger to operate an autonomously or semi-autonomously functioning control unit of a vehicle, which in particular ensures driving safety. Furthermore, it is an object of the invention to provide a corresponding vehicle control system and a vehicle with such a vehicle control system.

[0010] One aspect of the invention relates to a method for operating an autonomously or semi-autonomously functioning control unit of a vehicle. This method can be executed by a vehicle control system.

[0011] The vehicle can be a road-bound vehicle, such as a passenger car, a truck, or a special-purpose vehicle. The vehicle can also be, for example, a boat or an aircraft. In the latter two cases, the words "driver" and "passenger" should be replaced by "captain" and "staff captain" or "pilot" and "co-pilot," respectively.

[0012] The autonomously or semi-autonomously operating control unit of the vehicle can in particular be understood as a control unit within the meaning of the SAE standard J3016, whereby "autonomously or semi-autonomously" specifically refers to levels 3, 4 and 5.

[0013] According to the procedure, authorization to control the vehicle is granted to a passenger. A passenger is defined here as a person who is in the vehicle and, due to their position within the vehicle, is able to control it by voice commands. In a passenger car, this can be determined, for example, by the fact that the person is sitting in the front passenger seat.

[0014] Furthermore, voice input from the passenger is recorded. Passenger recognition can be achieved, for example, via a secure driver and passenger authentication unit, which may be based on iris recognition using an interior camera and / or continuous biometric voice recognition. The passenger's voice input can be captured contextually from an ongoing conversation or triggered by a detection signal, such as the voice input of a code word (e.g., "Hello, car") or the activation of a corresponding voice input button.

[0015] Based on the passenger's voice input, a driving maneuver command is then recognized. This recognition of the driving maneuver command can be achieved, for example, via speech recognition, syntax analysis, and / or natural language processing, preferably in conjunction with a list of possible driving behavior instructions.

[0016] When a driving maneuver command is recognized, an audio signal corresponding to that command is generated and output. This output is primarily via loudspeakers, particularly those positioned near the passenger's head, for example, in or on the passenger seat's headrest. The audio signal corresponding to the recognized driving maneuver command can be retrieved from a database where each possible driving maneuver command is assigned a corresponding audio signal. Alternatively, the database can contain text modules for the corresponding audio signal, and / or the audio signal can be generated using speech synthesis. The audio signal can include, in particular, a brief description of the driving maneuver command, confirmation that the command has been recognized, and a request for acknowledgment.

[0017] While the audio signal corresponding to the recognized driving maneuver command is being emitted, a signal output is recorded, for example, by microphones. Based on this recorded signal output, it is then determined whether the audio signal was reliably emitted. For example, it can be checked whether the audio signal corresponding to the driving maneuver command is contained within the recorded signal output. If the audio signal corresponding to the driving maneuver command is drowned out by strong ambient noise, such as road noise or conversations, it can be determined that the audio signal was not reliably emitted.

[0018] The system then determines whether the passenger has confirmed the command. Only when it has been established that the audio signal was successfully transmitted and that the passenger has confirmed the command is the driving maneuver command forwarded to the vehicle's autonomous or semi-autonomous control unit.

[0019] This ensures that the driving maneuver command reflects the passenger's intention and that no incorrect driving maneuver command is executed.

[0020] In some embodiments, granting permission to control the vehicle to the passenger involves detecting a signal from a control button operated by the driver of the vehicle for the transfer of control. This control button may be a dedicated button used solely for granting permission to control the vehicle to the passenger, or it may be part of an input system that allows the driver, via a menu, to grant or transfer control of the vehicle to the passenger.

[0021] Alternatively or additionally, the authorization to control the vehicle can be granted to the passenger via voice input. For this purpose, the driver's voice input is recorded. Based on this voice input, a control transfer command is recognized. The recognition of this control transfer command can be analogous to the recognition of the driving maneuver command described above.

[0022] A further audio signal corresponding to the detected control transfer command is then generated and output. This can also be done analogously to the generation and output of the audio signal described above. The output of the further audio signal can preferably be via loudspeakers located near the driver's head, for example in or on a driver's seat headrest, and / or via loudspeakers located near the passenger's head, for example in or on the passenger's seat headrest.

[0023] Furthermore, another signal output is detected, and based on this detected signal output, it is determined whether the additional audio signal was successfully output. This is done analogously to the signal output detection described above and the determination, based on the detected signal output, of whether the audio signal was successfully output.

[0024] It is then determined whether further confirmation has been given by the driver and / or the passenger, and only if it has been determined that the further audio signal was definitely given and that confirmation has been given by the driver and / or passenger, is the permission to control the vehicle assigned to the passenger.

[0025] The additional audio signal to the driver and the driver's confirmation serve to ensure that the driver wishes to transfer control of the vehicle to the passenger. The additional audio signal to the passenger and the passenger's confirmation serve to ensure that the passenger also agrees to take control of the vehicle.

[0026] In some embodiments, determining whether the passenger has given confirmation and / or determining whether the driver and / or passenger have given further confirmation involves detecting a further signal from a confirmation button operable by the passenger or driver. This confirmation button can be a dedicated button used solely for confirming the audio signals, or it can be part of an input system that allows the passenger or driver to confirm the audio signals via a menu. Alternatively or additionally, determining whether the passenger has given confirmation and / or determining whether the driver and / or passenger have given further confirmation is achieved through further voice input from the passenger or driver. In particular, this further voice input from the passenger or driver is detected and used to determine the appropriate response.The driver received a confirmation command.

[0027] In some embodiments, the method further includes relinquishing control of the vehicle from the passenger. This relinquishment of control can be carried out, in particular, by the driver and can be accomplished in various ways: for example, control can be relinquished by pressing the control button. Alternatively or additionally, control can be relinquished by a corresponding voice command. Again, alternatively or additionally, control can be relinquished by a manual vehicle operation, for example, when the driver operates the steering wheel and / or accelerator and / or brake pedals. In this way, the driver can regain control of the vehicle at any time.

[0028] In some embodiments, capturing the passenger's voice input, capturing the signal output, capturing the driver's voice input, capturing further signal output, and / or capturing further voice input from the passenger or driver, involves capturing sound signals using a microphone array and evaluating the sound signals using a sound processing unit. The microphone array can comprise several microphones, which are positioned in the vehicle, in particular, to optimally capture the driver's voice, driver's speaker, passenger's voice, and / or passenger's speaker. The captured sound signals are then processed in the sound processing unit. The sound processing unit can be configured, in particular, to separate individual sound signals from one another. For this purpose, the signals from the individual microphones of the microphone array can be combined (e.g.,The signals are delayed and / or filtered to amplify the driver's voice, the output from the driver's speakers, the passenger's voice and / or the output from the passenger's speakers, and to attenuate other sound sources inside and outside the vehicle. The processed signal is then passed on accordingly.

[0029] In some embodiments, the audio signal corresponding to the recognized driving maneuver command and / or the further audio signal corresponding to the recognized control transfer command is generated with a characteristic modulation. Determining whether the audio signal was reliably output and / or whether the further audio signal was reliably output is then based on this characteristic modulation. The characteristic modulation can, in particular, be imperceptible to humans, i.e., be in a frequency range that is not perceptible or disturbing to the human ear at the moment of output. More specifically, the characteristic modulation can be in the near-ultrasound range, for example, in a frequency range between 25 and 30 kHz. The characteristic modulation can, for example, correspond to a predefined or randomly generated signal sequence that corresponds to the audible audio signal or the further audio signal.An additional audio signal is added. If the characteristic modulation is then detected in the signal output or the subsequent signal output, it can be concluded that the audio signal or the subsequent audio signal was successfully output. It should be noted that if the characteristic modulation occurs in a frequency range inaudible to the human ear, the loudspeakers and microphones must be designed for playback or recording in this frequency range.

[0030] In some embodiments, the method further includes generating and outputting a graphic signal corresponding to the recognized driving maneuver command. This signal can, for example, be output to a visual display unit visible to the passenger. The graphic signal can, for example, include the driving maneuver command in written form in abbreviated form and / or a graphic animation of the driving maneuver command, for example, incorporating environmental information.

[0031] In some embodiments, the system also determines whether the passenger is attentive. The driving maneuver command is then only transmitted to the autonomous or semi-autonomous control unit if it has been determined that the passenger is attentive. This further enhances driving safety. Attentiveness can be determined, for example, by analyzing microphone signals, such as whether the voice sounds alert or sleepy. Alternatively or additionally, data from an interior camera can be used to determine attentiveness, analyzing, for example, the passenger's position and facial features. Alternatively or additionally, sensors (e.g., touch or proximity sensors) in the passenger headrest can provide information about the passenger's head position and thus their level of attentiveness.

[0032] In some embodiments, the method further includes an evaluation of the driving maneuver command. This evaluation can be performed, in particular, by the vehicle's autonomous or semi-autonomous control unit. The driving maneuver command can be assessed for feasibility, for example, using image signals from an external camera and / or signals from other sensors such as radar or lidar.

[0033] A further audio signal corresponding to the assessment of the driving maneuver command is then generated and output. Additionally, a graphic signal corresponding to the assessment of the driving maneuver command can also be output. In particular, the further audio signal can include information on whether the driving maneuver command is feasible or not, in order to prepare the driver and / or passenger for whether the driving maneuver command will be carried out or not.

[0034] To verify the successful output of the additional audio signal, a further signal output is detected, and based on this detection, it is determined whether the additional audio signal was successfully output. Details are analogous to the description above. If the maneuver command is deemed feasible and it has been determined that the additional audio signal was successfully output, the maneuver command is executed.

[0035] Another aspect of the invention relates to a vehicle control system. This vehicle control system can be a dedicated system for controlling an autonomously or semi-autonomously operating control unit of a vehicle, but it can also be part of a vehicle system. The vehicle control system comprises a processing unit, an interface to at least one loudspeaker, and an interface to a microphone array, and is configured to carry out the method according to the preceding description. Advantages, further embodiments, and details will become apparent from the preceding description.

[0036] A further aspect of the invention relates to a vehicle comprising a vehicle control system as described above. The vehicle can be a road-bound vehicle, such as a passenger car, a truck, or a special-purpose vehicle. The vehicle can also be, for example, a boat or an aircraft. Advantages, further embodiments, and details will become apparent from the preceding description.

[0037] A further aspect of the invention relates to a non-volatile, computer-readable storage medium containing a program which, when executed on a computing unit of a vehicle control system as described above, instructs the vehicle control system to execute the method described above. Advantages, further embodiments, and details will become apparent from the preceding description.

[0038] The above-described method, vehicle control system, vehicle and storage medium solve, among other things, the following problems with safety-critical or safety-relevant voice input to the autonomously or semi-autonomously operating control unit of the vehicle:

[0039] 1) The driver must be able to allow the passenger to enter verbal driving instructions or to withdraw them at any time.

[0040] 2) The passenger must clearly and unambiguously confirm permission to input driving instructions.

[0041] 3) The driver must be able to withdraw the passenger's permission to give driving instructions at any time by means of a manual driving operation and / or by voice command.

[0042] 4) The passenger's driving instruction must be clearly identifiable and must not be confused with any other possible driving instruction.

[0043] 5) The vehicle system must ensure that a driving instruction is correctly assigned to the vehicle occupant who gave it (passenger or driver).

[0044] 6) The vehicle system must ensure that a driving instruction from a vehicle occupant is only taken into account if the occupant has been previously authorized to do so.

[0045] 7) If the passenger and driver give verbal driving instructions simultaneously, both driving instructions should be recognized and evaluated.

[0046] 8) If oral driving instructions are given simultaneously, only one may be accepted as valid by the vehicle system and issued for confirmation.

[0047] 9) Confirmation of the driving instruction may only be accepted by the instructing speaker.

[0048] 10) Confirmation can be given manually or optionally by secure speech recognition of a keyword or phrase from the instructor.

[0049] 11) The vehicle's semi-autonomous or autonomous control unit should ensure that the authorized co-driver can visually assess the current driving situation before and during their voice input. 12) Optionally, it may be necessary to additionally check the co-driver's attention before executing a driving instruction, e.g., whether they are tired, distracted, or under the influence of alcohol.

[0050] 13) The acoustic environment (e.g. taking into account background noise) must allow clear and unambiguous communication between the driver and passenger and the vehicle's voice recognition system.

[0051] 14) Taking into account all information and potential risks known to the autonomous vehicle control system, a clearly identified and confirmed driving instruction must be assessed by the autonomous vehicle control system for its safe feasibility and rejected if it is not safe.

[0052] 15) In the case of driving instructions from the passenger, the reduced or non-existent manual control of the vehicle by the passenger must be taken into account, which may limit the possible driving instructions for the passenger.

[0053] 16) If the driving instruction can be carried out safely, it must be possible to execute it safely; otherwise, execution must be refused.

[0054] 17) Feedback must be given to the vehicle occupants regarding the feasibility or imfeasibility of the driving instructions.

[0055] For further clarification, the invention is described with reference to embodiments illustrated in the figures. These embodiments are to be understood as examples only, and not as limitations.

[0056] Brief description of the characters

[0057] This shows:

[0058] Fig. 1 shows a flowchart for a procedure for operating a vehicle control unit;

[0059] Fig. 2 is a schematic representation of a vehicle with a vehicle control system; Fig. 3 is a detailed schematic representation of a vehicle control system; and

[0060] Fig. 4 shows a flowchart for a computer program.

[0061] Detailed description of embodiments

[0062] In the figures, identical reference symbols denote either identical elements or elements with equivalent functions. Elements that have already been described are not necessarily described again in subsequent figures.

[0063] Figure 1 shows a flowchart for a method 1 for operating an autonomously or semi-autonomously operating control unit of a vehicle.

[0064] The vehicle can be a road-bound vehicle, such as a passenger car, a truck, or a special-purpose vehicle. The vehicle can also be, for example, a boat or an aircraft. In the latter two cases, the words "driver" and "passenger" should be replaced by "captain" and "staff captain" or "pilot" and "co-pilot," respectively.

[0065] The autonomously or semi-autonomously operating control unit of the vehicle can in particular be understood as a control unit within the meaning of the SAE standard J3016, whereby "autonomously or semi-autonomously" specifically refers to levels 3, 4 and 5.

[0066] Procedure 1 involves assigning S1 authorization to control the vehicle to a passenger. A passenger is defined here as a person who is in the vehicle and, due to their position within the vehicle, is able to control it by voice commands. In a passenger car, this can be determined, for example, by the fact that the person is sitting in the front passenger seat. Assigning S1 authorization to control the vehicle to the passenger may involve detecting a signal from a control button, operated by the driver of the vehicle, to transfer control of the vehicle.

[0067] Alternatively, or additionally, assigning S1 permission to control the vehicle to the co-driver may include the following: detecting a voice input from the driver; recognizing a transfer of control command based on the driver's voice input; generating and outputting a further audio signal corresponding to the recognized transfer of control command; detecting a further signal output; determining, based on the detected further signal output, whether the further audio signal was safely output; determining whether further acknowledgment has been given by the driver and / or co-driver; and, if it has been determined that the further audio signal was safely output and acknowledgment has been given by the driver and / or co-driver, assigning permission to control the vehicle to the co-driver.

[0068] Furthermore, procedure 1 includes capturing S2 a voice input from the passenger. Passenger recognition can be achieved, for example, via a secure driver and passenger authentication unit, which may be based on iris recognition with an interior camera and / or continuously operating biometric voice recognition. The passenger's voice input can be captured contextually from an ongoing conversation or can be triggered by a detection signal, such as the voice input of a codeword (e.g., "Hello, car") or the activation of a corresponding voice input button.

[0069] Based on the passenger's voice input, a driving maneuver command is recognized (S3). This recognition (S3) can be achieved, for example, through speech recognition, syntax analysis, and / or natural language processing, preferably in conjunction with a list of possible driving instructions. Furthermore, the method includes generating and outputting (S4) an audio signal corresponding to the recognized driving maneuver command. The output is provided, in particular, via loudspeakers, especially those located near the passenger's head, for example, in or on a headrest of the passenger seat. The audio signal corresponding to the recognized driving maneuver command can be retrieved, for example, from a database in which a corresponding audio signal is assigned to each possible driving maneuver command.Alternatively, text modules for the corresponding audio signal can be stored in the database and / or the audio signal corresponding to the driving maneuver command can be generated via speech synthesis. The audio signal corresponding to the driving maneuver command can include, in particular, a brief description of the driving maneuver command, as well as a statement that this driving maneuver command has been recognized and a request for confirmation.

[0070] While the audio signal corresponding to the recognized driving maneuver command is being emitted, a signal output is detected (S5). Based on this detected signal output, a determination (S6) is then made as to whether the audio signal was reliably emitted. For example, it can be checked whether the audio signal corresponding to the driving maneuver command is contained within the detected signal output. If the audio signal corresponding to the driving maneuver command is drowned out by strong ambient noise, such as road noise or conversations, it can be determined that the audio signal was not reliably emitted.

[0071] For this purpose, the audio signal corresponding to the recognized driving maneuver command can be generated with a characteristic modulation, and the determination S6 of whether the audio signal was safely output can be carried out based on this characteristic modulation.

[0072] Furthermore, a graphical signal corresponding to the recognized driving maneuver command can be generated and output. The procedure also includes determining S7 whether confirmation has been given by the passenger.

[0073] Determining whether the passenger has given confirmation (S7) and / or determining whether the driver and / or passenger has given further confirmation may include detecting a further signal from a confirmation button operable by the passenger or driver.

[0074] Alternatively, or additionally, determining whether confirmation has been given by the passenger and / or determining whether further confirmation has been given by the driver and / or passenger may include: capturing further voice input from the passenger or driver; and recognizing a confirmation command based on the further voice input from the passenger or driver.

[0075] Once it has been determined that the audio signal has been safely output and confirmation has been received from the passenger, the driving maneuver command S8 is forwarded to the autonomously or semi-autonomously operating control unit of the vehicle.

[0076] This ensures that the driving maneuver command reflects the passenger's intention and that no incorrect driving maneuver command is executed.

[0077] When capturing the passenger's voice input, capturing the signal output, capturing the driver's voice input, capturing further signal output, and / or capturing further voice input from the passenger or driver, sound signals can be captured using a microphone field and evaluated using a sound evaluation unit.

[0078] In a further embodiment, not shown here, method 1 can additionally include taking control of the vehicle away from the passenger. In yet another embodiment, not shown here, method 1 can additionally include determining whether the passenger is attentive, whereby the transmission of the driving maneuver command to the autonomously or semi-autonomously operating control unit only occurs if it has been determined that the passenger is attentive.

[0079] In a further embodiment, not shown here, the method 1 may additionally comprise: evaluating the driving maneuver command; generating and outputting a further audio signal corresponding to the evaluation of the driving maneuver command; detecting a further signal output; determining, based on the detected further signal output, whether the further audio signal was reliably output; and, if the driving maneuver command was evaluated as feasible and it was determined that the further audio signal was reliably output, executing the driving maneuver command.

[0080] Figure 2 shows a schematic representation of a vehicle 2 with a vehicle control system 3. The vehicle control system 3 comprises a computing unit 4 and interfaces (not shown) to at least one loudspeaker 103, 403 and to a microphone array 5, comprising a plurality of microphones 303. The vehicle control system 3 is specifically designed to carry out the method 1.

[0081] Figure 3 shows a detailed schematic representation of a vehicle control system and parts of a vehicle interior.

[0082] A driver's seat 101 is suitable for a person, in particular a driver 100, who can control the vehicle from this seat using user controls, for example, lateral and longitudinal user controls such as a steering wheel 111 or foot pedals 112 for braking and accelerating. A passenger seat 401 is also suitable for a person, in particular a passenger 400, who can see a traffic situation from this passenger seat 401, for example, through a windshield.

[0083] Near the driver's seat 101, an easily accessible confirmation button 110 is located for the driver 100. This button allows the driver 100 to transfer vehicle control to the passenger 400 or reactivate vehicle control. The activation button 110 is located, for example, on the steering wheel 111. A signal B from the activation button 110 can be transmitted to a secure dialogue control unit 150.

[0084] The driver's seat 101 may also include a driver's headrest 102 which has a device that detects the driver's 100 sitting upright and / or leaning his head against the driver's headrest 102, for example by means of a switch or a proximity sensor.

[0085] The front passenger seat 401 can also include a front passenger headrest 402, which likewise has a device that detects whether the front passenger 400 is sitting upright and / or leaning their head against the front passenger headrest 402, for example via a switch or a proximity sensor. If the front passenger 400's head is leaning against the front passenger headrest 402, this can be transmitted as signal K to the safe dialogue control 150.

[0086] The above-described determination of the head's position against the headrest serves to recognize that the driver (100) or the passenger (400) has their head in a position in which they can observe the traffic situation.

[0087] Alternatively or additionally to head position detection by the headrests 102, 402, an interior camera 120 can be provided for monitoring the driver 100 and front passenger 400, wherein the data from the interior camera 120 are processed to determine the position of the head of the driver 100 or front passenger 400, and thus to recognize whether the driver 100 or the front passenger

[0088] 400 have their heads in a position where they can observe the traffic situation.

[0089] Furthermore, one or more driver loudspeakers 103 are arranged on or near the head of the driver 100, for example in the driver's headrest 102. Likewise, one or more passenger loudspeakers 403 are arranged on or near the head of the passenger 400, for example in the passenger's headrest 402.

[0090] Furthermore, a microphone array consisting of several microphones 303 is arranged in the vehicle. These microphones 303 are preferably arranged such that they can optimally capture the voice of the driver 100, output from the driver's loudspeaker 103, the voice of the passenger 400, and output from the passenger's loudspeaker 403.

[0091] The signals captured by the microphones 303 can be forwarded to a sound evaluation unit 302 and processed there. In particular, the signals from the individual microphones 303 can be combined with one another to form a combined signal Ma, for example by delaying or filtering them, so that the voice of the driver 100 and the output of the driver's loudspeaker 103 or the voice of the passenger 400 and the output of the passenger's loudspeaker 403 are amplified and other sound sources inside and outside the vehicle are attenuated.

[0092] The combined signal Ma can then be forwarded, for example, to a secure audio output 130, to a speech recognition 140, to an attention detection 160 and / or to a driver and passenger authentication unit 170.

[0093] The secure audio output 130 generates an audio signal Sa, which can be reliably identified, for example, by means of a characteristic modulation from the microphone signal Ma that is not perceptible to humans, so that it can be ensured that the output audio signal Sa is a speech reproduction of a digital input signal, e.g., a driving maneuver command Fa.

[0094] The speech recognition system 140 recognizes instructions from the driver or passenger in the combined microphone signal.

[0095] The safe dialogue control 150 concludes from the recognized voice instruction Ft of the driver or passenger, if a clear assignment is possible from a list Lf of possible driving behavior instructions specified by an autonomous vehicle control unit 190, that a specific driving maneuver command Fa.

[0096] The specific driving maneuver command Fa is passed on to the safe audio output 130, which uses a database to output a modulated audio signal Sa defined for the driving maneuver command Fa to the driver 100 and / or passenger 400 and reports the success of the audio output back to the safe dialogue control 150 via a signal E.

[0097] The driving maneuver command Fa is optionally transmitted to a visual output unit 180 visible to the passenger 400 and / or driver 100, which graphically animates the driving maneuver command Fa, taking into account the environmental information contained in the driving maneuver command (e.g. overtaking a truck in front).

[0098] The secure driver and passenger authentication unit 170 can, for example, be implemented based on iris recognition using the interior camera 120 and continuous biometric voice recognition. If authentication fails (e.g., of the driver's or passenger's voice), a signal Na can be forwarded to the secure dialogue control unit 150.

[0099] The attention detection system 160 can, for example, detect the attention of the driver or passenger based on the combined microphone signals Ma and

[0100] Interior camera image data Kd assess and, for example, if fatigue is detected, communicate this via a signal Sh to the safe dialogue control 150, which then prevents the output and execution of a driving behavior instruction from the driver or passenger.

[0101] If no fatigue is present, the safe dialogue control 150 can issue a confirmed driving maneuver command Af to the autonomously or semi-autonomously operating vehicle control unit 190. The autonomously or semi-autonomously operating vehicle control unit 190 then evaluates the confirmed driving maneuver command Af for feasibility, for example, using image signals Sk from an external camera 191 and / or signals S from other sensors 192 such as radar or lidar.

[0102] A feasibility result Ef is then passed to the safe dialogue control 150, which forwards the result Ef to the safe audio output 130 and / or the visual output unit 180 for output to the passenger or driver.

[0103] If the confirmed driving maneuver command Af is feasible, it is executed by issuing control signals Ak using actuators 200 of the vehicle.

[0104] The safe dialogue control 150 is designed to allow the driver to designate the passenger 400 as responsible for the vehicle's driving behavior via voice input 140 and / or haptic input 110, for example, with the voice command "Hand over the steering wheel to the passenger." To do this, the safe dialogue control 140 analyzes the voice input to determine the most probable instruction from a list Lf of possible instructions using a suitable method, such as syntax analysis or natural language processing. If the passenger authorization is correctly recognized using this method, the passenger 400 is prompted to confirm it via a safe audio output 130, for example, with the voice output "Passenger XY, please confirm the transfer of control of the vehicle behavior!"The passenger 400 can do this either haptically by means of a passenger control button 410 or by voice input with authentication 170 and the input of a key phrase that does not occur in the other language area (e.g. "Confirm, Ha-Four-Te-Two-Nine").

[0105] If the voice input cannot be clearly assigned to a possible driving instruction, or if confirmation is not given or not clearly given by the passenger 400, the safe dialogue control 150 can repeat the driving dialogue.

[0106] The safe dialogue control 150 prevents the execution of a command by the passenger 400 if the driver 100 makes a haptic input 110 or a manual vehicle operation using the steering wheel 111 or the pedals 112, so that the driver 100 can revoke the passenger 400's permission to issue driving instructions at any time. This can optionally also occur if the driver makes a voice input initiated by a key phrase.

[0107] The driving instruction of the passenger 400 is always reliably recognized because it is clearly assigned by the safe dialogue control 150 to a possible driving instruction of the autonomous vehicle control unit 190, this is made audible to the passenger 400 by means of a safe voice output and is then only executed upon confirmation by the passenger 400.

[0108] The driving behavior instruction is only accepted by the passenger 400 from the safe dialogue control 150 if the passenger can correctly perceive the driving situation, which is ensured by the proximity sensor signals or switches in the passenger headrest 402 and / or by the interior camera 120 with image recognition of the passenger 400's head.

[0109] Optionally, the passenger's driving instruction 400 can be refused if the attention detection 160 or the interior camera 120 detects fatigue, absence or unfitness to drive (e.g. due to slurred speech, slowed speech input and / or closed eyes or strongly dilated pupils).

[0110] Each driving maneuver command Af confirmed by the safe dialogue control 150 is assessed for its safe feasibility as input to the at least partially autonomous vehicle control unit, based on the sensors 191, 192 and the driving, traffic, and environmental situation generated from these signals. The command is approved if its execution is deemed sufficiently safe. This decision Ef is communicated to the safe dialogue control 150, along with a possible reason for rejection. The control then either confirms or rejects the execution via a safe audio output 130 and, optionally, a visual output to the driver 100 and passenger 400.

[0111] In the event of loud ambient noise, the safe audio output 130, for example, detects excessive ambient noise based on the ratio of the combined power of the microphone signals and the audio signals output by the loudspeakers to the total power of the microphone signals over a period relevant for communication (e.g., 50 ms). It then terminates the safe voice output and notifies the safe dialogue control 150 of this termination via signal E. The safe dialogue control 150 then safely terminates the driving dialogue, preventing any driving instructions from being executed under unsafe acoustic communication conditions.

[0112] The safe dialogue control 150 always communicates with a confirmed driving maneuver command Af to the at least semi-autonomously operating vehicle control unit 190 whether it is an instruction from the passenger 400 or the driver 100, so that it can reject these instructions from the passenger 400 due to lack of controllability of the vehicle by the passenger 400 or delayed controllability of the inactive driver 100.

[0113] Figure 4 shows a flowchart for a computer program that implements safe dialogue control. At step 00, the safe driver dialogue is aborted and an abort message Ef is output to the safe audio output 130 and optionally to the visual output unit 180.

[0114] Step 1 checks whether a fully recognized speech input (Ft) has been received. If not, step 1 is repeated; if so, the process continues with step 2.

[0115] In step 02, the speech input Ft is analyzed for the most probable instruction from a list Lf of possible instructions using a suitable method, e.g., syntax analysis or natural language processing. The selected instruction Af is then output to the safe audio output 130.

[0116] Step 3 checks whether attention to attention control has been positively confirmed. If not, the process returns to step 00; if so, it proceeds to step 4.

[0117] In step 04, positive attention is noted for the recognized driving instruction Af.

[0118] Step 5 checks whether passenger or driver authentication is present. If not, the process returns to step 00; if so, it proceeds to step 06.

[0119] In step 06, a positive authentication is recorded for the recognized driving instruction Af, either for the driver 100 or the passenger 400.

[0120] Step 7 checks whether a list of possible driving instructions Lf from the at least partially autonomous vehicle control unit 190 can be read. If not, the process returns to step 00; if yes, it continues with step 08.

[0121] In step 08, the possible driving instructions Lf are read in. In step 09, it is checked whether the signal K of the proximity sensor of the passenger headrest 402 indicates that the passenger's head 400 is near the passenger headrest 402. If not, the process returns to step 00; if yes, it continues with step 10.

[0122] In step 10, a positive overview of the passenger 400 is noted for the recognized driving instruction Af.

[0123] Step 11 checks whether signal P indicates that the driver has activated the vehicle pedals 112. If so, the process returns to step 00; if not, it continues with step 12.

[0124] In step 12, no manual operation of the accelerator pedals is recorded for the recognized driving instruction Af.

[0125] Step 13 checks whether signal L indicates that the driver has actuated the handlebar 111. If yes, the process returns to step 00; if no, it continues to step 14.

[0126] In step 14, for the recognized driving instruction Af, no steering input from the driver is recorded, and the system waits for the end of the safe acoustic voice output Fa that corresponds to the recognized driving instruction Af.

[0127] Step 15 checks whether signal E of the safe voice output indicates the correct acoustic output Fa for the driving instruction Af. If not, the system returns to step 00; if yes, it proceeds to step 16.

[0128] In step 16, the system records the safe voice output for the detected driving instruction Af and waits for confirmation from the passenger by pressing button B4 or via a confirming voice input Ft. In step 17, it checks whether confirmation corresponding to step 16 is received within a period T (e.g., 2 seconds). If not, it returns to step 00; if so, it proceeds to step 18. In step 18, the passenger's confirmation for the detected driving instruction Af is recorded.

[0129] Step 19 checks whether all positively recorded signals from steps 4, 6, 10, 12, 14, 16, and 18 are present. If not, the process returns to step 1; if so, it continues with step 20.

[0130] In step 20, the driving behavior instruction Af, which was reliably detected by the passenger 400, is output to the at least partially autonomous vehicle control unit 190.

[0131] Reference symbol list

[0132] 1. Procedure for operating a control unit

[0133] 2 vehicles

[0134] 3 Vehicle control system

[0135] 4 Calculation unit

[0136] 5 microphone fields

[0137] 100 drivers

[0138] 101 Driver's seat

[0139] 102 Driver's headrest

[0140] 103 Driver speakers

[0141] 110 operating button

[0142] 111 Steering wheel

[0143] 112 foot pedals

[0144] 120 Interior camera

[0145] 130 secure audio output

[0146] 140 speech recognition

[0147] 150 secure dialog control

[0148] 160 Attention detection

[0149] 170 Driver and passenger authentication unit

[0150] 180 visual output units

[0151] 190 autonomous vehicle control units

[0152] 191 Outdoor camera

[0153] 192 additional sensors

[0154] 200 actuators

[0155] 302 Sound evaluation unit

[0156] 303 Microphone

[0157] 400 passengers

[0158] 401 Passenger seat

[0159] 402 Passenger headrest

[0160] 403 Passenger speaker

[0161] 410 Passenger control button Af confirmed driving maneuver command

[0162] Ak control signals

[0163] B Signal from the operating button

[0164] E Signal

[0165] Decision on implementation

[0166] Driving maneuver order

[0167] Ft instructions from the driver or passenger

[0168] K signal from the passenger headrest

[0169] Kd interior camera image data

[0170] List of possible driving behavior instructions

[0171] Ma combined signal

[0172] No non-authentication signal

[0173] S signals from other sensors

[0174] Sa audio signal

[0175] Sh fatigue signal

[0176] Sk Image signals from the outdoor camera

[0177] 51 Assigning a permit

[0178] 52 Capturing a voice input

[0179] 53 Recognizing a driving maneuver order

[0180] 54 Generating and outputting an audio signal

[0181] 55 Capturing a signal output

[0182] 56 Determining a safe audio signal output

[0183] 57 Determining a confirmation

[0184] 58 Passing on the driving maneuver order

Claims

Patent claims 1. A method (1) for operating an autonomously or semi-autonomously operating control unit (190) of a vehicle (2), comprising: Assigning (S1 ) permission to control the vehicle (2) to a passenger (400) of the vehicle (2); Capture (S2) a voice input from the passenger (400); Recognition (S3) of a driving maneuver command based on the voice input of the passenger (400); Generating and outputting (S4) an audio signal corresponding to the detected driving maneuver command; Capture (S5) a signal output; Determine (S6), based on the captured signal output, whether the audio signal was safely output; Determine (S7) whether confirmation has been given by the passenger (400); and if it has been determined that the audio signal has been safely emitted and confirmation has been given by the passenger (400), transmit (S8) the driving maneuver command to the autonomous or semi-autonomous control unit (190) of the vehicle (2).

2. The method (1) according to claim 1, wherein the assignment (S1) of permission to control the vehicle (2) to the passenger (400) comprises: Detection of a signal from an operating button (110) that can be operated by a driver (100) of the vehicle (2) for the transfer of control of the vehicle (2).

3. The method (1) according to claim 1 or 2, comprising the assignment (S1) of permission to control the vehicle (2) to the passenger (400): Capturing a voice input from the driver (100); Recognition of a control transfer command based on the driver's voice input (100); Generating and outputting a further audio signal corresponding to the recognized control transfer command; Capturing another signal output; Determine, based on the detected further signal output, whether the further audio signal was successfully output; Determine whether further confirmation has been given by the driver (100) and / or passenger (400); and if it has been determined that the further audio signal has been safely given and confirmation has been given by the driver (100) and / or passenger (400), assign permission to control the vehicle (2) to the passenger (400).

4. The method (1) according to any one of claims 1 to 3, wherein determining (S7) whether confirmation has been given by the passenger (400) and / or determining whether further confirmation has been given by the driver (100) and / or passenger (400), comprises: Detection of a further signal from a confirmation button operable by the passenger (400) or driver (100).

5. The method (1) according to any one of claims 1 to 4, wherein determining (S7) whether confirmation has been given by the passenger (400) and / or determining whether further confirmation has been given by the driver (100) and / or passenger (400), comprises: Capturing a further voice input from the passenger (400) or driver (100); and Recognition of a confirmation command based on further voice input from the passenger (400) or driver (100).

6. The method (1) according to any one of claims 1 to 5, further comprising: removing control of the vehicle (2) from the passenger (400).

7. The method (1) according to any one of claims 1 to 6, comprising the acquisition (S2) of the voice input of the passenger (400), the acquisition (S5) of the signal output, the acquisition of the voice input of the driver (100), the acquisition of the further signal output, and / or the acquisition of the further voice input of the passenger (400) or driver (100): Acquisition of sound signals using a microphone field (5) and evaluation of the sound signals using a sound evaluation unit.

8. The method (1) according to any one of claims 1 to 7, wherein the audio signal corresponding to the detected driving maneuver command and / or the further audio signal corresponding to the detected control transfer command is generated with a characteristic modulation and the determination (S6) of whether the audio signal was safely output and / or the determination of whether the further audio signal was safely output is based on this characteristic modulation.

9. The method (1) according to any one of claims 1 to 8, further comprising: generating and outputting a graphic signal corresponding to the recognized driving maneuver command.

10. The method (1) according to any one of claims 1 to 9, further comprising: determining whether the passenger (400) is attentive, wherein the transmission (S8) of the driving maneuver command to the autonomously or semi-autonomously operating control unit (190) only takes place if it has been determined that the passenger (400) is attentive.

11. The method (1) according to any one of claims 1 to 10, further comprising: evaluating the driving maneuver command; Generating and outputting a further audio signal corresponding to the evaluation of the driving maneuver command; Capturing yet another signal output; Determine, based on the detected further signal output, whether the further audio signal was reliably output; and if the driving maneuver command was assessed as feasible and it was determined that the further audio signal was reliably output, execute the driving maneuver command.

12. A vehicle control system (3) comprising a computing unit (4); an interface to at least one loudspeaker (103; 403); and an interface to a microphone array (5), wherein the vehicle control system (3) is configured to perform the method (1) according to any one of claims 1 to 11.

13. A vehicle (2) comprising the vehicle control system (3) according to claim 12.

14. Non-volatile, computer-readable storage medium with a program stored therein which, when executed on a computing unit (4) of a vehicle control system (3) according to claim 12, instructs the vehicle control system (3) to execute the method (1) according to any one of claims 1 to 11.

Images