Method and vehicle for shifting between manual and autonomous driving

Abstract

The present invention relates to a method for switching between manual and automatic driving of a vehicle (50), wherein the driver (30) of the vehicle (50) is observed by means of at least one interior space camera (10) for handover and / or takeover intentions. If a handover intention is detected, the steering wheel (20) is moved away from the driver (30), and driving is switched to automatic driving. If a takeover intention is detected, the steering wheel (20) is moved toward the driver's (30's) hand (32), and driving is switched to manual driving.

Description

Technical Field

[0001] This invention relates to a method for switching between autonomous and manual driving of a vehicle, wherein the driver of the vehicle is observed by at least one interior space camera for handover and / or takeover intentions. If a handover intention is detected, the steering wheel is moved away from the driver, and driving is switched to autonomous driving. If a takeover intention is detected, the steering wheel is moved towards the driver's hands, and driving is switched to manual driving. Background Technology

[0002] Autonomous driving allows drivers to use their driving time for other purposes, such as work, relaxation, or sleep. It is generally desirable to move the steering wheel from its current position (as far away from the driver as possible) to another position. Various solutions have been developed for this purpose. However, it may be necessary or desirable to switch back to manual driving, i.e., driver control of the vehicle, during autonomous driving. This can be based solely on the driver's intention, but it can also be initiated by the driver in emergency situations. Then, the steering wheel must be brought back to a position within the driver's reach, and the transition from autonomous to manual driving must be completed.

[0003] Switching between the aforementioned driving modes typically requires specific driver intervention; the driver must manipulate a control element or similar device to trigger the mode change. If this control element is located in the dashboard area, it can be uncomfortable if the driver moves their seat backward during automatic driving, especially when switching to manual driving. On the other hand, if a driving emergency necessitates transferring control to the driver, and the driver must first manipulate the control element to move the steering wheel in their direction and change the driving mode, this manipulation can be time-consuming. Therefore, it is advantageous that both steering wheel movement and driving mode changes can be independent of the control element. Summary of the Invention

[0004] Therefore, the objective of this invention is to provide a solution for switching between automatic and manual driving without the driver of the vehicle manipulating the control elements, wherein the steering wheel is moved to a position adapted to the driving mode. This is preferably achieved not only for switching from automatic to manual driving, but also, conversely, for switching from manual to automatic driving.

[0005] The objective of this invention is achieved by a method according to one aspect of the invention and a vehicle according to one aspect of the invention. Other preferred embodiments of the invention are derived from other features according to the invention.

[0006] In the method according to the invention for switching between manual and automatic driving in a vehicle, at least one interior space camera in the vehicle detects at least the driver's hands. The driver's intention to take over and / or hand over is detected based on the hand movements using image analysis. Upon detection of a takeover intention, the steering wheel is moved toward the driver's hands, and driving is switched to manual driving; and / or upon detection of a handover intention, the steering wheel is moved away from the driver, and driving is switched to automatic driving.

[0007] Therefore, the method should enable switching between manual and automatic driving, wherein the position of the steering wheel can be adjusted to match the driving mode. As already described, manual driving is driving in which the driver controls the vehicle. Automatic driving is driving in which the driver does not intervene in vehicle control. These two driving methods will also be referred to as driving modes below.

[0008] In order to switch between driving modes without having to manipulate control elements, at least one interior space camera in the vehicle observes at least the driver's hands. Therefore, the at least one interior space camera is oriented such that the driver's hands are within the camera's field of view. Advantageously, not only the hands, but also other areas of the driver, such as their arms, face, or entire upper body, may be detected by the at least one interior space camera, as further explained elsewhere.

[0009] The at least one interior space camera may be a camera specifically configured to perform this method, or it may be a camera already configured in the vehicle for other tasks but which can be used together with the method according to the invention. The method may also be implemented using more than one camera, particularly positioned at different locations within the vehicle, to obtain a better overview of the movement of the driver's hands or optionally other areas of the driver's (body).

[0010] Detection of at least the driver's hands can preferably be performed continuously, but can also be performed at predetermined time intervals. The images detected by the at least one interior space camera are evaluated using appropriate image analysis to determine whether the driver's hand movements allow for the identification of an intention to take over control for switching to manual driving and / or an intention to hand over control for autonomous driving. Therefore, image analysis checks or detects whether the driver, for example, moves their hands toward the steering wheel from a stationary position in autonomous driving, if possible, performing a grasping motion and thus signaling that the driver wants to take over control and switch to manual driving. If the driver, during manual driving, for example, places their hands between their knees or takes an object, such as a drinking bottle, into their hands, this can indicate an intention to hand over control to autonomous mode. Here, the term "hand movement" includes movements performed with one or both hands. Hand movements are determined here, in particular, by the position of the one or both hands, the distance between the one or both hands and the steering wheel or its usual position, the direction of movement of the one or both hands, and / or the speed of movement of the one or both hands. Hand movements can be intentional or involuntary.

[0011] For image analysis itself, known methods can be used. For example, known methods include pattern recognition methods and artificial neural networks.

[0012] If the driver's hand movements indicate a shift to manual driving, i.e., an intention to take over, then according to the invention, assuming the steering wheel is positioned away from the driver during automatic driving, the steering wheel is moved toward the driver's hands, i.e., toward the driver and therefore further toward the interior of the vehicle. For example, the steering wheel movement can be achieved using an electric motor that enables electric steering column adjustment.

[0013] Furthermore, the driving mode is also switched, specifically, the vehicle transitions from automatic driving mode to manual driving mode, in which the driver has control of the vehicle. The switching of driving mode preferably occurs only when the driver is able to grip the steering wheel, or at least almost able to grip it during its movement towards the driver. This ensures that the vehicle will not continue to move uncontrollably even if the driver cannot yet grip the steering wheel. Simultaneously, valuable time is saved because the driver can immediately take over driving upon contact with the steering wheel.

[0014] If image analysis detects the driver's intention to hand over control—that is, the driver's willingness to switch to autonomous driving—the steering wheel is moved away from the driver or their hands, giving the driver more space for their activities and thus greater comfort during autonomous driving. Furthermore, once the steering wheel is moved away from the driver, the driving mode switches to autonomous driving, ensuring the vehicle never continues to move uncontrollably.

[0015] The position to which the steering wheel is moved when there is an intention to transfer it can also be called the stationary position, while the position when actively operating the steering wheel can be called the driving position. The stationary position of the steering wheel can preferably be achieved by sinking or retracting the steering wheel into the vehicle's dashboard.

[0016] The method can be designed to function only when there is an intention to hand over, only when there is an intention to take over, or both. In the latter case, the vehicle reacts accordingly based on the detected intention.

[0017] A first preferred embodiment of the method according to the invention is configured to indicate to the driver whether a handover intention and / or takeover intention has been detected. Therefore, the driver should be informed whether a hand movement triggering a change in driving mode has occurred. This indication can be advantageous when it is precisely in a critical situation where the driver wants to switch to manual driving that the intention to switch has been successfully recognized, in order to convey a sense of security to the driver. Conversely, this is also meaningful because the driver can be assured that if the driver releases the steering wheel, the vehicle will not continue to move uncontrollably.

[0018] This design is particularly useful when the driver's hand movements cannot be unilaterally detected or evaluated, thus making it impossible to reliably detect the driver's intentions, and consequently, the steering wheel position and driving mode cannot be changed or not changed. In such cases, signals can be set to convey to the driver that the driver's intention to change cannot be reliably assigned, thereby allowing, for example, repetition of hand movements or manual confirmation of the intention change via operating elements. This also intercepts movements made by the driver, for example, unconsciously, i.e., movements performed without any intention to change the driving mode.

[0019] Signals indicating whether a handover and / or takeover intent has been detected can be emitted acoustically and / or visually. Acoustic signals can be a single tone or a series of tones, as used, for example, when using a computer to indicate success or failure of an action, or can include a warning tone or a series of warning tones if the transition is not successfully detected and cannot be executed. Visual signals can be the illumination of control lights, backlit symbols in the dashboard display area, vehicle displays, such as monitors or similar devices, and verbal and / or graphic confirmations.

[0020] As already described, the at least one interior space camera can be a vehicle-specific or existing interior space camera. Preferably, the at least one interior space camera is positioned above the windshield, in a mirror, in an interior lighting fixture, or on the roof of the vehicle.

[0021] The area above the windshield is unrestricted in the lateral direction to the driving direction. That is, at least one camera can be positioned arbitrarily along the entire width of the windshield, as long as the at least one camera can detect at least the driver's hand. The at least one interior space camera can be arranged at or within the rearview mirror, providing a good overview of the driver's hand movements and, if possible, other changing indicators, and can simultaneously utilize media connections within the rearview mirror area.

[0022] However, the at least one interior space camera can also be arranged in or at a reading light, ambient lighting device, or similar object and oriented towards the driver's hands. The arrangement of the at least one interior space camera in or on the vehicle's ceiling, and therefore in or on the vehicle's roof, is also suitable for detecting the driver's hand movements.

[0023] If the method is implemented using more than one camera, then the arrangement of the cameras at different locations proves advantageous, particularly when the field of view and / or orientation of the cameras are chosen such that the driver's hand movements can be detected from different angles. This improves the accuracy in detecting the driver's changing intentions.

[0024] For the at least one interior space camera, any type of camera can be used, particularly a stereo camera, a thermal imaging camera, or a time-of-flight camera. Since a stereo camera can generate a three-dimensional image using two and a half images, the driver's hand movements can be assessed more clearly and therefore more reliably. A time-of-flight camera (TOF camera) is a camera that uses a time-of-flight method to achieve a three-dimensional depiction of the area it captures and thus also allows for a more precise assessment of the driver's hand movements. Thermal imaging cameras have proven advantageous because they capture thermal radiation, and therefore, at least the detection of the driver's hands is not hindered by reflections and glare events.

[0025] Another embodiment of the method according to the invention involves detecting additional indications of takeover and / or handover intentions using at least one interior space camera, particularly driver gestures, driver's gaze direction, and / or driver's upper body movements. Besides driver hand movements that indicate takeover or handover intentions, other indications should also be detected. For this purpose, at least one camera is used to detect other areas of the driver's body and to evaluate them using image analysis. However, such detection can also be performed using other interior space cameras.

[0026] For example, additional indicators for takeover and / or handover intentions could be specially designed gestures as hand movements. Gestures can be understood as intentionally performed, repeatable hand movements correspondingly to the handover or takeover intention. Such gestures may be limited to one hand or require the use of both hands. These gestures can be preset as predefined gestures on the vehicle side and / or predefined by the driver as user-defined gestures. If such a gesture is detected by at least one interior space camera and recognized using image analysis, the takeover or handover intention is explicitly transmitted to the vehicle, allowing the steering wheel to immediately move to the corresponding position and the driving mode to change.

[0027] Another indicator could be the driver's direction of observation. This direction of observation can be determined based on the position and orientation of the driver's head, or it can be derived directly through image analysis from at least one interior space camera. For example, if a hand movement detects an intention to take over, but the driver is not looking in the direction of travel, but rather, for example, at the passenger, then handing over manual driving could be dangerous because the driver is not paying attention to traffic conditions. Therefore, it would be meaningful to, in such cases, recognize the intention to hand over without changing the driving mode and without adjusting the steering wheel, and / or by providing a warning indicating the conflict between the detected hand movement and the detected direction of observation.

[0028] The detection and analysis of the driver's observation direction can also lead to the conclusion that hand movements may be performed unconsciously, and therefore without any actual intention to hand over or take over. The observation direction, as another indicator, can therefore also be understood as a verification of the driver's hand movements.

[0029] Alternatively or additionally, the driver's upper body movements can serve as other indicators of intent to take over or hand over. Thus, the temporal correlation between an upright upper body and hand movements indicating an intent to take over can be used to confirm this intention. Similarly, following hand movements indicating an intent to hand over, leaning back until a lying position can also confirm this intent to hand over. Therefore, as with the analysis of the direction of observation, the analysis of the driver's upper body movements allows for the estimation of whether involuntary hand movements are detected or whether there is an actual intention to change driving modes, because, for example, while the driver is in a lying position, manual takeover is either not possible without danger or is not intentionally indicated.

[0030] One important application of the method according to the invention is vehicle takeover and transition to manual driving, especially in emergency situations. Therefore, it is suitable to initially move the steering wheel quickly upon detecting an intention to take over, and then slowly move the steering wheel toward the driver's hands as they approach the driving position. The aim is to minimize the distance between the driver's hands and the steering wheel as quickly as possible, and to use a reduced speed only in the final stages of steering wheel movement when adjusting the steering wheel, so that the driver can comfortably contact the steering wheel.

[0031] "Fast" here can mean, on the one hand, that the movement of the steering wheel is carried out at the highest possible speed (but at least a high speed), such as when the steering column is adjusted, but on the other hand, it specifically refers to the relative data given relative to the significantly slower movement shortly before the driver's hands come into contact with the steering wheel.

[0032] The rapid movement can preferably be performed at half of the travel segment that the steering wheel needs to travel, further preferably at 60%, and even more preferably at 70%.

[0033] To facilitate driver takeover or gripping of the steering wheel as it moves towards the driver, it is meaningful to move the steering wheel to a driver-specific position upon detecting a takeover intent. Therefore, the steering wheel should not simply emerge from its rest position and move towards the interior of the vehicle, but rather move to a position in which the driver can easily reach the steering wheel and is particularly familiar with the position. In the simplest case, this position corresponds to the position of the steering wheel previously manually driven. However, if a change of driver occurs during this period, the habitual or desired steering wheel position may differ. Therefore, the driver and their preferred steering wheel position should be correlated so that the steering wheel's driving position can be individually set based on the driver's identity upon detecting a takeover intent. For example, the driver's identity can be determined based on the vehicle key used or by means of image recognition.

[0034] In an alternative design of the method, upon detecting a takeover intent, the possible contact position between the driver's hand and the steering wheel should be determined based on the position of the driver's hand and / or the direction of movement of the driver's hand, and the steering wheel should be moved to the possible contact position.

[0035] The position and / or direction of movement of the driver's hands when indicating an intention to take over are determined through detection using at least one interior space camera and subsequent image analysis. Therefore, it can be calculated, for example, by a suitable computing unit, as to where the driver's hands are moving and at what point in time their position is. The expected or possible position and / or direction of movement of the driver's hands can be compared with the possible direction and / or speed of movement of the steering wheel, thereby enabling the determination of the possible contact point where the driver's hands touch the steering wheel.

[0036] Therefore, instead of setting an intentional or driver-specific driving position for the steering wheel, a driving position is set that allows the driver's hands to contact the steering wheel as quickly as possible. However, this can be based on and adapted to known driving positions of the steering wheel, especially driver-specific ones.

[0037] It can be advantageous for the steering wheel to stop moving when touched by the driver. Currently, it is generally possible to detect whether the steering wheel has been touched by the driver. Different techniques and methods are known for this. Independent of the chosen detection of steering wheel touch, the movement of the steering wheel toward the driver's hands should stop upon confirmation of such touch. This applies both to movement of the steering wheel from its rest position and to its driving position, because in both cases, the cessation of steering wheel movement is appropriate.

[0038] For example, if the steering wheel is moved to a stationary position to switch to autonomous driving, and then a critical traffic situation occurs, the movement of the steering wheel can be stopped by touching it, and the driver can grip the steering wheel to manage the situation. When moving the steering wheel out of the driving position, it is possible that the driver moves their hands faster than expected and / or the driver prefers another driving position at that moment. In both cases, further movement of the steering wheel towards the driver is undesirable or counterproductive. Therefore, in such cases, this further movement can also be stopped by touching the steering wheel. This does not preclude the possibility of subsequent adaptation to the driving position.

[0039] In addition to the method according to the invention, protection is also claimed for a vehicle capable of autonomous driving, and the vehicle is constructed and configured to implement the method according to the invention. For this purpose, such a vehicle shall include: at least one interior space camera that detects at least the driver's hands; means for processing and analyzing images detected by the at least one interior space camera; and a control device that triggers steering wheel adjustment and driving mode switching based on the image analysis results.

[0040] The vehicle using the method and construction according to the invention enables the switching of the vehicle's driving mode via hand movements, thereby triggering the steering wheel to a position adapted to the driving mode. Here, the switching from automatic to manual driving, and vice versa, can be easily achieved without first manipulating the operating elements. This improves driving comfort and, particularly, shortens the handover time until the driver gains control of the vehicle when switching to manual driving.

[0041] Unless otherwise stated in individual cases, the different embodiments of the invention mentioned in this application can be advantageously combined with each other. Attached Figure Description

[0042] The invention is described below with reference to the accompanying drawings and embodiments. In the drawings:

[0043] Figure 1 The image shows the driver in the vehicle during autonomous driving.

[0044] Figure 2 The driver's hands are shown when indicating an intention to take over; and

[0045] Figure 3 The possible locations of the interior space cameras used to detect the driver are shown. Detailed Implementation

[0046] Figure 1The image shows a driver 30 in vehicle 50, leaning back with his arms crossed. Steering wheel 20 is in a position away from the driver, referred to as the stationary position 22. Steering wheel 20 is moved to this position if it is not used during autonomous driving. A stereo camera 10 is positioned above steering wheel 20 and the vehicle's windshield (position 12), configured to detect at least the driver 30's hands 32.

[0047] Based on the photographed image of the driver's hand 32, the driver's hand movements can be analyzed to determine whether the driver intends to take control of the vehicle 50, i.e., whether they want to switch to manual driving. To demonstrate this, the driver can... Figure 2 The driver's hands 32 are shown gripping the steering wheel 20. For illustration, the driver's two hands 32 are shown at different stages of the grip, 32.R and 32.L. Once this gripping movement is detected based on images captured by at least one interior space camera 10 and subsequent image analysis, the steering wheel 20 is moved away from the steering wheel 20. Figure 2 The stationary position 22 moves toward the driver 30 into the driving position 24.

[0048] The steering wheel 20 moves at an increased speed during the first 70% of its travel to minimize the distance between the steering wheel 20 and the driver's hands 32 as quickly as possible. During the remaining 30% of the travel, the steering wheel 20 moves more slowly to the final driving position 24, allowing the driver 30 to grip the steering wheel 20 comfortably. Furthermore, once the hands 32 can almost fully grip the steering wheel 20, driving switches to manual mode.

[0049] If the driver's hand 32 has reached the steering wheel 20 before the target driving position 24, the movement of the steering wheel 20 will stop so as not to work against the movement of the driver 30.

[0050] Alternatively, the driving position 24 of the steering wheel 20 can be a driver-specific predetermined position, to which the steering wheel 20 moves when the currently acting driver 30 signals an intention to take over, or this position can be determined independently. For this purpose, the position, location, and timing of the driver 30's hands 32 are estimated based on their position, direction, and speed of movement. The steering wheel 20 is then moved from the driving position 24 to this possible contact position.

[0051] If the driver 30 now places his hands 32 on his knees or crosses his hands, as Figure 2As shown, this is also detected by at least one interior space camera 10 and identified as a handover intention after image analysis. Therefore, the steering wheel 20 moves to the stationary position 22, and the driving mode is switched to autonomous driving.

[0052] To indicate to the driver 30 that their intention to take over or hand over control has been detected, a symbol is used to display the success or failure of the detection on a display in the instrument panel area of ​​the vehicle 50. Optionally, an acoustic signal may be used to further indicate success or failure.

[0053] Depend on Figure 3 Alternative placement positions for the at least one interior space camera 10 or optionally other cameras are visible. In addition to the position 12 above the windshield, the interior space cameras may also be positioned in or at the rearview mirror 14 of the vehicle 50 and / or at or in the interior space lighting device 16 of the vehicle 50, and oriented toward the driver 30.

[0054] To ensure that involuntary hand movements do not unintentionally trigger a change in driving mode, in addition to the driver's hands 32, the position and orientation of the driver's upper body 36 and head 34 are also detected using the interior space camera 10 or optionally other cameras (not shown). If the detected hand movements indicate a takeover intention, image analysis can determine whether the driver 30 is looking forward in the direction of travel. If so, the driving mode is changed to manual driving, and the movement from the steering wheel 20 to the driver's position 22 is consistent. Conversely, if the driver 30 is not looking forward or has their eyes closed, no change is made, and a warning sound is issued.

[0055] The position of the driver's upper body (36) is also assessed in the same way. If the upper body is leaning back or even lying down, it is impossible to expect the driver to take over driving.

[0056] List of reference numerals

[0057] 10 interior space cameras

[0058] 12. The position above the windshield

[0059] 14 rearview mirrors

[0060] 16 Interior Space Lighting Fixtures

[0061] 20 steering wheel

[0062] 22. Steering wheel stationary position

[0063] 24-inch steering wheel driving position

[0064] 30 drivers

[0065] 32 Driver's Hands

[0066] 34 Driver's head

[0067] 36 Drivers' upper bodies

[0068] 50 vehicles

Claims

1. A method for switching between manual driving and automatic driving in a vehicle (50), wherein, - At least one interior space camera (10) in the vehicle (50) detects at least the hand (32) of the driver (30) of the vehicle (50); and - Using image analysis to detect the driver's (30's) intention to take over and / or hand over the vehicle (50) based on hand movements; and - Upon detecting an intention to take over, the steering wheel (20) of the vehicle (50) is moved toward the hand (32) of the driver (30), and driving is switched to manual driving; and / or - Upon detecting a handover intention, the steering wheel (20) is moved away from the driver (30), and driving is switched to automatic driving. - Wherein, upon detecting a takeover intent, the possible contact position between the driver's (30) hand (32) and the steering wheel (20) is determined based on the position of the driver's (30) hand (32) and / or the direction of movement of the driver's (30) hand (32), and the steering wheel (20) is moved to the possible contact position. -In addition, the movement of the steering wheel (20) stops when it is touched by the driver (30).

2. The method according to claim 1, characterized in that, The driver (30) is given a signal indicating whether a probe has been conducted regarding the intention to hand over and / or take over.

3. The method according to claim 1 or 2, characterized in that, The camera above the windshield (12), the camera in or at the mirror (14), the camera in or at the interior space lighting device (16), or the camera on the roof of the vehicle (50) may be used as the at least one interior space camera (10).

4. The method according to any one of claims 1 to 2, characterized in that, A stereo camera, a thermal imaging camera, or a time-of-flight camera may be used as the at least one interior space camera (10).

5. The method according to any one of claims 1 to 2, characterized in that, The at least one interior space camera (10) is used to detect additional instructions for takeover and / or handover intentions.

6. The method according to any one of claims 1 to 2, characterized in that, The at least one interior space camera (10) is used to detect additional instructions for takeover and / or handover intentions, wherein the additional instructions are the driver's (30) gestures, the driver's (30) viewing direction and / or the movement of the driver's (30) upper body (36).

7. The method according to any one of claims 1 to 2, characterized in that, Upon detecting an intention to take over, the steering wheel (20) is initially moved quickly and then slowly toward the driver's (30) hand (32) as it approaches the driving position.

8. The method according to any one of claims 1 to 2, characterized in that, Upon detecting an intention to take over, the steering wheel (20) is moved toward a position specific to the driver.

9. A vehicle (50) capable of autonomous driving, the vehicle being configured to implement the method according to any one of claims 1 to 8.

Images