Method and device for changing the operating function of a multifunction steering wheel of a vehicle assigned to at least one operating element

Abstract

In a method and device for changing the operating function of a multifunction steering wheel of a vehicle assigned to at least one operating element, the image data of at least one image with a depiction of at least part of the vehicle interior are processed. Based on the image data, at least one hand of a vehicle driver is detected, and it is determined whether the movement of the hand is restricted. If restricted movement of the hand is determined, the operating function of at least one operating element arranged in a first area of ​​the multifunction steering wheel is assigned to a control element arranged in a second area of ​​the multifunction steering wheel.

Description

[0001] In a method and device for changing the operating function of a multifunction steering wheel of a vehicle assigned to at least one operating element, the image data of at least one image with an image of at least one part of the vehicle interior are processed. Based on the image data, at least one hand of a vehicle driver is detected.

[0002] Multifunction steering wheels are common in modern vehicles, allowing drivers to comfortably and safely control a variety of infotainment and assistance systems without having to take their hands off the steering wheel. Numerous buttons and knobs are arranged on the front of such multifunction steering wheels, positioned for easy thumb reach. These are often provided with symbols or labels to clearly indicate their functions. Some multifunction steering wheels feature subtle backlighting, which facilitates operation in low-light conditions.

[0003] Multi-function steering wheels allow you to control various infotainment system functions, including volume, changing radio stations or music tracks, answering phone calls via Bluetooth, and activating voice control. In addition, dedicated buttons or menus allow you to activate driver assistance systems such as adaptive cruise control, lane departure warning, and traffic sign recognition, or adjust driving modes. Some multi-function steering wheels even offer haptic feedback or vibration warnings to alert the driver to potential hazards or system interventions.

[0004] The concept of the multifunction steering wheel is an essential component of modern vehicle cockpits, which not only increases safety by minimizing distractions, but also significantly improves driving comfort and user-friendliness while driving.

[0005] Document WO 2005 096131 A2 discloses a steering wheel input device for vehicles that utilizes a touch-sensitive surface to intuitively control a variety of functions. By detecting and analyzing the position of palms and fingers, the device enables dynamic adjustment of the input surfaces, allowing users to operate both a virtual keyboard and specific vehicle controls such as turn signals, lights, and windshield wipers without having to remove their hands from the steering wheel.

[0006] Document DE 10 2012 018 354 A1 discloses a control device for a vehicle, in particular for controlling at least one vehicle system, with a touchpad interface which is designed to be coupled to a touchpad having at least two at least logically separate areas.

[0007] Document DE 10 2017 216 674 A1 discloses a device for controlling vehicle functions in a motor vehicle as a function of a touch of the steering wheel acting on the steering wheel of the motor vehicle. At least one sensor unit detecting a touch of the steering wheel detects a touch of the steering wheel and generates measurement signals. An evaluation and control unit receives the measurement signals generated by the sensor unit and transmits control signals as a function of the received measurement signals.

[0008] It may happen that the driver's mobility in one hand is restricted. This often makes it difficult to operate all the buttons on the multifunction steering wheel. If the other hand is used to operate all the controls on the multifunction steering wheel, the other hand must be removed from the steering wheel to operate some of the controls, which makes it difficult to steer the vehicle safely.

[0009] The object of the invention is to provide a device and a method for changing the operating function of a multifunction steering wheel of a vehicle assigned to at least one operating element, by means of which simple operation is possible.

[0010] This object is achieved by a device having the features of claim 1 and by a method having the features of the independent method claim. Advantageous further developments are specified in the dependent claims.

[0011] The device according to claim 1 ensures that, if the movement of one of the driver's hands is restricted, the operating function of at least one operating element arranged in a first area of ​​the multifunction steering wheel is assigned to an operating element arranged in a second area of ​​the multifunction steering wheel, whereby use of the operating function is possible with the driver's other hand, which is not restricted in its movement. The original operating function of the operating element arranged in the second area is then, at least temporarily, no longer assigned to this operating element. However, the operating function originally assigned to this operating element can still be restored or temporarily restored in combination with another input via a human-machine interface.

[0012] It is advantageous if the processing unit is configured to detect a restriction in the mobility of the driver's hand if, through processing the image data, it is determined that the driver is holding food, a drink, a smartphone, a tablet computer, a laptop computer, a book, a document, a passenger's hand, contacting the passenger, holding the gearshift lever, or contacting a pet in the hand or the hand of the relevant arm. This allows for easy detection of an existing restriction in mobility.

[0013] Furthermore, it is advantageous if the processing unit is configured to detect the hand by determining its position within the vehicle interior. This position can contribute to the assessment of the movement restriction.

[0014] If the driver's hand is not visible in the image, the processing unit can also be configured to determine the position of the hand based on an arm of the driver that is at least partially captured in the image. This allows a restriction of the hand's movement to be determined even if the hand itself is not visible in the image, e.g., if the hand is obscured in the image by a baby seat or the passenger's hand.

[0015] Furthermore, the processing unit can be configured to perform a human body pose estimation method and to determine the hand, a position of the hand, and / or the position of an arm of the vehicle driver using a human body estimation method. This allows for a simple determination of the position of the vehicle driver's hand. This is possible even if the hand itself is not visible in the image. Using the human body estimation method, the position of the hand can be determined based on the position of the arm, even if it is not visible in the image.

[0016] The processing unit can also be configured to determine which hand has restricted mobility. This makes it easy to identify the first area of ​​the multifunction steering wheel that cannot be easily reached and operated safely due to the restricted mobility of the driver's hand.

[0017] Furthermore, if the mobility of the right hand is restricted, the processing unit can be configured to assign at least the operating function of at least one operating element arranged in a first area on the right side of the multifunction steering wheel to a operating element arranged in a second area on the left side of the multifunction steering wheel. This allows the operating functions assigned to the operating elements in the first area to continue to be operated via operating elements arranged in the second area.

[0018] Furthermore, the processing unit can be designed to output the change in the function assignment through a human-machine interface, in particular via a display unit, preferably via a change in a symbol display on or next to the operating element to which the operating function has been assigned.

[0019] If the driver's hand, food, a drink, a smartphone, a book, and / or a document in the driver's hand, and / or the passenger's hand, and / or the gearshift lever are classified using an object recognition method, a restricted movement of the hand can be detected. The classification of these objects in contact with the hand can be carried out using a classification method using artificial intelligence or by conventional classification inside or outside the vehicle. In the case of classification outside the vehicle, the image data is transmitted to a central processing unit via a wireless communication network, and the classification result is transmitted back to the vehicle.

[0020] It is particularly advantageous if the processing unit is configured to automatically change the assignment of the operating function when a conversation between the driver and front passenger is detected and audio output is provided via a loudspeaker in the vehicle. For this purpose, a microphone is provided to detect a conversation between the driver and another passenger. The audio output can be provided via a loudspeaker in the vehicle, in particular an audio output from a streaming service (audio and / or video), a radio program, a navigation system, a driver assistance system, and / or a telephone.

[0021] The processing unit can also be configured to automatically change the operating function if the volume of an audio output is greater than a preset first value and / or to automatically change the function if the volume of an audio output is less than a preset second value. This makes it easy to change the assignment of the operating function for changing the volume to another control element if there is a high probability that the volume needs to be changed and the freedom of movement of one of the driver's hands is restricted.

[0022] The first and / or second threshold values ​​can be stored in a driver-specific database and / or on a mobile data storage device. The mobile data storage device can be the vehicle key, a mobile phone or other mobile device of the driver, such as a smartwatch or tablet computer, or a cloud storage device, preferably at the vehicle manufacturer.

[0023] It is particularly advantageous if the control element is a volume control element and / or a telephone control element, preferably for answering and / or ending a telephone call. The telephone control element can, in particular, be a green and / or red handset button. This allows for the continued easy use of frequently used operating functions.

[0024] The image data can also be processed using a neural network. This allows for simple and efficient processing of the image data for hand detection and / or for classifying objects in the vehicle interior.

[0025] The camera can be a camera integrated into the vehicle's interior rearview mirror or into the housing of the interior rearview mirror, a camera arranged above the driver and / or front passenger in the roof area and / or in the B-pillar and / or A-pillar on the driver's side and / or the front passenger side and / or on a central display unit of the vehicle. This makes it possible to classify not only the driver and their hand(s) but also other objects located in the interior of the vehicle that influence the driver, such as other people and / or animals, as well as possible and probable sources of interference. Furthermore, it is advantageous if the camera's field of view has a detection angle in the range of 100° to 150°, with the camera preferably being a monocular camera and / or an RGB-IR camera. This allows, in particular, image processing by the processing unit based on the recorded IR images.The use of IR images is advantageous because these images are independent of ambient light. Alternatively, the camera can be a pure IR camera and / or a stereo camera (3D camera). The camera's image data can be processed in a computer vision system provided by the processing unit. The use of an RGB-IR camera combines the ability to be independent of ambient light while simultaneously providing RGB information when visible light is available.

[0026] A second aspect relates to a method for changing the operating function of a multifunction steering wheel of a vehicle assigned to at least one operating element, having the features of the independent method claim. This method has the same advantages as the claimed device. In particular, the method can be further developed with the features of the dependent claims directed to the device.

[0027] Embodiments of the invention are explained in more detail below with reference to the figures, in which: Fig. 1 a perspective schematic representation of a cockpit of a vehicle; Fig. 2 shows a schematic representation of a first image of the interior of the vehicle taken by an interior camera of the vehicle; Fig. 3 shows a schematic representation of a second image of the interior of the vehicle taken by an interior camera of the vehicle; Fig. 4 shows a schematic representation of a third image of the interior of the vehicle taken by an interior camera of the vehicle, and Fig. 5 shows a flow chart for changing the operating function of a multifunction steering wheel of a vehicle assigned to at least one operating element.

[0028] Fig. 1 shows a perspective schematic representation of a cockpit 100 of a vehicle 102. A vehicle driver 104 sits on a driver's seat 106 of the vehicle 102. A display 108 for displaying information when the passenger airbag 146 of the vehicle 102 is deactivated is arranged, for example, in the area of ​​a center console 110 of the cockpit 100. The cockpit 100 also includes a central information display (CID) 112, a head-up display 114, and a graphical instrument cluster 116, which are arranged in a dashboard 118 of the vehicle 102. The dashboard 118 further includes air outlet nozzles 120 of an air conditioning system of the vehicle 102. In Fig. 1 shows exemplary air outlet nozzles 120 in the area of ​​the center console 110 and on a passenger side. The cockpit 100 further comprises a so-called multifunction steering wheel 122 with a first right-hand area 124 containing first control elements and a second left-hand area 125 containing second control elements, a gear selector switch 126, a pedal assembly 128, and an input unit 130 with a rotary wheel and a push-button function and / or a touch input field. This input unit 130 is also referred to as the Ergo Commander.

[0029] A camera 134 is arranged in the upper region of a windshield 132 of the vehicle 102. The camera 134 is arranged and aligned such that it captures images 200, 300, 400 with an image of an area of ​​the interior of the vehicle 102 comprising at least a portion of the driver's seat 106 and the passenger seat 202. Exemplary images 200, 300, 400 of the camera 134 are shown in the Fig. 2, Fig. 3 and Fig. 4. The camera 134 is particularly configured to capture a plurality of temporally successive images as a video sequence, generate image data corresponding to the images 200, 300, 400, and transmit this data to a control unit 136. The camera 134 has a field of view of 120 degrees. In other embodiments, the camera 134 may also have a field of view in the range of 100 degrees to 150 degrees or more.

[0030] The control unit 136 has data outputs 138 and data inputs 140, which serve for connection to other units of the vehicle 102, for example, to other sensors, input and output units, and control units of assistance systems and / or safety systems. In particular, the control unit 136 is connected to an airbag control unit 142, which is designed to trigger the airbags of the vehicle 102 in accident situations, i.e., in the event of a crash, in particular to control the triggering of the passenger airbag 146.

[0031] The control unit 136 further comprises a communication module 144 which is designed to establish a connection to a telecommunications network, in particular a mobile radio network.

[0032] Based on the image data from camera 134, control unit 136 can detect a hand 150, 152 of the driver 104 and determine whether there is a restriction in the movement of one of the two hands 150, 152 of the driver 104. In left-hand drive vehicles, i.e., vehicles 102 with the steering wheel 122 on the left and primarily intended for right-hand traffic, it is sufficient to detect only a restriction in the movement of the right hand 150 of the driver 104. If control unit 136 detects a restriction in the movement of the right hand 150 of the driver 104, the operating function of at least one operating element 160 arranged in the first right-hand area 124 of the multifunction steering wheel 122 is assigned to an operating element 162 arranged in the second left-hand area 125 of the multifunction steering wheel 122.This allows the vehicle driver 104 to execute the operating function by actuating an operating element 162 in the second right-hand area 125 by actuating the operating element 162 with his left hand 152. This generally allows the operating function to be used with the other hand 152 of the vehicle driver 104, which is not restricted in its movement. The original operating function of the operating element 162 arranged in the second area 125 is then no longer assigned to this operating element 162, at least temporarily. However, the operating function originally assigned to this operating element 162 can still be restored or temporarily restored in combination with another input via a human-machine interface.

[0033] The camera 134 can alternatively also be a camera 134 integrated into the interior rearview mirror of the vehicle 102 or integrated into the housing of the interior rearview mirror, a camera 134 located above the driver 104 and / or the front passenger in the roof area and / or in the B-pillar on the driver's side and / or the front passenger side, and / or a camera arranged on a central display unit of the vehicle (CID). This allows objects located in the interior of the vehicle 102 to be easily detected and classified. Several cameras can also be provided, the images of which are then preferably captured simultaneously, and the image data from images from at least two cameras are processed in a processing unit (not shown) arranged by the control unit 136 or outside the vehicle 102 in order to achieve even better detection of the hand 150 or hands 150, 152 of the driver 104 and to reliably determine any restriction in the movement of the hand 150.The hand 152 is particularly restricted in movement when the vehicle driver 104 is holding food, a drink, a smartphone, a tablet computer, a laptop computer, a book, a document, a passenger's hand in the hand 152, is otherwise contacting the passenger, or is holding the gear lever or is contacting a pet.

[0034] Furthermore, in other embodiments, the field of view of the camera can have a detection angle in the range of 100° to 150°, wherein the camera is preferably a monocular camera and / or an RGB-IR camera. This allows, in particular, image processing by the processing unit based on the recorded IR images. The use of IR images is advantageous because these images are independent of the ambient light. Alternatively, the camera can also be a pure IR camera and / or a stereo camera (3D camera). This allows a suitable camera to be used for the reliable detection of the hand 152 of the vehicle driver 104 and objects in the interior of the vehicle 102.

[0035] Fig. 2 shows a schematic representation of an image 200 of the interior of the vehicle 102 taken by the interior camera 134. The camera 134 is directed towards the driver's seat 106, a passenger seat 202 and a rear seat bench 204 of the vehicle 102. In the Fig. 2, the vehicle driver 104 sits in the driver's seat 106 and the other occupant 206 sits in the front passenger seat 202. Thus, in the present embodiment, the other occupant is front passenger 206.

[0036] When processing the image data of the image 200 captured by the camera 134, the control unit 136 detects at least the right hand 150 of the vehicle driver 104 and determines that this hand 150 is restricted in its mobility because it is in contact with the left hand 154 of the front passenger 206. The control unit 136 then assigns the operating function of at least one operating element 160 arranged in the first right-hand region 124 of the multifunction steering wheel 122 to an operating element 162 arranged in a second left-hand region 125 of the multifunction steering wheel 122.

[0037] Fig. 3 shows a schematic representation of a second image 300 of the interior of the vehicle 102, taken by the camera 134. A child 310 is sitting in a child seat 312 on the rear seat 204. When processing the image data of the image 300 taken by the camera 134, the control unit 136 detects at least the right hand 150 of the vehicle driver 104 and determines that this hand 150 is restricted in its mobility because he is holding a mobile phone 320 in this hand 150. The control unit 136 then assigns the operating function of at least one operating element 160 arranged in the first right-hand region 124 of the multifunction steering wheel 122 to an operating element 162 arranged in a second left-hand region 125 of the multifunction steering wheel 122.

[0038] Even when detecting the driver's glance at their smartwatch, the hand 150 of the driver 104 is restricted in its mobility. The control unit 136 then assigns the operating function of at least one control element 160 arranged in the first right-hand area 124 of the multifunction steering wheel 122 to a control element 162 arranged in a second left-hand area 125 of the multifunction steering wheel 122.

[0039] Fig. 4 shows a schematic representation of a third image 400 of the interior of the vehicle 102, captured by the camera 134. When processing the image data of the image 400 captured by the camera 134, the control unit 136 detects at least the right hand 150 of the driver 104 and determines that this hand 150 is restricted in its mobility because he is holding a beverage bottle 410 in this hand 150. The control unit 136 then assigns the operating function of at least one operating element 160 arranged in the first right area 124 of the multifunction steering wheel 122 to an operating element 162 arranged in a second area of ​​the multifunction steering wheel 122.

[0040] Fig. 5 shows a flowchart for changing the operating function of a multifunction steering wheel 122 of a vehicle 102 assigned to at least one operating element 160. The process is started in step S100. Subsequently, in step S102, an image is captured with the camera 134, and image data corresponding to the image is transmitted to the control unit 136. Subsequently, in step S104, the control unit 136 detects the right hand 150 of the vehicle driver 104 and then checks, in step 106, whether the freedom of movement of the hand 150 is restricted. If this is not the case, the process is terminated in step S110.However, if the control unit 136 determines in step S106 that the freedom of movement of the hand 150 is restricted, the control unit 136 subsequently assigns the operating function of at least one operating element 160 arranged in the first right-hand area 124 of the multifunction steering wheel 122 to an operating element 162 arranged in a second left-hand area 125 of the multifunction steering wheel 122 in step S108.

[0041] The process is then terminated in step S110. The process is preferably repeated regularly, in particular at intervals ranging from 1 second to 1 minute.

[0042] Alternatively or additionally, the process can also be carried out for the left hand 152 of the vehicle driver 104, in which case the operating function of at least one operating element 162 arranged in the second left area 125 of the multifunction steering wheel 122 is assigned to an operating element 160 arranged in a first right area of ​​the multifunction steering wheel 122.

[0043] The ones in connection with the Fig. The operating function mentioned in 1 to 5 can be a volume adjustment function and / or a telephone operating function, preferably for answering and / or ending a telephone call.

[0044] In the embodiments according to the Fig. 1 to 5, an output can also be made to inform the vehicle driver 104 about the change in the operating function, preferably via at least one display next to or on the operating element 160 and / or 162. The image processing can also be carried out with the aid of a neural network, in particular with the aid of artificial intelligence.

[0045] In the Fig. 1 to 5, the camera 134 and the control unit 136 form a device for changing the operating function of a multifunction steering wheel 122 of a vehicle 102 assigned to at least one operating element 160, 162. Further embodiments described in the Fig. Elements and features shown in Figures 1 to 5 and mentioned in the preceding description may be part of this device. The control unit 136 is also referred to as a processing unit. List of reference symbols 100 cockpits 102 vehicles 104 drivers 106 Driver's seat 108 ad 110 Center console 112, 114, 116 display element 118 Dashboard 120 air outlet nozzles 122 steering wheel 124 Controls right steering wheel area 125 Controls left steering wheel area 126 gear selector switch 128 pedals 130 input unit 132 Windshield 134 Camera 136 Control unit 138 Data output 140 Data input 142 Airbag control unit 144 Communication module 146 Passenger airbag 150, 156 right hand 152, 154 left hand 200, 300, 400 image 202 passenger seat 204 rear seat 206 passengers 310 child 312 Child seat 318 Smartwatch 320 smartphones 410 beverage bottle S100 to S110 process steps

Claims

[1] Device for changing the operating function of a multifunction steering wheel of a vehicle assigned to at least one operating element, with at least one camera (134) of the vehicle (102), which is designed to capture at least one image (200, 300, 400) with an image of at least part of the vehicle interior and to generate image data corresponding to the image (200, 300, 400), with a processing unit (136) which is designed to process the image data, wherein the processing unit (136) is further configured to detect at least one hand (150, 152) of a vehicle driver (104) based on the image data, wherein the processing unit (136) is further configured to determine whether the movement capability of the hand (150, 152) is restricted, wherein the processing unit (136) is designed, when the processing unit (136) has determined that the movement possibility of the hand (150, 152) is restricted, to assign the operating function of at least one operating element (160) arranged in a first region (124) of the multifunction steering wheel (122) to an operating element (162) arranged in a second region (125) of the multifunction steering wheel (122). [2] Device according to claim 1, wherein the processing unit (136) is designed to determine a restriction of the possibility of movement of the hand (150, 152) of the vehicle driver (104) when it is determined by processing the image data that the vehicle driver (104) is holding food, a drink (410), a smartphone (320), a tablet computer, a laptop computer, a book, a document, a hand of the passenger or holding a gear lever or contacting a pet in the hand (150, 152). [3] Device according to claim 1 or 2, wherein the processing unit (136) is designed to detect the hand (150, 152) by determining the position of the hand (150, 152) in the interior of the vehicle (102). [4] Device according to claim 3, wherein the processing unit (136) is designed, when the hand (150, 152) of the vehicle driver (104) is not visible in the image (200, 300, 400), to determine the position of the hand (150, 152) based on an arm of the vehicle driver (104) that is at least partially captured by the image. [5] Device according to one of the preceding claims, wherein the processing unit (136) is designed to carry out a human body pose estimation method and to determine the hand (150, 152), a position of the hand (150, 152) and / or the position of an arm of the vehicle driver (104) with the aid of a human body estimation method. [6] Device according to one of the preceding claims, wherein the processing unit (136) is designed to determine which hand (150, 152) is restricted in its ability to move. [7] Device according to one of the preceding claims, wherein the processing unit (136) is designed, when the mobility of the right hand (150) is restricted, to assign at least the operating function of at least one operating element (160) arranged in a first area (124) on the right side of the multifunction steering wheel (122) to an operating element (162) arranged in a second area (125) on the left side of the multifunction steering wheel (122). [8] Device according to one of the preceding claims, wherein the processing unit (136) is designed to output the change in the function assignment through a human-machine interface, in particular via a display unit, preferably via a change in a symbol display on or next to the operating element (160, 162) to which the operating function has been assigned. [9] Device according to one of the preceding claims, wherein if food, a drink (410), a smartphone (320), a book and / or a document in the hand of the driver (104), and / or a hand of the passenger (206) and / or a gear lever are classified with the aid of an object recognition method, a movement restriction of the hand (150, 152) is detected. [10] Device according to one of the preceding claims, wherein the processing unit (136) is designed to automatically change the assignment of the operating function when a conversation between the vehicle driver (104) and the passenger (206) is detected and an acoustic output is made via a loudspeaker in the vehicle (102). [11] Device according to one of the preceding claims, wherein the processing unit (136) is designed to automatically change the assignment of the operating function when the volume of an audio output is greater than a preset first value, and / or to automatically change the operating function when the volume of an audio output is less than a preset second value. [12] Device according to claim 11, wherein the first and / or the second limit value is stored in a database and / or on a mobile data carrier (320) in relation to the vehicle driver. [13] Device according to one of the preceding claims, wherein the operating element (160, 162) is a volume adjustment element and / or a telephone operating element, preferably for accepting and / or ending a telephone call. [14] Device according to one of the preceding claims, wherein the processing of the image data is carried out by means of a neural network. [15] Method for changing the operating function of a multifunction steering wheel of a vehicle assigned to at least one operating element, in which the image data of at least one image (200, 300) with an image of at least part of the vehicle interior are processed, in which at least one hand (150, 152) of a vehicle driver (104) is detected based on the image data, in which it is determined whether the movement of the hand (150, 152) is restricted, in which, if a restricted movement possibility of the hand (150, 152) has been determined, the operating function of at least one operating element (160) arranged in a first area (124) of the multifunction steering wheel (122) is assigned to an operating element (162) arranged in a second area (125) of the multifunction steering wheel (122).

Images