Gesture recognition device and method for recognizing a gesture of a vehicle occupant

Abstract

Method for detecting a gesture of a vehicle occupant, comprising the following steps: - Capturing the interior of the vehicle using an imaging sensor system with an initial frame rate and an initial lighting setting to generate an initial series of images; - Searching for a hand (104) of the occupant in the first series of pictures; - if a hand (104) of the occupant is detected in the first series of images, capture the interior of the vehicle using the imaging sensor system with a second frame rate and a second lighting setting to produce a second series of images; - Evaluating the second series of images to identify the occupant's gesture; - Determining the distance of the occupant's hand (104) from a light source (110) that produces the second lighting setting; and - Adjusting the second lighting setting depending on the distance of the hand (104) from the light source (110) that produces the second lighting setting.

Description

[0001] The present invention relates to a gesture recognition device and a method for recognizing a gesture of a vehicle occupant. The present invention relates in particular to the multi-stage recognition of a gesture of a vehicle occupant.

[0002] Gesture recognition systems using a human-machine interface require a system for detecting the hand and fingers of the vehicle occupant. The occupant places their hand within an interaction area and can control a vehicle system through finger movements and gestures. Camera-based systems are typically used to detect the occupant's hand. To achieve the greatest possible independence of the occupant's hand illumination from ambient light, infrared light sources are often employed to additionally illuminate the interaction area and, consequently, the occupant's hand.

[0003] If the system only needs to recognize a hand gesture from an occupant, refresh rates between approximately 30 and 60 Hz are common and sufficient. However, operating a vehicle system requires faster recognition of the hand or the current finger position, similar to using a mouse. Consequently, a high refresh rate of over 200 Hz is necessary to avoid latency that could be perceived by the user. Such a high refresh rate necessitates more powerful additional illumination using infrared light sources due to the short exposure time.

[0004] Conventional systems for recognizing occupant gestures in a motor vehicle use cameras with a standard frame rate or high-speed cameras. The standard frame rate of approximately 30 to 60 Hz allows for good hand and / or finger recognition, but it results in significant latency. This latency is perceptible to the user and is therefore undesirable.

[0005] A high refresh rate, for example, a refresh rate of approximately 250 Hz or higher, requires very strong additional infrared illumination due to the short exposure times. Such strong infrared illumination can cause eye irritation for a user of the system due to glare from the intense infrared light. Furthermore, strong infrared light sources generate unwanted heat.

[0006] German patent DE 10 2014 201 313 A1 discloses that gestures can be tracked with a camera. The depth of field can be defined by adjusting the illuminance of a light source.

[0007] German patent DE 11 2013 000 590 T5 discloses an infrared light-emitting diode as a light source and a camera with an exposure time of 100 µs. This also allows gestures to be observed.

[0008] DE 10 2013 100 521 A1 discloses a sensor arrangement for capturing operating gestures on vehicles. This includes a control and evaluation unit, a light source and a capture device, which interact as a time-of-flight camera.

[0009] From DE 10 2008 031 697 A1 a device and a method for object identification are known, wherein image areas with a strong contrast difference to each other are used to infer the presence of an object.

[0010] DE 11 2013 005 337 T5 teaches an imaging system in which light elements can be controlled and aligned in such a way that the illumination of objects can be reduced or increased as required.

[0011] WO 2015 / 053451A1 discloses a gesture recognition module for a mobile device. Infrared light is emitted and a gesture is recognized based on the reflected infrared light, with movement of the gesture being detected based on changes in the reflected infrared light.

[0012] The invention aims to create a method and a device for recognizing a gesture of an occupant that causes the shortest possible latency times.

[0013] The object of the invention is achieved by a method according to claim 1, a gesture recognition device according to claim 5, and a vehicle according to claim 8. The dependent claims describe preferred embodiments.

[0014] The inventive method for detecting a gesture of a vehicle occupant comprises the step of capturing the interior of the vehicle using an imaging sensor system with a first frame rate and a first illumination setting to generate a first series of images. In the first series of images, the system searches for the occupant's hand. If the occupant's hand is detected in the first series of images, the interior of the vehicle is captured using the imaging sensor with a second frame rate and a second illumination setting to generate a second series of images. Finally, the second series of images is evaluated to detect a gesture of the occupant.

[0015] The initial detection and classification of the hand and / or fingers occurs at a lower frame rate of the imaging sensor, such as the camera. This allows for adequate illumination and image quality even with low infrared illumination intensity. This enables hand recognition. Once the hand is clearly detected, latency-free tracking is performed at a high frame rate with dynamically adjusted infrared illumination. The intensity of the infrared illumination is dynamically controlled for each frame of the second image series, taking into account the ambient light, the distance of the hand from the infrared light source, and the skin color and / or clothing.

[0016] The invention has the advantage of enabling robust interaction because tracking only occurs after a clear initial classification of the hand and / or fingers. No tracking occurs with other objects, in particular no tracking with enhanced infrared illumination.

[0017] This reduces the glare caused by infrared lighting, which is often perceived as unpleasant by the occupant. Furthermore, it reduces heat generated by the lighting, leading to simpler technical integration into a vehicle and ultimately to energy savings.

[0018] According to the invention, the method determines the distance of an occupant's hand from a light source that produces the second lighting setting. The second lighting setting is adjusted depending on the distance of the hand from the light source that produces the second lighting setting. The brightness of the light produced by the light source decreases with the square of the distance. Consequently, the light source must emit light with a higher intensity when the occupant's hand is farther away from the light source.

[0019] The second refresh rate can be higher than the first. Alternatively, or additionally, the second lighting setting can produce a brighter illumination of the occupant's fingers than the first lighting setting. Since the first refresh rate is lower than the second, it only requires a weaker illumination. The higher refresh rate, and consequently the brighter lighting setting, is only necessary for latency-free finger tracking.

[0020] The system can determine the speed of the fingers during the gesture. It can adjust the second refresh rate based on the occupant's finger speed during the gesture. Alternatively, or additionally, the second lighting setting can be adjusted based on the occupant's finger speed during the gesture. The higher the finger speed, the higher the refresh rate should be. However, a higher refresh rate requires brighter lighting.

[0021] The method can also determine the skin color of the fingers and / or clothing, in particular the color of the occupant's clothing. The second lighting setting can be adjusted depending on the skin color of the fingers and / or the occupant's clothing.

[0022] The invention also relates to a computer program product which, when loaded into a memory of a computer with a processor, performs the steps of the method described above.

[0023] The invention also discloses a gesture detection device with an imaging sensor system comprising an image acquisition sensor with a variable frame rate and an illumination device. The gesture detection device includes an evaluation control unit that can control the imaging sensor system and evaluate the images it generates. The evaluation control unit is configured to control the imaging sensor system such that the imaging sensor system generates a first series of images with a first frame rate and a first image control setting. The evaluation control unit is configured to search for an occupant's hand in the first series of images.The evaluation control unit is designed, if the occupant's hand is detected, to control the imaging sensor system in such a way that the vehicle's interior is captured by the imaging sensor system with a second frame rate and a second lighting setting in order to generate a second series of images. The evaluation control unit can then analyze this second series of images to recognize a gesture by the occupant.

[0024] The gesture recognition device can be further developed as previously described with regard to the procedure.

[0025] According to the invention, the gesture detection device includes a distance detection unit configured to determine the distance of the occupant's hand from the light source. The distance detection unit can be implemented by the evaluation control unit. The evaluation control unit is configured to control the imaging sensor system such that the second lighting setting is varied depending on the distance of the hand from the light source that generates the second lighting setting.

[0026] The second refresh rate can be higher than the first. The second lighting setting can produce a stronger illumination of the occupant's fingers than the first lighting setting.

[0027] The evaluation control unit can be configured to determine the speed of the fingers during the gesture. The evaluation control unit can be configured to determine the second frame rate depending on the speed of the occupant's fingers during the gesture and / or the second lighting setting depending on the speed of the occupant's fingers during the gesture.

[0028] The invention also relates to a motor vehicle with the gesture recognition device described above.

[0029] The invention will now be explained with reference to the figures, which show an exemplary and non-limiting embodiment of the invention, wherein Fig. 1 shows a gesture recognition device; and Fig. Figure 2 shows a flowchart of a procedure for latency-free gesture recognition.

[0030] Fig. Figure 1 shows a gesture recognition device installed in a vehicle. A display unit 102 shows content that a vehicle occupant can select. An evaluation control unit 108 controls an infrared illumination device 110 and a camera 112 such that a detection area or interaction area, in which an occupant's hand 104 and fingers 106 are located, is illuminated and evaluated. By moving their fingers 106, the user can select an element on the display unit 102. This element can be highlighted in color on the display unit 102. An action can be performed in response to the selection of an element on the display unit 102, a submenu can be opened in response to the selection of a single element on the display unit 102, or the like, as is known to those skilled in the art.

[0031] The camera 112 can be a two-dimensional semiconductor sensor, a two-dimensional CMOS sensor, a two-dimensional CCD sensor, and / or a stereo sensor comprising a plurality of semiconductor sensors. Three-dimensional imaging can be achieved, for example, by structured light projection. Such sensors and illumination methods are known to those skilled in the art and need not be described further for the sake of conciseness. The camera can include a bandpass filter that allows only IR light to pass through.

[0032] The gesture recognition device 100 also includes a distance detection device 114, which determines the distance of the hand 104 and / or fingers 106 from the infrared illumination device 110. Depending on the distance of the hand 104 and / or fingers 106, the evaluation control device 108 can adjust the intensity of the infrared illumination 110. The distance detection device can be implemented by the evaluation control device 108.

[0033] The procedure will now be carried out with reference to Fig. Figure 2 describes in more detail and shows a flowchart of the method 200 according to the invention. In step 202, the interaction area is illuminated with low-intensity infrared lighting. The occupant must place their hand 104 into the interaction area. There, the interaction area is illuminated by the low-intensity infrared lighting corresponding to the first lighting setting.

[0034] In step 204, the camera attempts to capture an image of hand 104. For this, an initial frame rate is used, which is between approximately 10 Hz and approximately 60 Hz, preferably between approximately 50 Hz and 60 Hz. This frame rate corresponds to the first frame rate. Due to the lower frame rate, only the low-intensity infrared illumination set in step 202 is required. Since only the hand needs to be detected or classified in step 204, a delay or latency of a few hundred milliseconds due to the comparatively low frame rate and / or the resource-intensive classification is not a problem.

[0035] In step 206, it is checked whether a hand 104 and / or a finger 106 can be detected in the first image or image series generated in step 204. If no hand and / or finger are detected in step 206, the procedure returns to step 204 to generate another first image and / or another first image series with the first illumination setting and the first frame rate.

[0036] If a hand 104 and / or fingers 106 are detected in step 206, the procedure proceeds to step 208. In step 208, the distance d of the hand 104 and / or fingers 106 from the lighting device 110 is determined using the distance detection device 114. In one embodiment, the distance detection device can be formed by the evaluation control device 108, which determines the distance d of the hand 104 and / or fingers in three-dimensional space.

[0037] Furthermore, the evaluation control unit 108 can determine the average brightness I of the areas of the hand to be tracked on average 210. If the ratio of the distance of the hand d to the brightness I falls below a predetermined threshold, the intensity of the infrared illumination is increased. If the ratio of distance d to brightness I is higher than a predetermined threshold, the infrared illumination is reduced. To ensure the stability of the system and the detection, in step 208 the hand 104 is first illuminated with the maximum possible infrared illumination.

[0038] In step 212, the method uses the infrared illumination set in steps 208 and 210 to determine the position of hand 104 and / or fingers 106 at a second frame rate. This second frame rate is approximately 150 Hz to approximately 1000 Hz. While hand 104 and / or fingers 106 are tracked at this second frame rate, they are illuminated with the second illumination setting determined in steps 208 and 210.

[0039] In step 214, it is checked whether the position of a hand 104 and / or at least a finger 106 can be determined in step 212. If a position of the hand 104 and / or at least a finger 106 can be determined, the procedure returns to step 204 to begin again with the detection of the hand at the first illumination setting and the first frame rate.

[0040] If the position of hand 104 and / or at least one finger 106 can be detected in step 214, the procedure proceeds to step 216 and transmits the position of hand 104 and / or at least one finger 106 to a gesture evaluation device configured to detect gestures entered by the user. Furthermore, the procedure returns to step 208 to determine the current distance of hand 104 from the infrared illuminator 110.

[0041] The invention provides a method and a device for latency-free recognition of gestures of a vehicle occupant without the occupant being affected by lighting effects.

Claims

[1] Method for detecting a gesture of an occupant of a vehicle, comprising the following steps: - Capturing the interior of the vehicle using an imaging sensor system with an initial frame rate and an initial lighting setting to generate an initial series of images; - Searching for a hand (104) of the occupant in the first series of pictures; - if a hand (104) of the occupant is detected in the first series of images, capture the interior of the vehicle using the imaging sensor system with a second frame rate and a second lighting setting to produce a second series of images; - Evaluating the second series of images to identify the occupant's gesture; - Determining the distance of the occupant's hand (104) from a light source (110) that produces the second lighting setting; and - Adjusting the second lighting setting depending on the distance of the hand (104) from the light source (110) that produces the second lighting setting. [2] Method according to claim 1, wherein - the second refresh rate is higher than the first refresh rate and / or - the second lighting setting produces a stronger illumination of the occupant's fingers (106) than the first lighting setting. [3] A method according to any one of claims 1 to 2, comprising the following steps: - Determining the speed of the fingers (106) during the gesture; and - Adjusting the second frame rate depending on the speed of the occupant's fingers (106) during the gesture; and / or - Adjusting the second lighting setting depending on the speed of the occupant's fingers (106) during the gesture. [4] A method according to any one of claims 1 to 3, comprising the following steps: - Determining the skin color of the fingers (106) and / or clothing of the inmate; and - Adjusting the second lighting setting depending on the skin color of the fingers (106) and / or the clothing of the occupant. [5] Gesture detection device (100), comprising: - an imaging sensor system for detecting the interior of a vehicle, comprising an image acquisition sensor (112) with a variable frame rate and a lighting device (110); - an evaluation control unit (108) which is designed to - to control the imaging sensor system for capturing the interior of the vehicle in such a way that the imaging sensor system generates an initial series of images with an initial frame rate and an initial lighting setting; - to look for a hand (104) of an inmate in the first series of images, - if the occupant's hand (104) is detected, to control the imaging sensor system in such a way that the interior of the vehicle is captured by the imaging sensor system with a second frame rate and a second lighting setting in order to generate a second series of images; and - to evaluate the second series of images in order to identify a gesture by the occupant; - a distance sensing device (114) designed to determine the distance of the occupant's hand (104) from the light source (110) that produces the second lighting setting, - wherein the evaluation control device (108) is configured to control the imaging sensor system in such a way that the second illumination setting is varied depending on the distance of the hand (104) from the light source (110) that generates the second illumination setting. [6] Gesture detection device (100) according to claim 5, wherein -the second refresh rate is higher than the first refresh rate and / or -the second lighting setting produces a stronger illumination of the occupant's fingers (106) than the first lighting setting. [7] Gesture detection device (100) according to one of claims 5 to 6, wherein the evaluation control device (108) is configured to - to determine the speed of the fingers (106) during the gesture; and - to determine the second frame rate as a function of the speed of the occupant's fingers (106) during the gesture and / or the second lighting setting as a function of the speed of the occupant's fingers (106) during the gesture. [8] Motor vehicle with a gesture recognition device (100) according to one of claims 5 to 7.

Images