Gesture detection system

Abstract

To provide a gesture detection system capable of detecting gestures with improved accuracy while considering hand shaking of a user resulting from vibration imparted by a vehicle.SOLUTION: A gesture detection system is mounted on a vehicle and configured to detect a hand gesture operation of a user of a vehicle. The gesture detection system comprises: a sensor detecting vibration of the vehicle; a camera imaging the gesture operation; and a controller controlling the camera. The controller obtains first vibration, being vibration of the vehicle, from the sensor, estimates second vibration, being vibration transmitted to the user's hand from the first vibration, corrects video captured by the camera according to the second vibration, and detects the gesture operation on the basis of the corrected video.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a gesture detection system that is mounted on a vehicle and detects a gesture operation of a hand of a vehicle user. [Background technology]

[0002] Conventionally, a gesture detection system that detects hand gestures of a vehicle user has been known (see, for example, Patent Document 1). Such a gesture detection system is used in a system that operates a vehicle device in response to a user's hand gestures.

[0003] In a gesture detection system, a camera is generally used to capture an image of a user's hand gesture to detect the gesture. When such a camera is mounted on a vehicle, the camera is subjected to vibrations from the vehicle. A technique using a vibration sensor attached to the vehicle to correct image blur caused by the vibrations received by the camera is also known (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-166058 [Patent Document 2] Japanese Patent Application Publication No. 11-331681 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in a gesture detection system, the user himself / herself is subjected to vibrations from the vehicle in addition to the camera, so that, for example, a gesture made by the user's hand may be shaken by the vibrations from the vehicle.

[0006] An object of the present disclosure is to provide a gesture detection system that can detect gestures with higher accuracy by taking into account shaking of the user's hands due to vibrations received from a vehicle. [Means for solving the problem]

[0007] A gesture detection system according to the present disclosure is mounted on a vehicle and detects a gesture operation by a hand of a user of the vehicle. The gesture detection system includes a sensor that detects vibrations of the vehicle, a camera that captures images of the gesture operation, and a control device that controls the camera, wherein the control device acquires a first vibration that is a vibration of the vehicle from the sensor, estimates a second vibration that is a vibration transmitted to the user's hand from the first vibration, corrects an image captured by the camera based on the second vibration, and detects the gesture operation based on the corrected image. [Effects of the Invention]

[0008] This gesture detection system can estimate the second vibration received by the user's hand from a sensor that detects vehicle vibrations. The control device can detect gesture operations more accurately by correcting the image captured by the camera based on the second vibration. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a system diagram of a gesture detection system according to one embodiment of the present disclosure. [Figure 2] 4 is a flowchart showing a control procedure executed by a control device according to an embodiment of the present disclosure. [Figure 3] FIG. 10 illustrates a second vibration according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0011] 1, the gesture detection system 1 includes a sensor 2, a camera 4, a control device 6, an in-vehicle device 8, and slip detection means 10. The gesture detection system 1 of this embodiment is a system that is mounted on a vehicle C and detects gesture operations of a hand H of a user U seated on a seat S of the vehicle C.

[0012] The sensor 2 detects vibrations received by the vehicle C. Specifically, the sensor 2 is attached to the body of the vehicle C. The sensor 2 detects vibrations received by the vehicle C from the road surface and transmitted to the body, and calculates the detected vibrations of the vehicle C inside the sensor 2 to output the magnitude (amplitude), direction, and period (frequency) of the detected vibrations as a first vibration. The sensor 2 is, for example, a vibration sensor or an acceleration sensor. The sensor 2 is electrically connected to the control device 6 and transmits the first vibration to the control device 6.

[0013] The camera 4 captures images of the movements of the user U of the vehicle C. In this embodiment, the camera 4 captures images of gesture operations of the hand H of the user U of the vehicle C. In this embodiment, the camera 4 is fixed to, for example, an instrument panel (not shown) located at the front of the interior of the vehicle C. However, the camera 4 may be fixed to any position as long as it is fixed to a position where the hand of the user U can be captured.

[0014] The camera 4 includes a lens, an image sensor, a video processing circuit, a correction circuit, and a video output circuit. In this embodiment, the camera 4 is a CCD camera (CCD stands for Charge Coupled Devices) that uses a charge-coupled device as the semiconductor of the image sensor. When the camera 4 captures an image of the gesture of the user U's hand H, the image processing circuit converts the image into a digital video signal and sends it to the correction circuit. The correction circuit corrects this video signal and sends it to the video output circuit. The camera 4 is electrically connected to the control device 6, and sends the image from the video output circuit to the control device 6.

[0015] The control device 6 receives an output signal from the sensor 2 and controls the camera 4. The control device 6 is also capable of controlling an in-vehicle device 8 of the vehicle C and a power source (not shown) of the vehicle C. The control device 6 acquires the first vibration and can estimate the second vibration, which is the vibration of the hand H of the user U. Specifically, the control device 6 pre-stores a second vibration estimation program capable of estimating the second vibration. The second vibration estimation program is a program including an arithmetic expression for estimating the vibration of the hand H when the first vibration is transmitted to the hand H via the buttocks of the user U seated on the seat S. The second vibration estimation program of this embodiment is a program for calculating the second vibration that reflects the shift in the period and the shift in amplitude of the hand H when the first vibration is transmitted to the buttocks of the user U. When the control device 6 acquires the first vibration, it performs calculations using the second vibration estimation program to estimate the second vibration.

[0016] The control device 6 is actually an ECU (Electronic Control Unit) and is configured by a microcomputer including an arithmetic unit, memory, input / output buffers, etc. The control device 6 controls the various devices based on signals from the sensors and various devices, as well as maps and programs stored in the memory, so that the vehicle C, the camera 4, and the on-board device 8 are in a desired operating state.

[0017] The in-vehicle device 8 is an electronic device mounted on a vehicle C and operable by a user U of the vehicle C. In this embodiment, the in-vehicle device 8 is, for example, an air conditioner or an audio device. However, the in-vehicle device 8 may be another device operable by the user U. The in-vehicle device 8 is electrically connected to the control device 6 and is controlled by the control device 6 when the control device 6 detects a gesture operation of the user U's hand H. In one example, if the in-vehicle device 8 is an air conditioner, when the user U performs a first predetermined gesture operation with the hand H, the control device 6 detects it and changes the set temperature of the air conditioner. In another example, if the in-vehicle device 8 is an audio device, when the user U performs a second predetermined gesture operation with the hand H, the control device 6 detects it and changes the volume of the audio. In this embodiment, the in-vehicle device 8 further includes a display. The display may display the detection result of the gesture operation, a warning regarding the suspension of the detection of the gesture operation, and the like.

[0018] The slip detection means 10 is a device that detects skidding of the wheels W of the vehicle C. The slip detection means may be a brake sensor that detects the operating state of an anti-block brake, a wheel speed sensor that detects the rotational speed of the wheels W of the vehicle C, or the like. Alternatively, if the sensor 2 is an acceleration sensor, the sensor 2 may be used to detect slippage of the vehicle C.

[0019] Next, the control procedure executed by the control device 6 will be described with reference to FIG.

[0020] In step S1, the control device 6 starts gesture detection. For example, the control device 6 may start gesture detection at the same time as an ignition switch (not shown) is turned on. Alternatively, the control device 6 may start gesture detection at the same time as a gesture detection button (not shown) is turned on. After starting gesture detection, the control device 6 proceeds to step S2.

[0021] In step S2, the control device 6 acquires a first vibration from the sensor 2. Upon acquiring the first vibration, the control device 6 advances the process to step S3.

[0022] In step S3, the control device 6 estimates the amplitude, period (frequency), and direction of the second vibration using the second vibration estimation program. Once the control device 6 has estimated the second vibration, the process proceeds to step S4.

[0023] In step S4, the control device 6 determines whether the second vibration is outside the allowable range. The graph shown by the solid line in FIG. 3 illustrates a case where the second vibration occurs in the vertical direction and has an amplitude and frequency outside the allowable range. With such an amplitude and frequency, the control device 6 can estimate that the speed of the second vibration is low at points A and B where the amplitude is maximum. Therefore, the control device 6 can estimate that the shaking of the hand is also small. On the other hand, at point C between points A and B, the control device 6 can estimate that the speed of the second vibration is high. Therefore, the control device 6 can estimate that the shaking of the hand is also large. If the control device 6 determines that the second vibration is outside the allowable range (step S4: YES), the control device 6 proceeds to step S5.

[0024] In step S5, the control device 6 corrects the image. Specifically, the control device 6 performs correction to generate an image by extracting images near points A and B of the second vibration estimated in step S3 from among the images captured by the camera 4 a predetermined number of times per unit time (hereinafter referred to as a predetermined frame rate) and deleting images near point C. In this embodiment, the control device 6 transmits to the camera 4 the time periods near points A and B of the second vibration, and causes the correction circuit of the camera 4 to generate an image in which the number of images captured per unit time (frame rate) other than near points A and B is reduced to less than the predetermined frame rate. This allows the camera 4 to generate an image with fewer images of areas where the hand shake is large. As a result, the control device 6 can acquire an image with less shaking of the hand H based on the second vibration. After correcting the image, the control device 6 proceeds to step S6.

[0025] In step S6, the control device 6 detects a gesture operation based on the corrected image. As described above, the control device 6 can acquire an image with little shaking of the hand H, which makes it easier to accurately detect a gesture operation of the hand H of the user U. When the control device 6 detects a gesture operation, the process proceeds to step S7.

[0026] In step S4, if the control device 6 determines that the second vibration is within the allowable range (NO in step S4), the control device 6 proceeds to step S6 and detects the gesture operation without correcting the image. The graph shown by the dashed dotted line in FIG. 3 is a graph when the amplitude and frequency of the second vibration are within the allowable range. With such amplitude and frequency, the shaking of the hand is small. Therefore, the control device 6 can accurately detect the gesture operation of the hand H of the user U.

[0027] In step S7, the control device 6 determines whether the second vibration is equal to or greater than the upper detection limit. The upper detection limit is at least one of the amplitude of the second vibration, which is the upper detection limit for a gesture operation (hereinafter referred to as the upper detection limit amplitude), and the frequency of the second vibration, which is the upper detection limit for a gesture operation (hereinafter referred to as the upper detection limit frequency). The dashed line in the graph of FIG. 3 indicates a case where the second vibration exceeds the upper detection limit amplitude and the upper detection limit frequency. With such amplitude and frequency, the user U may be swaying up and down to the extent that it is dangerous for the user U to perform a gesture operation. If the control device 6 determines that the second vibration is equal to or greater than the upper detection limit (YES in step S7), the control device 6 proceeds to step S8.

[0028] In step S8, the control device 6 acquires the first vibration again and determines whether the first vibration is outside the allowable range. Here, the allowable range of the first vibration may be, for example, an amplitude and frequency that does not cause the second vibration to exceed the upper detection limit in the second vibration estimation program. As described above, even if the second vibration is equal to or greater than the upper detection limit, if the second vibration exceeds the upper detection limit due to a sudden vibration, such as when the vehicle C drives over a curb, the actual shaking of the user U's hand H may quickly converge to below the upper detection limit. Therefore, the control device 6 proceeds to step S9 only if the first vibration is outside the allowable range in step S8 (YES in step S8).

[0029] In step S9, the control device 6 stops gesture detection and proceeds to step S10.

[0030] In step S10, the control device 6 displays a warning on the display to the effect that gesture detection has been stopped, and ends the process. In this way, the control device 6 stops gesture detection, thereby allowing the user U to stop the gesture operation. This improves the safety of the vehicle C.

[0031] In step S7, if the control device 6 determines that the second vibration is less than the detection upper limit value (NO in step S7), the control device 6 advances the process to step S12.

[0032] In step S8, if the control device 6 determines that the first vibration is within the allowable range (NO in step S8), the control device 6 advances the process to step S11.

[0033] In step S11, the control device 6 determines whether or not a slip has been detected by the slip detection means 10. In this embodiment, if the sensor 2 detects the first vibration not only in the up-down direction but also in the left-right direction (the width direction of the vehicle C), the control device 6 determines that a slip of the vehicle C has been detected. If the control device 6 determines that a slip has been detected (YES in step S11), the control device 6 proceeds to step S9 and stops gesture detection. On the other hand, if the control device 6 determines that a slip has not been detected (NO in step S11), the control device 6 proceeds to step S12.

[0034] In step S12, the control device 6 continues to detect the gesture operation, and proceeds to step S13.

[0035] In step S13, the control device 6 determines whether or not the detection of the gesture operation has been completed. For example, as described above, the control device 6 may determine that the detection of the gesture operation has been completed when it determines that the gesture operation of the hand H of the user U is the first predetermined gesture operation or the second predetermined gesture operation. When the control device 6 determines that the detection of the gesture operation has been completed (YES in step S13), the control device 6 proceeds to step S14.

[0036] In step S14, the control device 6 operates the in-vehicle device 8 based on the gesture operation and displays the detection result on the display. As described above, in one example, if the in-vehicle device 8 is an air conditioner, when the user U performs a first predetermined gesture operation with his / her hand H, the control device 6 detects it and changes the set temperature of the air conditioner. In another example, if the in-vehicle device 8 is an audio device, when the user U performs a second predetermined gesture operation with his / her hand H, the control device 6 detects it and changes the volume of the audio. The control device 6 displays these processes on the display. After executing these processes, the control device 6 ends the process.

[0037] In step S13, if the control device 6 determines that the detection of the gesture operation has not been completed (step S13 NO), the control device 6 proceeds to the process of step S2 and continues the gesture detection.

[0038] As described above, according to the present disclosure, it is possible to provide a gesture detection system that takes into account shaking of the user's hand due to vibrations received from the vehicle C and can detect gestures with higher accuracy.

[0039] <Other embodiments> Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above embodiments, and various modifications are possible within the scope of the gist of the invention. In particular, the multiple modifications described in this specification can be arbitrarily combined as necessary.

[0040] (a) In the above embodiment, the sensor 2 is described as a vibration sensor and an acceleration sensor fixed to the vehicle C, but the present disclosure is not limited to this. The sensor 2 may be any sensor that can detect the first vibration.

[0041] (b) In the above embodiment, an air conditioner and an audio system are used as examples of the in-vehicle device 8, but the present disclosure is not limited to this. The in-vehicle device 8 may also be a navigation system or a cruise control system for the vehicle C. The gesture detection system 1 may be any system that enables the operation of the in-vehicle device 8 by gesture operation. [Explanation of symbols]

[0042] 1: Gesture detection system 2: Sensor, 4: Camera, 6: Control device, 8: In-vehicle device C: vehicle, H: hand, U: user

Claims

1. A gesture detection system that is mounted on a vehicle and detects a hand gesture operation of a user of the vehicle, a sensor that detects vibrations of the vehicle; a camera that captures an image of the gesture operation; a control device for controlling the camera; Equipped with The control device acquiring a first vibration, which is a vibration of the vehicle, from the sensor; estimating a second vibration that is a vibration transmitted to the user's hand from the first vibration; correcting the image captured by the camera based on the second vibration; Detecting the gesture operation based on the corrected image. Gesture detection system.

2. the control device stops the image capture when the second vibration is equal to or greater than an upper detection limit for the gesture operation. The gesture detection system of claim 1 .

3. the detection upper limit value is at least one of an amplitude of the second vibration that is an upper limit for detecting the gesture operation and an upper limit of a frequency of the second vibration that is an upper limit for detecting the gesture operation. The gesture detection system of claim 2 .

4. The control device, even when the second vibration exceeds the detection upper limit value, If the first vibration is within a tolerable range, the detection of the gesture operation is continued. The gesture detection system of claim 2 .

5. The vehicle further includes a slip detection means for detecting slippage of the vehicle, When the control device detects a slip of the vehicle, Stopping the detection of the gesture operation. The gesture detection system according to any one of claims 1 to 4.

Images