Gesture control method for light of whole vehicle based on combination of various sensors

Abstract

The invention relates to a method for clearing and controlling light of a whole vehicle based on combination of multiple sensors, which comprises a processing unit, ToF sensors, an infrared camera, an IMU (Inertial Measurement Unit) sensor, a first ToF sensor, a second ToF sensor and a rear ToF sensor, identifying a key point by the depth image, and capturing a gesture space position and a motion speed according to the key point; the infrared camera is adopted to capture gesture styles and patterns, the first ToF sensor, the second ToF sensor and the rear ToF sensor divide the space in the vehicle body into at least two recognition areas, the multiple sets of ToF sensors are combined with the infrared camera and the IMU sensor to obtain high-precision and high-reliability gesture command recognition, the precise characteristic of gesture recognition is greatly optimized, and the recognition accuracy of the vehicle body is improved. And the feasibility of gesture operation is ensured. And the vehicle state is linked to divide the recognition area in the vehicle, so that the risk of gesture command false triggering is effectively reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automotive electronics, in particular to a gesture control method for vehicle lights that combines multiple sensors and vehicle state information. BACKGROUND

[0002] Existing vehicle intelligent systems rely on single sensor data, such as a single infrared array or infrared camera. Single sensors are susceptible to environmental light and vehicle vibration interference. During vehicle operation, vibration can cause image blurring and command recognition errors. On the other hand, existing control methods cannot distinguish between operators, and false triggering of commands can cause The technical solution adopted by the present application is a gesture control method for vehicle lights based on the combination of multiple sensors, including a processing unit and ToF sensors, infrared cameras, and IMU sensors mounted on the vehicle body. The ToF sensors include first and second ToF sensors located on both sides of the front end of the detection area and a rear ToF sensor located at the rear end of the detection area. The gesture position is confirmed by the multiple ToF sensors to generate a high-precision depth image. The key points are identified in the depth image, and the gesture spatial position and motion speed are captured based on the key points. The infrared camera captures the gesture pattern, The processing unit combines the captured gesture pattern and high-precision depth image to form a command pattern. The processing unit confirms the gesture command by comparing the command pattern with the gesture command database. The gesture command database includes gesture command images and execution codes, and the execution codes have priority. The processing unit executes the command based on the execution code and IMU sensor data. The first, second, and rear ToF sensors divide the vehicle body space into at least two recognition areas. The processing unit sets different permissions based on the command occurrence location area.

[0003] As one of the preferred ways of the gesture control method for vehicle lights based on the combination of multiple sensors, it also includes a space modeling processing unit that combines the first, second, and rear ToF sensors to form a high-precision depth image of the infrared camera's field of view based on gesture spatial points.

[0004] As one of the preferred ways of the gesture control method of the whole vehicle light based on the combination of multiple sensors, the recognition area is divided into a main driver recognition area and other recognition areas, the gesture command database includes a main driver database and other databases, the gesture in the main driver recognition area can call the main driver database and other databases, and the gesture in the other recognition area can only control the other database.

[0005] As one of the preferred ways of the gesture control method of the whole vehicle light based on the combination of multiple sensors, gesture entry is further included, the first ToF sensor, the second ToF sensor, and the rear ToF sensor capture images for gesture entry in combination with the infrared camera, and the user sets a custom gesture command through a set command.

[0006] As one of the preferred ways of the gesture control method of the whole vehicle light based on the combination of multiple sensors, the rear ToF sensor is divided into two groups at the rear end of the vehicle body and a side ToF sensor located on the side of the vehicle body.

[0007] As one of the preferred ways of the gesture control method of the whole vehicle light based on the combination of multiple sensors, the processing unit adopts a Kalman filter algorithm to perform data filtering on the data of at least three ToF sensors and the infrared camera.

[0008] As one of the preferred ways of the gesture control method of the whole vehicle light based on the combination of multiple sensors, the ToF sensor data is solved by a least square method to obtain an optimal three-dimensional coordinate input to the processing unit, and the ToF sensor data is tracked in a three-dimensional space by a Kalman filter and an example filter algorithm.

[0009] As one of the preferred ways of the gesture control method of the whole vehicle light based on the combination of multiple sensors, gesture command confirmation is further included, and the processing unit has a command confirmation step before executing the code in combination with the IMU sensor data, and the command confirmation is performed through a display device and a button.

[0010] The present application has the advantages that: multiple groups of ToF sensors are combined with an infrared camera and an IMU sensor to obtain high-precision and high-reliability gesture command recognition, which greatly optimizes the precision characteristics of gesture recognition and ensures the feasibility of gesture operation. The vehicle state is linked to divide the recognition area in the vehicle, effectively reducing the risk of false triggering of gesture commands. The present application supports original vehicle line modification, which can be simply installed by adding a ToF module. The ToF depth map and the infrared contour map are combined to construct a more robust gesture interaction system, and the robustness and accuracy of gesture feature extraction are improved. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 Layout diagram of Tof sensor.

[0012] Figure 2 The data processing flow chart is shown in Figure 2.

[0013] Figure 3 To collect logic diagram. Specific implementation plan The present invention will be further described below with reference to the accompanying drawings.

[0014] like Figures 1 to 3 The figure shows a gesture-based vehicle lighting control method based on a combination of multiple sensors. The method includes a processing unit, a ToF sensor, an infrared camera, and an IMU sensor mounted on the vehicle. The ToF sensors include a first ToF sensor and a second ToF sensor located on either side of the front of the detection area, as well as a rear ToF sensor located at the rear of the detection area. These sensors confirm the position of gestures and generate a high-precision depth image. The depth image identifies key points and captures the spatial position and speed of the gesture based on these key points. The ToF sensor can read distance and direction. It calculates distance by measuring the time it takes for photons to be emitted and reflected by an object, thereby generating a high-resolution depth map. Multiple ToF sensors are formed into an array to cover the spatial range to be monitored. Each sensor provides depth information within its field of view. The sensor independently measures the distance to the hand and, based on the positional relationship between the sensors, reconstructs three-dimensional coordinates from multiple perspectives. The distance information from multiple sensors is combined to calculate the hand's position. As the hand moves, information from each sensor is collected to determine the hand's trajectory. The infrared camera is used to identify the gesture's contour, and the IMU sensor filters vehicle vibration. The ambiguity of single-point distance measurement is eliminated by multi-view measurement. After obtaining the distance parameter, each ToF sensor forms a spherical equation with the sensor as the sphere and the distance as the radius. The intersection of multiple ToF sensors is the received three-dimensional coordinates. An infrared camera is used to capture gesture patterns. The hand reflects infrared light, and the camera uses an infrared filter to block other light, receiving only the reflected infrared signal. This generates a "black and white heat map." The camera then takes an infrared photo in real time, removes background light, enhances the hand outline, and outlines the finger shapes. After extracting the outline features, static and dynamic gestures are classified using support vector machines (SVMs), random forests, or dynamic time warping (DTW). Multi-target tracking algorithms such as SORT and DeepSort are used to distinguish different hand targets.

[0015] The processing unit combines the captured gesture pattern and the high-precision depth image to form a command pattern. The processing unit confirms the gesture command by comparing the command pattern with a gesture command database. The gesture command database includes gesture command images and execution codes, and the execution codes have priority. The processing unit executes the command based on the execution code and IMU sensor data. The first, second, and rear ToF sensors divide the vehicle interior into at least two recognition areas, and the processing unit sets different permissions based on the command location area. The processing unit obtains vehicle status via the vehicle CAN bus and dynamically enables or disables specific gesture functions.

[0016] As one of the preferred methods of controlling vehicle lighting through gestures based on a combination of multiple sensors, the method also includes a spatial modeling processing unit, which combines the first ToF sensor, the second ToF sensor, and the rear ToF sensor to form a high-precision depth image of the gesture spatial point output from the perspective of the infrared camera.

[0017] As one of the preferred methods of controlling vehicle lighting through gestures based on the combination of multiple sensors, the recognition area is divided into a main driver recognition area and other recognition areas. The gesture command database includes a main driver database and other databases. Gestures in the main driver recognition area can call the main driver database and other databases, while gestures in other recognition areas can only control other databases.

[0018] As one of the preferred methods of controlling vehicle lighting through gestures based on a combination of multiple sensors, it also includes gesture entry. The first ToF sensor, the second ToF sensor, and the rear ToF sensor are combined with the infrared camera to capture images for gesture entry. The user sets custom gesture commands through setting commands.

[0019] As one of the preferred methods of gesture control of vehicle lighting based on the combination of multiple sensors, the rear ToF sensor is composed of two groups of discrete rear end vehicles and side TOF sensors located on the sides of the vehicle body.

[0020] As one of the preferred methods for controlling vehicle lighting through gestures based on a combination of multiple sensors, the processing unit uses a Kalman filter algorithm to filter the data of at least three ToF sensors. Kalman filtering is used to eliminate ambient light interference and sensor noise, ensuring that the time points at which each sensor collects data are consistent, increasing sensor density or adopting a three-dimensional layout to ensure that at least three sensors can see the target at the same time. Historical trajectories are used to predict movement during occlusion. Errors caused by the superposition of reflected signals are suppressed by modulating the light frequency or algorithm. Position offsets between sensors are corrected in real time. As one of the preferred ways of the gesture control method of the whole vehicle light based on the combination of multiple sensors, the ToF sensor data is solved by the least square method to obtain the optimal three-dimensional coordinate input to the processing unit, and the ToF sensor data is tracked in the three-dimensional space position by the Kalman filter and the example filter algorithm.

[0021] As one of the preferred ways of the gesture control method of the whole vehicle light based on the combination of multiple sensors, gesture command confirmation is also included, and the processing unit has a command confirmation step before executing the code combined with the IMU sensor data to execute the command, and the command confirmation is performed through the display device and the button.

Claims

1. A gesture-based method for controlling vehicle lighting based on a combination of multiple sensors includes a processing unit, a ToF sensor, an infrared camera, and an IMU sensor mounted on the vehicle body, and is characterized by: The ToF sensor includes a first ToF sensor and a second ToF sensor located on both sides of the front end of the detection area, and a rear ToF sensor located at the rear end of the detection area. The plurality of ToF sensors are used to confirm the gesture position and generate a high-precision depth image. The depth image identifies key points and captures the spatial position and movement speed of the gesture based on the key points. Using infrared camera to capture gesture patterns, The processing unit superimposes the captured gesture pattern and the high-precision depth image to form a command pattern; The processing unit confirms the gesture command by comparing the command pattern with the gesture command database; The gesture command database includes gesture command images and execution codes, and the execution codes have priorities; The processing unit executes the command according to the execution code in combination with the IMU sensor data; The first ToF sensor, the second ToF sensor and the rear ToF sensor divide the space inside the vehicle into at least two recognition areas. The processing unit sets different permissions according to the command generation location area.

2. The method for controlling vehicle lighting through gestures based on a combination of multiple sensors according to claim 1, characterized in that: It also includes a space modeling processing unit, which combines the first ToF sensor, the second ToF sensor, and the rear ToF sensor to form a high-precision depth image of the gesture space point output from the perspective of the infrared camera.

3. The method for controlling vehicle lighting through gestures based on a combination of multiple sensors according to claim 1, characterized in that: The recognition area is divided into a main driver recognition area and other recognition areas. The gesture command database includes a main driver database and other databases. Gestures generated in the main driver recognition area can call the main driver database and other databases, and gestures generated in other recognition areas can only control other databases.

4. The method for controlling vehicle lighting through gestures based on a combination of multiple sensors according to claim 1, characterized in that: It also includes gesture recording. The first ToF sensor, the second ToF sensor, and the rear ToF sensor are combined with the infrared camera to capture images for gesture recording. Users set custom gesture commands by setting commands.

5. The method for controlling vehicle lighting by gestures based on a combination of multiple sensors according to claim 1, characterized in that: The rear ToF sensors are two separate groups at the rear end of the vehicle body, and the side TOF sensors are located on the side of the vehicle body.

6. The method for controlling vehicle lighting by gestures based on a combination of multiple sensors according to claim 5, characterized in that: The processing unit performs data filtering on at least three ToF sensor data using a Kalman filter algorithm.

7. The method for controlling vehicle lighting through gestures based on a combination of multiple sensors according to claim 6, characterized in that: The ToF sensor data is solved by the least square method to obtain the optimal three-dimensional coordinates which are input to the processing unit, and the ToF sensor data is tracked in three-dimensional space by the Kalman filter and the example filter algorithm.

8. The method for controlling vehicle lighting by gestures based on a combination of multiple sensors according to claim 7, characterized in that: It also includes gesture command confirmation. The processing unit also has a command confirmation step before executing the command according to the execution code combined with the IMU sensor data, and the command confirmation is performed through the display device and buttons.

9. The method for controlling vehicle lighting through gestures based on a combination of multiple sensors according to claim 8, wherein the display device is a HUD, and the HUD displays the recognition gesture form and the execution command style.

Images