A DLP-based intelligent interactive unlocking system and method

Abstract

A DLP-based intelligent interactive unlocking system and method, including power supply, DLP projection, information collection, preprocessing, decision fusion and execution unit. When the system is turned on, the DLP projection unit projects the unlocking interface onto the ground, and the user performs interactive operations by stepping on the unlocking interface. The information collection unit is responsible for collecting the user's foot information and transmitting it to the controller. The controller determines the interactive information corresponding to the user's foot stepping action based on the collected information after a series of processes such as signal preprocessing, data fusion, feature extraction, classification recognition, and control decision-making, outputs the corresponding instructions, and executes the vehicle unlocking operation. The present invention is a fusion detection method based on two sensors, millimeter-wave radar and infrared camera, and is applied to projection interaction in a vehicle-mounted environment. The millimeter-wave radar can accurately obtain the speed information of the foot, and the camera can provide accurate object category information. The two sensors are fused and complementary, which greatly improves the accuracy of detection.

Description

Technical Field

[0001] The present invention belongs to the field of projection interaction technology, and in particular to a vehicle-mounted projection interaction component and a human-vehicle interaction technology. Background Art

[0002] With the development of vehicle intelligence and interactive multimedia technology, people have more and more demands for human-vehicle interaction, and their requirements are getting higher and higher. In many cases, cars are no longer just a means of transportation. Their functions are intelligent and diversified, and they can even become a mobile office or entertainment room, thus solving the needs of different groups of people for cars and adapting to the requirements of modern people for high-quality, convenient and intelligent life.

[0003] In recent years, in-vehicle infotainment and information communication systems are gradually becoming hot spots pursued by automobile manufacturers, and more and more intelligent operations are being applied in cars. However, the existing vehicles and intelligent systems are not enough to support people's specific needs, the interaction between people and cars is poor, and the vehicle unlocking is not intelligent enough. For example, the owner forgets or loses the car key, or the phone is turned off and the APP cannot be used to open the car door. In addition, people are no longer satisfied with the human-vehicle interaction achieved by these traditional methods, but need a more natural and entertaining way of interaction.

[0004] Therefore, how to solve the problems of single automobile functionality, poor interactive performance, poor customer experience and insufficient novelty of interaction in the prior art has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a DLP-based intelligent interactive unlocking system and method.

[0006] The technical solution of the present invention: an intelligent interactive unlocking system and method based on DLP (DLP is the abbreviation of "Digital Light Processing"). The system mainly includes a power supply unit, a DLP projection unit, an information acquisition unit, a preprocessing unit, a decision fusion unit and an execution unit. After the system is turned on, the DLP projection unit projects the unlocking interface onto the ground, and the user can step on the unlocking interface to perform interactive unlocking. The information acquisition unit is responsible for collecting the user's foot information, including category, speed, position, etc., transmitting it to the decision fusion unit, and outputting the corresponding instructions after comparing it with the coordinates of the corresponding projection area, and handing it over to the execution unit to perform the vehicle unlocking operation.

[0007] The power supply unit is responsible for the power supply of the entire system and can diagnose information such as short circuit and disconnection of the power supply of the external device, communication failure, etc., thereby playing a protective role.

[0008] The DLP projection unit has a self-correction function to ensure that the projection effect is not distorted.

[0009] The information collection unit is composed of a short-range millimeter-wave radar and an infrared camera, which is installed on the main driver's side and is responsible for collecting user interaction information. The short-range radar is more suitable for the detection scenario of this system. The use of the infrared camera filters out the visible light information of the image, solves the interference of the projector light, provides the possibility for the system's image processing process, and also realizes the camera's night detection function.

[0010] The preprocessing unit and decision fusion unit are both controller components. The components communicate through CAN. After the sensor obtains the foot data, the controller performs signal preprocessing, data fusion, feature extraction, classification recognition and control decision.

[0011] A DLP-based intelligent interactive unlocking method, the method comprising:

[0012] After the DLP projection unit self-calibrates, it accurately projects the unlocking interface onto the ground in the corresponding projection area.

[0013] The user interacts with the vehicle by stepping on it. The signal acquisition unit transmits the detected user foot data to the preprocessing unit for analysis and filtering, and then transmits it to the decision fusion unit.

[0014] After obtaining the millimeter-wave radar data and image data, the decision fusion unit performs spatiotemporal fusion and feature fusion to complete the coordinate system conversion and synchronous alignment of the two sensors in the time dimension, as well as obtain feature information and extract effective targets.

[0015] Set the speed threshold v τ The foot position coordinates at that time are compared with the corresponding projection area coordinates of the unlocking interface to obtain the interaction result, and the execution unit performs the unlocking operation.

[0016] Beneficial effects of the present invention: The technical solution provided by the present invention can project the unlocking interface onto the ground by projection, and the user can achieve human-vehicle interaction by stepping on the corresponding projection area. The system uses visual fusion radar technology, and after a series of processes including signal preprocessing, data fusion, feature extraction, classification recognition, and control decision-making, it accurately determines the interactive information corresponding to the user's foot-stepping action, outputs the corresponding instructions, and executes the vehicle unlocking operation. It solves the technical problems in the prior art that the automobile has a single functionality, poor interactive performance, poor customer experience, and insufficient novelty of interaction, greatly improves the functionality, interactivity, and entertainment of the automobile, and greatly improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is the overall block diagram of the system of the present invention;

[0018] Figure 2 It is an overall schematic diagram of the system of the present invention;

[0019] Figure 3 is an effective filtering flow chart of the preprocessing unit of the present invention;

[0020] Figure 4 It is a schematic diagram of the conversion between the radar coordinate system and the world coordinate system of the present invention;

[0021] Figure 5 It is a schematic diagram of the conversion between the camera coordinate system and the world coordinate system of the present invention;

[0022] Figure 6 It is a schematic diagram of the conversion between the pixel coordinate system and the image coordinate system of the present invention;

[0023] Figure 7 It is a schematic diagram of the time fusion of the fusion decision unit of the present invention. DETAILED DESCRIPTION

[0024] Below we describe in detail a DLP-based intelligent interactive unlocking system of the present invention in combination with actual product principles and drawings.

[0025] like Figure 1 As shown, a DLP-based intelligent interactive unlocking system of the present invention includes a power supply unit, a DLP projection unit, an information collection unit, a preprocessing unit, a decision fusion unit and an execution unit. The power supply unit is responsible for the power supply and circuit protection of the entire system. The DLP projection unit has a self-correction function and is responsible for projecting the unlocking interface. The information collection unit is composed of a 4D millimeter-wave radar and an infrared camera, which is installed on the main driving side and is responsible for collecting the user's interactive information. The preprocessing unit and the decision fusion unit are both controller components. After the sensor obtains the foot data, the controller performs signal preprocessing, data fusion, feature extraction, classification recognition and control decision-making. After receiving the corresponding instructions, the execution unit is responsible for executing the vehicle unlocking operation.

[0026] like Figure 2 As shown in the figure, after the DLP projection unit self-calibrates, it accurately projects the unlocking interface onto the ground in the corresponding projection area. The user interacts with the vehicle by stepping on it, and the signal acquisition unit transmits the detected user foot data to the preprocessing unit for analysis and filtering, and then transmits it to the decision fusion unit. After obtaining the millimeter-wave radar data and image data, the decision fusion unit performs spatiotemporal fusion and feature fusion, completes the coordinate system conversion and synchronous alignment of the two sensors in the time dimension, and obtains feature information and extracts effective targets. Finally, the foot position coordinates are compared with the coordinates of the corresponding projection area of ​​the unlocking interface to obtain the interaction result, and the execution unit performs the unlocking operation.

[0027] like Figure 3 As shown in the figure, the preprocessing unit sets the millimeter wave radar effective target data filtering conditions according to the characteristics of the 4D millimeter wave radar product and the situations encountered in actual use, and provides reasonable judgment rules for the screening of effective targets of the millimeter wave radar. The filtering program will preprocess the data in each radar cycle according to the judgment rules to eliminate interference data.

[0028] The spatial fusion of the fusion decision unit of the present invention is mainly completed by the coordinate transformation of the two sensor data, including the transformation between the radar coordinate system and the world coordinate system, the transformation between the camera coordinate system and the world coordinate system, the transformation between the camera coordinate system and the image coordinate system, and the transformation between the pixel coordinate system and the image coordinate system.

[0029] The first step is the conversion between the radar coordinate system and the world coordinate system, such as Figure 4 As shown, assuming that the millimeter wave radar position is O r , the millimeter wave radar beam emission surface is X r O r Y r , the direction of the millimeter wave radar transmitting beam is Z r , the coordinate axis is perpendicular to the millimeter wave radar, Y r Direction perpendicular to the ground and upward, X r Direction perpendicular to the right, O r -X r Y r Z r Determine the millimeter wave radar coordinate system. w -X wr Y w Z w The world coordinate system is determined, and its three coordinate planes are parallel to the above radar coordinate system, and the coordinate axes point in the same direction. The distance between the radar coordinate system plane and the world coordinate system plane is H, the direction perpendicular to the radar surface is set to 0 degrees, the clockwise angle is positive, and the counterclockwise angle is negative. Assuming that the millimeter-wave radar detects the foot target as P, its radial distance is R, and the yaw angle is α, its coordinates in the world coordinate system can be calculated by the following formula:

[0030]

[0031] The conversion between the camera coordinate system and the world coordinate system is as follows Figure 5 As shown, O w -X w Y w Z w is the world coordinate, O c -X c Y c Z c is the camera coordinate system, with the origin O cis the center of the camera lens, Z c is the optical axis of the camera, which passes vertically through the imaging plane xO1y and intersects with the imaging plane at O1.

[0032] The transformation relationship between the camera coordinate system and the world coordinate system can be expressed by the rotation matrix R and the translation vector T. For a point P in three-dimensional space, its coordinates in the world coordinate system and the camera coordinate system are (X w ,Y w ,Z w ) and (X c ,Y c ,Z c ), there is the following relationship between them:

[0033]

[0034] Conversion between camera coordinate system and image coordinate system. Let a point P(X c ,Y c ,Z c ), the projection point in the imaging plane is P', and the coordinates in the image coordinate system are (x, y). Point P' is the point between point P and the optical center of the camera O c Connection O c The intersection of P and the imaging plane xO1y. According to the principle of pinhole imaging, there is a similarity between point P and point P'. According to the similarity principle, we can get:

[0035]

[0036] Among them, f is the focal length of the camera. The above formula is converted into a matrix form for representation, that is, the conversion from the camera coordinate system to the image coordinate system is as follows:

[0037]

[0038] The conversion from image coordinate system to pixel coordinate system is as follows Figure 6 As shown, a pixel coordinate system uO0v is established with the upper left corner as the origin on the imaging plane. The coordinate axes O0u and O0v are parallel to the coordinate axes O1x and O1y in the image coordinate system respectively. For each pixel point on the image, its horizontal coordinate u represents the column number of the pixel in the image array, and the vertical coordinate v represents the row number. The unit is pixel unit. It is necessary to relate the pixel unit to the physical unit. Use d x and d y Respectively represent the physical size of each pixel on the horizontal axis x and the vertical axis y. The coordinates of the origin O1 in the image coordinate system in the pixel coordinate system are (u0, v0). Then, the coordinates of each pixel in the image in the pixel coordinate system uO0v and in the image coordinate system xO1y have the following relationship:

[0039]

[0040] It can be expressed in matrix form as:

[0041]

[0042] like Figure 7 As shown in the figure, the sampling frequency of the millimeter wave radar used in the present invention is 20Hz, and the sampling frame rate of the camera is 30fps. Since the sampling rates of the two sensors are inconsistent, a multi-threaded time synchronization method is required to obtain environmental information at the same time. The lowest common multiple of the sampling periods of the two sensors is selected as the sampling period of the system, that is, the sampling period of the sensor fusion system is 100ms.

Claims

1. A DLP-based intelligent interactive unlocking system, characterized in that: It includes power supply unit, DLP projection unit, information acquisition unit, preprocessing unit, decision fusion unit and execution unit. CAN communication is adopted between each unit. The power supply unit is responsible for the power supply of the entire intelligent interactive unlocking system, and can diagnose the short circuit, open circuit and communication failure information of the external device power supply to play a protective role; The DLP projection unit is used to project the unlocking interface and has a self-correction function to ensure that the projection effect is not distorted; The information collection unit is composed of a 4D millimeter-wave radar and an infrared camera, which is set on the main driving side and is responsible for collecting user interaction information. The short-range mode of the 4D millimeter-wave radar is used for scene detection of the intelligent interactive unlocking system, and the infrared camera is used to filter out the visible light information of the image to avoid interference from the projector light. The pre-processing unit belongs to the controller component, and uses the YoloX algorithm to detect the target on the image information returned by the information acquisition unit, parses each field returned by the information acquisition unit to calculate the true value, and filters the target information output by the 4D millimeter wave radar to obtain the target position and motion data for further decision-making; The decision fusion unit belongs to the controller component, which fuses the 4D millimeter wave radar data with the image data in time, space and features, and then makes a control decision; The execution unit unlocks the vehicle after receiving the relevant unlocking instruction.

2. A DLP-based intelligent interactive unlocking method, characterized in that: The method is implemented by a DLP-based intelligent interactive unlocking system as described in claim 1, comprising the following steps: Step 1: After the DLP projection unit is self-calibrated, the unlocking interface is accurately projected onto the ground in the corresponding projection area; Step 2: The signal acquisition unit transmits the detected user foot data to the preprocessing unit for analysis and filtering, and then transmits it to the decision fusion unit; Step 3: After obtaining the 4D millimeter wave radar data and image data, the decision fusion unit performs spatiotemporal fusion and feature fusion to complete the coordinate system conversion and synchronous alignment of the two sensors in the time dimension, as well as extract feature information and effective targets; Step 4: The decision fusion unit uses the feature fusion information to determine the footsteps, and further compares the foot position coordinates with the corresponding projection area coordinates of the unlocking interface to obtain the interaction result after the decision, and the execution unit performs the unlocking operation.

3. The DLP-based intelligent interactive unlocking method according to claim 2, characterized in that: The specific scheme of spatiotemporal fusion in the method is as follows: The spatial fusion of 4D millimeter-wave radar and infrared camera is completed by the coordinate transformation of the two sensor data, including the transformation of radar coordinate system and world coordinate system, camera coordinate system and world coordinate system, camera coordinate system and image coordinate system, and pixel coordinate system and image coordinate system. In addition, the infrared camera needs to be calibrated during the coordinate transformation process, and the two sensors need to be jointly calibrated to complete the spatial fusion. The time fusion of 4D millimeter-wave radar and infrared camera is achieved by selecting the least common multiple of the sampling periods of the two sensors as the sampling period of the system.

4. The DLP-based intelligent interactive unlocking method according to claim 2, characterized in that: The decision fusion unit is in the process of making decisions: after the data is fused in time and space and features, a speed threshold v needs to be set. τ , when the foot moving speed is greater than v τ When the foot is moving, it is considered to be moving, that is, the foot is considered to be in the lifting action; when the foot moving speed is less than or equal to v τ When the foot stops moving, that is, the foot is deemed to be in a stepping action, and the foot position coordinates when the foot is stepped on are selected for comparison with the corresponding projection area coordinates of the unlocking interface. If the foot position coordinates completely match the projection area coordinates corresponding to the preset password, the execution unit unlocks the door. The foot movement speed and foot position coordinates are selected after feature fusion.

5. The DLP-based intelligent interactive unlocking method according to claim 2, characterized in that: The feature fusion in the method is used to resolve the data conflict between the two sensors. When the detection data of the two sensors conflict, the target feature information is provided by the two sensors based on their advantages. The specific scheme is as follows: Target type: infrared camera has higher priority than 4D millimeter wave radar. In terms of horizontal distance, 4D millimeter-wave radar has higher priority than infrared camera; In terms of longitudinal distance, 4D millimeter-wave radar has higher priority than infrared camera; In terms of relative speed, 4D millimeter-wave radar has a higher priority than infrared camera.

Images