Automobile active door opening system and method based on millimeter wave and vision fusion

Abstract

The invention provides an automobile active door opening system and method based on millimeter wave and vision fusion, and the system comprises a vision sensor module which is used for collecting image data of the surrounding environment of an automobile door; the millimeter wave sensor module is used for transmitting and receiving electromagnetic wave signals so as to detect distance, speed, angle and vertical angle information of targets around the vehicle door; a controllable power supply; a key signal module; a sound-light alarm module; the vehicle door controller is used for controlling opening and closing of a vehicle door according to the instruction of the main controller; and the main controller module comprises a data processing unit and a control unit and is used for processing data acquired by the visual sensor and the millimeter wave sensor, generating a control instruction and controlling the vehicle door controller to execute door opening or closing operation. The invention aims to overcome the defects of an existing vehicle door active opening system, meet the requirement of a user for intelligent development of a vehicle, and provide more convenient and safer vehicle door opening experience for a driver and passengers.

Description

Technical Field

[0001] The present invention relates to the field of automotive electronic technology, and in particular to an automotive active door opening system and method based on millimeter wave and vision fusion. Background Art

[0002] With the rapid development of the automotive industry, intelligence and automation have become the main trends in the industry. Consumers have increasing requirements for the convenience and comfort of vehicles, and expect vehicles to provide more intelligent and user-friendly door opening solutions. Traditional door opening methods are mostly passive, that is, the driver and passengers need to manually operate the door handle to open the door. However, this method can cause inconvenience in some cases. For example, when the driver and passengers have their hands full of items, it is difficult to free their hands to open the door. In addition, when opening the door manually, the driver and passengers may fail to observe the environment around the door due to negligence, resulting in collisions with pedestrians, bicycles or other vehicles when the door is opened, thus causing traffic accidents.

[0003] To solve this problem, remote door opening systems have emerged, providing certain conveniences for drivers and passengers. However, remote door opening systems usually do not have the ability to perceive the environment around the door, and may still cause harm to nearby people and objects when the door is opened. In addition, the remote control signal may be interfered with, or the door may be opened at an inappropriate time when the remote control button is accidentally touched, posing a safety hazard.

[0004] With the advancement of science and technology, millimeter-wave radar technology and vision technology provide new solutions for active door opening systems. Millimeter-wave radar technology can accurately measure distance, speed and angle, can work normally in bad weather, and is not affected by lighting conditions, providing important data support for door opening safety. In recent years, the cost of millimeter-wave radar technology has gradually decreased, while its performance has continued to improve, making its application in automobile door opening systems possible.

[0005] On the other hand, camera-based vision technology can obtain rich image information and identify the shape and posture of pedestrians, vehicles and other obstacles near the door through image recognition algorithms, so as to determine whether there is a risk of collision. The development of deep learning algorithms has greatly improved the target recognition accuracy and processing speed of vision technology, providing a solid technical guarantee for the application of vision technology in door opening systems.

[0006] However, millimeter-wave radar and vision technology each have their own advantages and disadvantages. Millimeter-wave radar is accurate in distance and speed measurement, but its target recognition capability is relatively weak; while vision technology can perform detailed target recognition, its performance may be limited in harsh environments. Therefore, the integration of these two technologies can achieve data complementarity and fully and accurately perceive the environment around the door. After integrating millimeter-wave radar and vision data, the system can comprehensively consider multiple factors to make decisions on door opening, avoiding possible misjudgments caused by a single technology, thereby significantly improving the reliability and safety of door opening decisions. Summary of the invention

[0007] In response to the problems existing in the prior art, the present invention provides an active automobile door opening system and method based on millimeter wave and vision fusion, aiming to solve the shortcomings of the existing active door opening system, meet users' needs for the development of automobile intelligence, and provide drivers and passengers with a more convenient and safe door opening experience.

[0008] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0009] An active door opening system for automobiles based on millimeter wave and vision fusion, comprising:

[0010] A visual sensor module, used to collect image data of the environment around the door;

[0011] A millimeter wave sensor module for transmitting and receiving electromagnetic wave signals to detect the distance, speed, angle and vertical angle information of targets around the door;

[0012] Controllable power supply, used to provide power support for the entire system;

[0013] The key signal module is used to detect the key control signal and trigger partial wake-up of the system;

[0014] Sound and light alarm module, used to issue an alarm prompt when verification fails or there is a risk of collision;

[0015] The door controller is used to control the opening and closing of the door according to the instructions of the main controller;

[0016] The main controller module includes a data processing unit and a control unit, which are used to process the data collected by the visual sensor and the millimeter wave sensor, generate control instructions, and control the door controller to perform door opening or door closing operations.

[0017] A method for actively opening a car door based on the fusion of millimeter wave and vision, which is applied to the above-mentioned active door opening system for a car based on the fusion of millimeter wave and vision, comprises the following steps:

[0018] When the key signal module detects the key locking signal, it wakes up the millimeter wave sensor through the CAN message and enters the low-power operation state;

[0019] The millimeter wave sensor switches to motion detection mode to determine whether there is a moving target approaching within the detection range;

[0020] When a moving target is detected approaching, the millimeter wave sensor sends a CAN signal to trigger the controllable power supply to power on the entire system and switch to the biometric detection mode;

[0021] The main controller starts to collect visual sensor data, uses the deep learning YOLO5Face algorithm to detect pedestrians, recognizes the detected face information, and compares it with the preset face library data;

[0022] When face recognition is successful and the biometric verification success command from the millimeter-wave radar is received, the main controller controls the millimeter-wave sensor to switch to obstacle detection mode, collects the distance, speed and angle information of the obstacle, and calculates a reasonable door opening command;

[0023] The door controller controls the door to open according to the door opening command.

[0024] According to an active vehicle door opening method based on millimeter wave and vision fusion provided by the present invention, after the millimeter wave sensor module switches to the motion detection mode, it detects the detection area and receives the echo signal, and its ADC data is algorithmically processed to obtain the distance, speed, angle and vertical angle information of the target. The specific steps include:

[0025] Range FFT processing: Perform fast Fourier transform processing on the collected continuous echo data in the time dimension to extract the distance information of the target;

[0026] Static background removal: After completing the range FFT processing, the data is processed by using the data mean value to remove the DC component to eliminate static background interference;

[0027] Doppler FFT processing: Perform two-dimensional Doppler FFT processing on the data after static background removal to calculate the amplitude information, thereby obtaining the distance-velocity amplitude matrix;

[0028] Multi-channel non-coherent accumulation: The corresponding units of the range-velocity amplitude matrix generated by all receiving channels are accumulated and summed to obtain the range-Doppler detection matrix for target detection;

[0029] CA-CFAR detection: Perform constant false alarm rate CA-CFAR processing on the detection matrix to detect the target point from the background noise and obtain the distance and speed information of the target point;

[0030] DOA angle estimation: The minimum variance distortion-free response (MVDR) algorithm in the beamforming method is used to calculate the azimuth and pitch angles of the target, and finally generate point cloud data containing the target's distance, speed, angle, and vertical angle information.

[0031] According to an active door opening method for an automobile based on millimeter wave and vision fusion provided by the present invention, in DOA angle estimation, the receiving signal model function of the MVDR algorithm is:

[0032]

[0033] Where x(t) is the M x 1 received signal vector, k is the number of signal sources, and a(θ k ) is the direction vector of the kth signal source, θ k is the direction of the kth signal, s k (t) is the signal of the kth source, n(t) is the additive noise;

[0034] The objective function formula of the MVDR algorithm is:

[0035] J(w)=w H Rw+λ(w H a(θ d )-1)

[0036] Where w is the weight vector, R = E[x(t)x H (t)] is the covariance matrix of the received signal, λ is the Lagrange multiplier, θ d is the desired signal direction.

[0037] According to an active door opening method for an automobile based on millimeter wave and vision fusion provided by the present invention, the derivative of J(w) with respect to w is calculated and set to zero, and the optimal weight vector is obtained as follows:

[0038]

[0039] The output signal after beamforming is:

[0040]

[0041] According to the active door opening method for automobiles based on millimeter wave and vision fusion provided by the present invention, the acquired target data within the detection range is processed by the DBScan algorithm clustering EKF tracking algorithm;

[0042] Perform effective target judgment on the tracked target, that is, determine whether the tracked target is moving and gradually approaching the vehicle.

[0043] According to a method for actively opening a vehicle door based on millimeter wave and vision fusion provided by the present invention, the logic of effective target judgment specifically includes the following steps:

[0044] Track starting point confirmation: Determine whether the starting point of the target track tracked by the millimeter wave sensor is outside the pre-set detection area;

[0045] Calculation of the distance deviation between the current position and the starting point: By calculating the distance deviation between the target's current position and the starting point of its track, it is determined whether the target is moving continuously;

[0046] Speed ​​judgment: Based on the speed information of the target, it is judged whether it is approaching or moving away from the vehicle. Only targets close to the vehicle are further judged as valid targets, and targets far away from the vehicle are regarded as invalid targets and discarded;

[0047] Valid target confirmation: When a target is confirmed to be a target close to the vehicle and the distance difference between its current position and the starting position is greater than a preset threshold, the target is judged to be a valid target, that is, it is judged to be a legitimate target with the intention of approaching the vehicle.

[0048] According to a method for actively opening a car door based on millimeter wave and vision fusion provided by the present invention, the switching of the biometric detection mode includes:

[0049] Switch from the millimeter wave sensor's motion detection mode to the breathing and heartbeat detection mode;

[0050] Based on the target position information tracked by the millimeter wave sensor, determine whether the target enters the confirmation area formed by the vehicle's B-pillar as the center and extending a certain distance on both sides;

[0051] After confirming that the target has entered the confirmation area, the millimeter wave sensor detects the target's breathing and heart rate, and compares the detected breathing and heart rate with the preset range of the real breathing and heart rate of the human body;

[0052] The breathing and heartbeat comparison results are transmitted to the main controller via the CAN bus and saved in the memory of the main controller for secondary comparison with the face recognition results collected by the visual sensor.

[0053] According to a method for active door opening of a car based on millimeter wave and vision fusion provided by the present invention, when collecting data for target detection and identity recognition, the specific steps include:

[0054] Image data acquisition: The main controller captures the image data of the visual sensor and obtains the visual information of the environment around the door;

[0055] Image data preprocessing: perform data normalization and image enhancement algorithm processing on the captured image data;

[0056] Human target detection: The main controller uses the YOLOV5 algorithm to perform pedestrian detection on the pre-processed image data to determine whether there are human targets in the image;

[0057] Confirmation area judgment: After detecting a human target, the main controller receives the target position information sent by the millimeter wave sensor and determines whether the target has entered the preset confirmation area to decide whether to activate the face recognition function;

[0058] Image segmentation: When the target is confirmed to enter the confirmation area, the main controller extracts the face information from the captured image data;

[0059] Face recognition: The YOLO5Face algorithm is used to identify the extracted face information, and the similarity is compared with the data in the pre-stored face database of the vehicle to determine whether the currently detected pedestrian is a legal member of the vehicle.

[0060] According to a method for active door opening of a car based on millimeter wave and vision fusion provided by the present invention, the training process of the YOLO5Face algorithm model used in face recognition specifically includes:

[0061] Dataset collection: Collect user face data to ensure that the collection process includes at least face images in different scenes, postures, and lighting conditions;

[0062] Data annotation: Use annotation tools to annotate the collected face images and mark the bounding box of the face;

[0063] Data preprocessing: resizing, image enhancement, image data diversity processing, and image clarity processing;

[0064] Environment construction and model configuration: Install the deep learning framework PyTorch and its related dependent libraries, install other auxiliary libraries, obtain the YOLO5Face model code, and configure the model parameters and anchor box parameters as required;

[0065] Model training: Divide the dataset into training, validation, and test sets, select the optimizer and loss function, set the training parameters, iterate the model parameters through multiple rounds of training, and regularly evaluate the model performance on the validation set.

[0066] It can be seen that compared with the prior art, the present invention has the following beneficial effects:

[0067] 1. The present invention achieves accurate perception of the environment around the car door by fusing the data of millimeter-wave radar and visual sensor. The millimeter-wave radar provides the system with stable and reliable physical environment data with its ability to accurately measure distance, speed and angle; while the visual sensor further enhances the system's ability to understand the surrounding environment by acquiring rich image information. This fusion of multi-source data enables the system to fully and accurately grasp the dynamic situation around the car door.

[0068] The present invention uses advanced intelligent algorithms to analyze and process the fused data, and intelligently decides the timing and method of door opening. The algorithm can automatically identify obstacles, pedestrians and authorized users, and dynamically adjust the door opening strategy according to the real-time environmental conditions, such as the opening timing, opening angle and opening speed, so as to ensure the safety and convenience of door opening.

[0069] The present invention adopts a multi-source data fusion architecture design. The millimeter-wave radar and visual sensor cooperate with each other and back up each other. When facing complex road conditions and harsh environments such as direct sunlight, heavy rain, and electromagnetic interference, even if the performance of one sensor is affected, the other sensor can still operate stably and continue to provide key data, ensuring the system's accurate judgment and safe control of the conditions around the vehicle door, laying a solid foundation for the stable and reliable operation of the system.

[0070] In summary, the automobile active door opening system and method proposed in the present invention effectively improve the intelligence level and safety of automobile door opening, and provide drivers and passengers with a more comfortable and safe travel experience.

[0071] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a schematic diagram of an embodiment of an active door opening system for an automobile based on the fusion of millimeter waves and vision according to the present invention.

[0073] Figure 2 It is a flow chart of an embodiment of an active vehicle door opening method based on millimeter wave and vision fusion of the present invention.

[0074] Figure 3 It is a system workflow diagram of an embodiment of an active vehicle door opening method based on millimeter wave and vision fusion of the present invention.

[0075] Figure 4 It is an algorithm flow chart for target cloud generation in an embodiment of an active door opening method for a car based on millimeter wave and vision fusion according to the present invention.

[0076] Figure 5The present invention is a logic flow chart of the control of the door actuator in an embodiment of the active door opening method for a car based on the fusion of millimeter waves and vision.

[0077] Figure 6 This is a diagram of the sensor installation position and effective action area in an embodiment of an active vehicle door opening method based on millimeter wave and vision fusion of the present invention.

[0078] Figure 7 It is a schematic diagram of the composition of multi-mode millimeter wave sensor modes in an embodiment of an active door opening method for an automobile based on millimeter wave and vision fusion of the present invention. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0081] An embodiment of an active door opening system for automobiles based on millimeter wave and vision fusion

[0082] See also Figure 1 This embodiment provides an active door opening system for automobiles based on millimeter wave and vision fusion, including:

[0083] The visual sensor module is used to collect image data of the environment around the door, identify pedestrians, vehicles and other targets through image processing technology, and provide rich visual data support for the system;

[0084] Millimeter wave sensor module, used to transmit and receive electromagnetic wave signals to detect the distance, speed, angle and vertical angle information of targets around the door, monitor the environmental changes around the door in real time, and provide reliable radar data for the system;

[0085] Controllable power supply, used to provide power support for the entire system, ensuring that the system can be powered on quickly when needed, and cut off the power supply when not needed to save energy;

[0086] The key signal module is used to detect the key control signal, such as locking and unlocking commands, and trigger partial wake-up of the system as one of the trigger conditions for starting or shutting down the system;

[0087] The sound and light alarm module is used to issue an alarm when verification fails or there is a risk of collision, such as illegal intrusion, collision risk, etc., and issue an alarm through sound and light signals to remind drivers and passengers to pay attention to safety;

[0088] The door controller is used to control the opening and closing of the door according to the instructions of the main controller;

[0089] The main controller module, including a data processing unit and a control unit, is used to process the data collected by the visual sensor and the millimeter wave sensor, and perform advanced processing such as target recognition and trajectory prediction. The control unit generates control instructions based on the processing results, generates control instructions, and controls the door controller to perform door opening or closing operations.

[0090] The millimeter wave sensor, main controller and door controller realize data exchange through the same CAN bus to ensure the real-time and reliability of data transmission. The millimeter wave sensor sends the collected radar data to the main controller through the CAN bus. The main controller processes the data and generates control instructions, which are then sent to the door controller through the CAN bus.

[0091] The visual sensor sends the collected image data to the main controller, which uses image processing technology to analyze and process the image, identify the type, location and other information of the target object, and provide support for system decision-making.

[0092] The key signal module receives the signal from the key and transmits it to the main controller. The main controller determines the user's intention based on the key signal, such as locking or unlocking the car, and generates corresponding control instructions.

[0093] When the main controller detects an abnormal situation, such as illegal intrusion, collision risk, etc., it will send an alarm command to the sound and light alarm module. After receiving the command, the sound and light alarm module will immediately send out sound and light signals to remind the driver and passengers to pay attention to safety.

[0094] The door controller receives the control command sent by the main controller and drives the door to perform the corresponding action. At the same time, the door controller also feeds back the current state of the door (such as open, closed, etc.) to the main controller so that the main controller can monitor and control the door in real time.

[0095] In actual applications, after the system is powered on, each module performs initialization operations to ensure that the system is in normal working condition. The millimeter wave sensor and visual sensor start to collect data and send the data to the main controller. The main controller processes and analyzes the data to identify the type, location and other information of the target object. The main controller generates corresponding control instructions based on the information of the target object and inputs such as key signals. For example, when a legitimate user is detected approaching the door, the main controller generates a door opening instruction. The door controller receives the control instructions sent by the main controller and drives the door to perform the corresponding actions. At the same time, the door controller also feeds back the current state of the door to the main controller so that the main controller can monitor and control it in real time. When the system detects an abnormal situation, such as illegal intrusion, collision risk, etc., the main controller will immediately send an alarm instruction to the sound and light alarm module to remind the driver and passengers to pay attention to safety.

[0096] An embodiment of an active door opening method for an automobile based on millimeter wave and vision fusion

[0097] See also Figures 2 to 7 This embodiment provides an active door opening method for an automobile based on the fusion of millimeter waves and vision. The method is applied to the above-mentioned active door opening system for an automobile based on the fusion of millimeter waves and vision, and includes the following steps:

[0098] Step S1, when the key signal module detects the key locking signal, the millimeter wave sensor is awakened through the CAN message to enter a low power consumption operation state;

[0099] Step S2, the millimeter wave sensor switches to a motion detection mode to determine whether there is a moving target approaching within the detection range;

[0100] Step S3, when a moving target is detected approaching, the millimeter wave sensor sends a CAN signal to trigger the controllable power supply to power on the entire system and switch to the biometric detection mode;

[0101] Step S4: The main controller starts to collect visual sensor data, uses the deep learning YOLO5Face algorithm to detect pedestrians, and recognizes the detected face information and compares it with the preset face library data;

[0102] Step S5, when the face recognition is successful and the biometric verification success command of the millimeter wave radar is received, the main controller controls the millimeter wave sensor to switch to the obstacle detection mode, collects the distance, speed and angle information of the obstacle, and calculates a reasonable door opening instruction;

[0103] Step S6, the door controller controls the door to open according to the door opening instruction.

[0104] In the above step S1, when the user uses the key to trigger the door locking function, the system is not fully powered on, but enters a partial power-on state, which can balance the system response speed and energy efficiency, and ensure that only key modules are activated to maintain basic functions when the full system operation is not required. In the partial power-on state, the system only starts the core circuits related to the door locking and subsequent wake-up process, such as the power management module, CAN communication interface, etc., and wakes up the 77GHz millimeter-wave radar sensor through the CAN message to enter a low-power operation state; the message contains the wake-up instruction and necessary configuration information, which is used to instruct the radar sensor to enter a low-power operation state from a dormant state.

[0105] After receiving the wake-up message, the millimeter wave radar sensor will start its internal power management module and switch the sensor from sleep mode to low power operation mode. In this mode, the radar sensor only maintains basic signal transmission and reception functions to detect moving targets in the surrounding environment, while greatly reducing power consumption.

[0106] In the above step S2, after switching to the motion detection mode, the millimeter wave sensor module detects the detection area and receives the echo signal, and its ADC data is processed by algorithm to obtain the target point cloud information, which mainly includes the distance, speed, angle and vertical angle information of the target, such as Figure 4 As shown, the specific steps include:

[0107] Range FFT processing: Perform fast Fourier transform processing on the collected continuous echo data in the time dimension to extract the distance information of the target;

[0108] Static background removal: After completing the range FFT processing, the data is processed by using the data mean value to remove the DC component to eliminate static background interference;

[0109] Doppler FFT processing: Perform two-dimensional Doppler FFT processing on the data after static background removal to calculate the amplitude information, thereby obtaining the distance-velocity amplitude matrix;

[0110] Multi-channel non-coherent accumulation: The corresponding units of the range-velocity amplitude matrix generated by all receiving channels are accumulated and summed to obtain the range-Doppler detection matrix for target detection;

[0111] CA-CFAR detection: Perform constant false alarm rate CA-CFAR processing on the detection matrix to detect the target point from the background noise and obtain the distance and speed information of the target point;

[0112] DOA angle estimation: The minimum variance distortion-free response (MVDR) algorithm in the beamforming method is used to calculate the azimuth and pitch angles of the target, and finally generate point cloud data containing the target's distance, speed, angle, and vertical angle information.

[0113] Furthermore, in DOA angle estimation, the mathematical principle and advantages of the MVDR algorithm are as follows: The MVDR mathematical expression is as follows, and the received signal model function is:

[0114]

[0115] Where x(t) is the M x 1 received signal vector, k is the number of signal sources, and a(θ k ) is the direction vector of the kth signal source, θ k is the direction of the kth signal, s k (t) is the signal of the kth source, n(t) is the additive noise;

[0116] The objective function formula of the MVDR algorithm is:

[0117] J(w)=w H Rw+λ(w H a(θ d )-1)

[0118] Where w is the weight vector, R = E[x(t)x H (t)] is the covariance matrix of the received signal, λ is the Lagrange multiplier, θ d is the desired signal direction.

[0119] Taking the derivative of J(w) with respect to w and setting it to zero, we can get the optimal weight vector:

[0120]

[0121] The output signal after beamforming is:

[0122]

[0123] The MVDR algorithm has high resolution, strong adaptability, excellent robustness, and significant signal enhancement. It can improve the expected signal strength, suppress interference noise, improve the signal-to-noise ratio, and facilitate weak signal detection and processing.

[0124] In this embodiment, the acquired target data within the detection range is processed by the DBScan algorithm clustering EKF tracking algorithm;

[0125] Perform effective target judgment on the tracked target, that is, determine whether the tracked target is moving and gradually approaching the vehicle.

[0126] In this embodiment, the logic of valid target judgment specifically includes the following steps:

[0127] Track starting point confirmation: Determine whether the starting point of the target track tracked by the millimeter wave sensor is outside the pre-set detection area;

[0128] Calculation of the distance deviation between the current position and the starting point: By calculating the distance deviation between the target's current position and the starting point of its track, it is determined whether the target is moving continuously;

[0129] Speed ​​judgment: Based on the speed information of the target, it is judged whether it is approaching or moving away from the vehicle. Only targets close to the vehicle are further judged as valid targets, and targets far away from the vehicle are regarded as invalid targets and discarded;

[0130] Valid target confirmation: When the target is confirmed to be a target close to the vehicle, and the distance difference between its current position and the starting position is greater than a preset threshold (such as greater than 3m), the target is judged to be a valid target, that is, it is judged to be a legitimate target with the intention of approaching the vehicle.

[0131] In this embodiment, when the system detects a moving target approaching the vehicle, the system will immediately send a command to the controllable power supply through the CAN (Controller Area Network) bus. After receiving the CAN command, the controllable power supply will quickly power on to provide the required power support for the entire vehicle active door opening system. Among them, the CAN bus, as the backbone network for internal communication in the vehicle, can efficiently and reliably transmit control commands.

[0132] It can be seen that the system adopts a sleep-wake mechanism and only powers on as a whole when a moving target is detected, which effectively reduces energy consumption and prolongs battery life. Once a moving target is detected, the system can quickly activate the entire system through CAN commands to ensure fast response and improve user experience.

[0133] In this embodiment, the switching of the biometric detection mode includes:

[0134] Biometric detection mode switching: switching from the millimeter wave sensor's motion detection mode to the breathing and heartbeat detection mode;

[0135] Area judgment: Based on the target position information tracked by the millimeter wave sensor, it is judged whether the target enters the confirmation area formed by the B-pillar of the vehicle as the center and extending a certain distance on both sides, such as the length and width of the left and right areas are 1 meter;

[0136] Biometric detection: After confirming that the target has entered the confirmation area, the millimeter wave sensor detects the target's breathing and heart rate, and compares the detected breathing and heart rate with the preset human body's real breathing and heart rate range;

[0137] Data upload: The breathing and heartbeat comparison results are transmitted to the main controller via the CAN bus and saved in the main controller's memory for secondary comparison with the face recognition results collected by the visual sensor.

[0138] Waiting for controller command: If the door opening command is received, the millimeter wave switches to obstacle detection mode and sends the distance, speed, and angle data of the detected target to the door controller in real time, providing data basis for the door opening anti-collision warning;

[0139] In this embodiment, when collecting data for target detection and identity recognition, the specific steps include:

[0140] Image data acquisition: The main controller captures the image data of the visual sensor and obtains the visual information of the environment around the door;

[0141] Image data preprocessing: Perform data normalization and image enhancement algorithm processing on the captured image data. Data normalization and image data enhancement technology can greatly reduce the impact of insufficient data volume, improve the robustness of the model, provide various "invariances" for the model, and increase the model's ability to resist overfitting;

[0142] Human target detection: The main controller uses the YOLOV5 algorithm to perform pedestrian detection on the pre-processed image data to determine whether there are human targets in the image;

[0143] Confirmation area judgment: After detecting a human target, the main controller receives the target position information sent by the millimeter wave sensor and determines whether the target has entered the preset confirmation area to decide whether to activate the face recognition function;

[0144] Image segmentation: When the target is confirmed to enter the confirmation area, the main controller extracts the face information from the captured image data;

[0145] Face recognition: The YOLO5Face algorithm is used to identify the extracted face information, and the similarity is compared with the data in the pre-stored face database of the vehicle to determine whether the currently detected pedestrian is a legal member of the vehicle.

[0146] Furthermore, the YOLO5Face algorithm is used to recognize the collected faces. It is necessary to first collect the face data of users (such as family members) for training and finally obtain a trained model, which specifically includes the following steps:

[0147] Dataset collection: Collect facial data of users (such as all family members), ensuring that the collection process includes at least facial images under different scenes, postures, and lighting conditions;

[0148] Data labeling: label the face position, use labeling tools (such as LabelImg, etc.) to label the collected face images, and label the bounding box of the face;

[0149] Data preprocessing: resize, enhance, diversify, and sharpen image data, and uniformly adjust images to the input size required by the model. For example, YOLO5Face may require the input image size to be a specific size (such as 640×640 pixels). Randomly crop images to simulate partial occlusion, flip images horizontally or vertically to increase data diversity; adjust image brightness, contrast, saturation, and hue, etc.

[0150] Environment construction and model configuration: Install the deep learning framework PyTorch and its related dependent libraries, install other auxiliary libraries, obtain the YOLO5Face model code, and configure the model parameters and anchor box parameters as required;

[0151] Model training: Divide the dataset into training, validation, and test sets, select the optimizer and loss function, set the training parameters, iterate the model parameters through multiple rounds of training, and regularly evaluate the model performance on the validation set.

[0152] In the above environment construction and model configuration steps, the following steps are specifically included:

[0153] (1) Install the deep learning framework, PyTorch, and ensure that the version suitable for training and its related dependent libraries are installed.

[0154] (2) Install other auxiliary libraries, such as numpy, opencv-python, etc. for data processing, as well as related libraries for model evaluation.

[0155] (3) Model selection and configuration, obtain the YOLO5Face model code, and obtain the YOLO5Face model code from the official or reliable source.

[0156] (4) Configure model parameters and select the appropriate network structure corresponding to the YOLO5 version according to requirements. Different versions differ in network depth and width.

[0157] (5) Anchor box parameters: adjust the anchor box according to the characteristics of the dataset. The anchor box is a priori box used to predict the target bounding box. A suitable anchor box can improve the detection accuracy.

[0158] In the above model training steps, it is mainly divided into the following parts:

[0159] (1) Divide the data set into training set, validation set and test set. Generally, the data set is divided into training set, validation set and test set according to a certain ratio (such as 8:1:1). The training set is used to train the model, the validation set is used to evaluate the performance of the model during the training process to prevent overfitting, and the test set is used to finally evaluate the performance of the trained model.

[0160] (2) Select the optimizer and loss function. For optimizer selection, usually Adam or SGD (stochastic gradient descent) optimizers are selected. The Adam optimizer has the advantage of adaptive learning rate, which can make the model converge quickly in the early stage of training; loss function definition, bounding box loss: use CIoU (Complete IoU) loss to measure the difference between the predicted face bounding box and the true bounding box; classification loss, use the cross entropy loss function to calculate the classification error. Key point loss, use the mean square error (MSE) loss to calculate the error between the predicted key point and the true key point.

[0161] (3) Start training and set training parameters. Learning rate: The initial learning rate is usually set within a reasonable range (such as 0.001), and a learning rate decay strategy can be adopted according to the training situation; the number of training rounds is determined according to the size of the data set and the convergence of the model. Generally, multiple rounds of training are required; batch size: Determine the appropriate batch size based on hardware resources (such as GPU memory), such as 32 or 64;

[0162] (4) Model training iteration: In each round of training, the model performs forward propagation on a small batch of data in the training set, calculates the loss, and then performs backpropagation through the optimizer to update the model parameters. The model performance is regularly evaluated on the validation set, and indicators such as accuracy, recall, and average precision are recorded.

[0163] In this embodiment, the model evaluation and optimization steps are also included: the model evaluation includes evaluation on the test set and analysis of the evaluation results. After the training is completed, the model is evaluated on the test set to obtain the final performance indicators of the model; the evaluation results are analyzed, and the advantages and disadvantages of the model are analyzed according to the evaluation results, such as the inability to detect small-sized faces and the low accuracy of face detection in specific postures.

[0164] Model optimization includes hyperparameter adjustment and model improvement: hyperparameter adjustment: adjust hyperparameters such as learning rate, batch size, and number of training rounds according to evaluation results; model improvement: consider improving the network structure, such as adding or modifying certain network layers. Adjust anchor box parameters to make them more suitable for the size distribution of faces in the dataset;

[0165] In this embodiment, a model deployment step is also included, and model deployment is divided into model saving and conversion and deployment to actual application. Model saving and conversion, saving the trained model: saving the model's weight file to ensure that it can be loaded and used in subsequent applications; deploying to actual applications, integrating into the system, integrating the trained YOLO5Face model into the target application system to realize the face detection function.

[0166] In the above step S6: the door opening command is generated, combined with the biometric data sent by the millimeter wave for fusion judgment, if all conditions are met, a control command is sent to the millimeter wave sensor to switch the obstacle detection mode, and the target distance, speed and angle information output by the millimeter wave radar are collected to calculate the door opening command. If there is no collision risk, a door full opening command is sent. If there is a collision risk, the main controller generates instructions such as door opening angle and door opening speed according to the obstacle information;

[0167] The door opening command is sent to the door controller via the CAN bus;

[0168] Active door opening control: the door controller actively controls the door opening according to the door opening command received from the main controller, mainly based on the regional location information, whether it is area 1 or area 2, to determine whether to open the main driving door or the passenger door, and automatically controls it according to the door opening angle and motor speed.

[0169] like Figure 6 As shown in the figure, the installation position and functional area of ​​the automotive active door opening system sensor are divided. The millimeter wave radar is installed at the front or side of the vehicle to cover the detection area around the vehicle. The visual sensor is usually installed at the front, side or near the rearview mirror of the vehicle to obtain image information around the vehicle. Figure 6 In the figure, the visual sensor is shown installed on the side of the vehicle, working in conjunction with the millimeter-wave radar. The detection area is defined as a rectangular area 5 meters wide and 5 meters wide with the millimeter-wave sensor as the center, and the confirmation area is centered on the B-pillar, with the length and width of the left and right areas being 1 meter. Through the coordinated work of the millimeter-wave radar and the visual sensor, the active door opening system of the car can achieve comprehensive perception and accurate judgment of the vehicle's surrounding environment, thereby ensuring the safety and convenience of the door during the opening process. At the same time, the addition of the sound and light alarm further enhances the system's safety and user experience.

[0170] like Figure 7 As shown in the figure, the multi-mode millimeter wave sensor has three main working modes for different scenarios. The motion detection mode meets the low-power operation requirements before the main system wakes up; the biometric detection mode provides the target's breathing and heartbeat and other biological information for secondary confirmation with the visual processing results; the obstacle detection mode provides the main controller with the target distance, speed and angle information, so that the main controller can intelligently calculate the door opening position, angle speed and other door opening instructions based on the obstacle information, providing data support for door collision prevention.

[0171] In summary, the active door opening system and method for automobiles based on the fusion of millimeter waves and vision provided in this embodiment has many beneficial effects, such as efficient awakening and intelligent linkage, multiple verifications to ensure door opening safety, real-time obstacle detection and dynamic adjustment, fusion perception to improve the level of intelligence, and effective solution to the problem of active door opening for automobiles. These advantages make the present invention have broad application prospects and important practical value in the field of active door opening for automobiles.

[0172] Furthermore, the millimeter-wave radar, as the "pioneer sentinel" of the system, can quickly detect and track targets outside the vehicle. When a moving target is detected approaching the vehicle, a CAN message is immediately sent to wake up the entire system to ensure that the system responds in time and enters the working state. The visual detection module then takes over the work, accurately determines whether the target is a person, and starts the face recognition process after successful detection. This intelligent linkage between millimeter-wave radar and visual sensors realizes a seamless connection from target detection to identity recognition, greatly improving the response speed and accuracy of the system.

[0173] Furthermore, the system uses face recognition technology to compare the collected target face with the pre-stored face database to ensure that only authorized users can trigger the door opening signal. This link effectively prevents illegal intrusion and misoperation, and improves the safety of door opening. The millimeter wave radar further detects the target's breathing and heartbeat frequency. When it detects that the target's breathing frequency meets the human breathing range, it will send an active door opening signal to the door controller. This double verification mechanism further enhances the security and reliability of the system.

[0174] Furthermore, the millimeter-wave radar sends obstacle target distance angle information to the door controller in real time, and the door controller comprehensively determines whether the door should slow down, accelerate or stop based on the obstacle information received. This real-time obstacle detection and dynamic adjustment function ensures that the door will not collide with surrounding obstacles during the opening process, improving the safety and smoothness of the door opening.

[0175] Furthermore, by integrating the data from millimeter-wave radar and visual sensors, the system can accurately perceive the environment around the door, including the location, speed, angle, and identity information of the target. This fusion perception of multi-source data enables the system to have a more comprehensive understanding of the surrounding environment and provides rich data support for intelligent decision-making. The system uses intelligent algorithms to analyze and process the fused data, and intelligently decides the timing and method of door opening, making the door opening more in line with user needs and usage scenarios, and improving the intelligence level of car door opening.

[0176] Furthermore, the present invention, based on the method of fusion of millimeter wave and vision, effectively solves the problems existing in active opening of automobiles, such as misjudgment, missed judgment, slow response speed, etc. Through multiple verifications and real-time obstacle detection, the system can more accurately identify authorized users and safely open the door, while avoiding the risk of collision with surrounding obstacles.

[0177] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0178] The above-mentioned embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and substitutions made by technicians in this field on the basis of the present invention shall fall within the scope of protection required by the present invention.

Claims

1. An active door opening system for automobiles based on millimeter wave and vision fusion, characterized in that: include: A visual sensor module, used to collect image data of the environment around the door; A millimeter wave sensor module for transmitting and receiving electromagnetic wave signals to detect the distance, speed, angle and vertical angle information of targets around the door; Controllable power supply, used to provide power support for the entire system; The key signal module is used to detect the key control signal and trigger partial wake-up of the system; Sound and light alarm module, used to issue an alarm prompt when verification fails or there is a risk of collision; The door controller is used to control the opening and closing of the door according to the instructions of the main controller; The main controller module includes a data processing unit and a control unit, which are used to process the data collected by the visual sensor and the millimeter wave sensor, generate control instructions, and control the door controller to perform door opening or door closing operations.

2. A method for actively opening a car door based on the fusion of millimeter waves and vision, characterized in that: The method is applied to an active door opening system for an automobile based on millimeter wave and vision fusion as claimed in claim 1, and the method comprises the following steps: When the key signal module detects the key locking signal, it wakes up the millimeter wave sensor through the CAN message and enters the low-power operation state; The millimeter wave sensor switches to motion detection mode to determine whether there is a moving target approaching within the detection range; When a moving target is detected approaching, the millimeter wave sensor sends a CAN signal to trigger the controllable power supply to power on the entire system and switch to the biometric detection mode; The main controller starts to collect visual sensor data, uses the deep learning YOLO5Face algorithm to detect pedestrians, recognizes the detected face information, and compares it with the preset face library data; When face recognition is successful and the biometric verification success command from the millimeter-wave radar is received, the main controller controls the millimeter-wave sensor to switch to obstacle detection mode, collects the distance, speed and angle information of the obstacle, and calculates a reasonable door opening command; The door controller controls the door to open according to the door opening command.

3. The method according to claim 2, characterized in that After switching to the motion detection mode, the millimeter wave sensor module detects the detection area and receives the echo signal, and the specific steps of performing algorithm processing on the ADC data to obtain the distance, speed, angle and vertical angle information of the target include: Range FFT processing: Perform fast Fourier transform processing on the collected continuous echo data in the time dimension to extract the distance information of the target; Static background removal: After completing the range FFT processing, the data is processed by using the data mean value to remove the DC component to eliminate static background interference; Doppler FFT processing: Perform two-dimensional Doppler FFT processing on the data after static background removal to calculate the amplitude information, thereby obtaining the distance-velocity amplitude matrix; Multi-channel non-coherent accumulation: The corresponding units of the range-velocity amplitude matrix generated by all receiving channels are accumulated and summed to obtain the range-Doppler detection matrix for target detection; CA-CFAR detection: Perform constant false alarm rate CA-CFAR processing on the detection matrix to detect the target point from the background noise and obtain the distance and speed information of the target point; DOA angle estimation: The minimum variance distortion-free response (MVDR) algorithm in the beamforming method is used to calculate the azimuth and pitch angles of the target, and finally generate point cloud data containing the target's distance, speed, angle, and vertical angle information.

4. The method according to claim 3, characterized in that: In DOA angle estimation, the received signal model function of the MVDR algorithm is: Where x(t) is the M x 1 received signal vector, k is the number of signal sources, and a(θ k ) is the direction vector of the kth signal source, θ k is the direction of the kth signal, s k (t) is the signal of the kth source, n(t) is the additive noise; The objective function formula of the MVDR algorithm is: J(w)=w H Rw+λ(w H a(θ d )-1) Where w is the weight vector, R = E[x(t)x H (t)] is the covariance matrix of the received signal, λ is the Lagrange multiplier, θ d is the desired signal direction.

5. The method according to claim 4, characterized in that: Taking the derivative of J(w) with respect to w and setting it to zero, we can get the optimal weight vector: The output signal after beamforming is:

6. The method according to claim 1, characterized in that: The acquired target data within the detection range is processed by the DBScan algorithm clustering EKF tracking algorithm; Perform effective target judgment on the tracked target, that is, determine whether the tracked target is moving and gradually approaching the vehicle.

7. The method according to claim 6, characterized in that: The logic of effective target judgment specifically includes the following steps: Track starting point confirmation: Determine whether the starting point of the target track tracked by the millimeter wave sensor is outside the pre-set detection area; Calculation of the distance deviation between the current position and the starting point: By calculating the distance deviation between the target's current position and the starting point of its track, it is determined whether the target is moving continuously; Speed ​​judgment: Based on the speed information of the target, it is judged whether it is approaching or moving away from the vehicle. Only targets close to the vehicle are further judged as valid targets, and targets far away from the vehicle are regarded as invalid targets and discarded; Valid target confirmation: When a target is confirmed to be a target close to the vehicle and the distance difference between its current position and the starting position is greater than a preset threshold, the target is judged to be a valid target, that is, it is judged to be a legitimate target with the intention of approaching the vehicle.

8. The method according to any one of claims 1 to 7, characterized in that: The switching of the biometric detection mode includes: Switch from the millimeter wave sensor's motion detection mode to the breathing and heartbeat detection mode; Based on the target position information tracked by the millimeter wave sensor, determine whether the target enters the confirmation area formed by the vehicle's B-pillar as the center and extending a certain distance on both sides; After confirming that the target has entered the confirmation area, the millimeter wave sensor detects the target's breathing and heart rate, and compares the detected breathing and heart rate with the preset range of the real breathing and heart rate of the human body; The breathing and heartbeat comparison results are transmitted to the main controller via the CAN bus and saved in the memory of the main controller for secondary comparison with the face recognition results collected by the visual sensor.

9. The method according to any one of claims 1 to 7, characterized in that: When collecting data for object detection and identification, the specific steps include: Image data acquisition: The main controller captures the image data of the visual sensor and obtains the visual information of the environment around the door; Image data preprocessing: perform data normalization and image enhancement algorithm processing on the captured image data; Human target detection: The main controller uses the YOLOV5 algorithm to perform pedestrian detection on the pre-processed image data to determine whether there are human targets in the image; Confirmation area judgment: After detecting a human target, the main controller receives the target position information sent by the millimeter wave sensor and determines whether the target has entered the preset confirmation area to decide whether to activate the face recognition function; Image segmentation: When the target is confirmed to enter the confirmation area, the main controller extracts the face information from the captured image data; Face recognition: The YOLO5Face algorithm is used to identify the extracted face information, and the similarity is compared with the data in the pre-stored face database of the vehicle to determine whether the currently detected pedestrian is a legal member of the vehicle.

10. The method according to claim 9, characterized in that The training process of the YOLO5Face algorithm model used in the face recognition specifically includes: Dataset collection: Collect user face data to ensure that the collection process includes at least face images in different scenes, postures, and lighting conditions; Data annotation: Use annotation tools to annotate the collected face images and mark the bounding box of the face; Data preprocessing: resizing, image enhancement, image data diversity processing, and image clarity processing; Environment construction and model configuration: Install the deep learning framework PyTorch and its related dependent libraries, install other auxiliary libraries, obtain the YOLO5Face model code, and configure the model parameters and anchor box parameters as required; Model training: Divide the dataset into training, validation, and test sets, select the optimizer and loss function, set the training parameters, iterate the model parameters through multiple rounds of training, and regularly evaluate the model performance on the validation set.

Images