In-vehicle living body detection method and device
Through the multi-sensor combination and multi-source data fusion detection algorithm, the problem of large light and errors of the existing single solution of in-vehicle live detection technology is solved, achieving higher detection accuracy and user interaction, ensuring the safety of children or pets.
Patent Information
- Application Number
- CN202411868820.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-18
- Publication Date
- 2025-03-21
AI Technical Summary
The existing in-vehicle live detection technologies are mostly based on a single solution, and there is a problem that visual detection is affected by light and has large errors in a single sensor, making it difficult to effectively detect the existence of children or pets in the car.
Using a multi-sensor combination method, temperature information and respiratory micro-movement signals are collected through infrared sensors and millimeter wave radars, data is fused with data to output detection results, and interaction with users through mobile apps, and measures such as on-board ventilation equipment are activated.
It improves the accuracy and reliability of in-vehicle live detection, reduces false alarms and missed reports, promptly reminds users and reduces the safety risks brought by forgetting children or pets.
Smart Images
Figure CN119659516A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of liveness detection in an intelligent cockpit, and in particular to a method and device for liveness detection in a vehicle. Background Art
[0002] It's common for children and pets to be left behind in cars due to driver negligence. This is especially true in the hot summer months, when leaving children, pets, and cars in the car can have serious consequences. There are numerous cases of drivers and passengers negligently leaving children or pets in cars, resulting in dangerous situations. For the public, if this happens, the inability to contact the vehicle owner and obtain authorization to break the window could lead to unresolved dangers. Breaking the window without authorization could also lead to property disputes. Therefore, multi-sensor fusion in-car liveness detection and notification technology is essential. This system can maximize the safety of children and pets while minimizing property losses.
[0003] Existing in-vehicle liveness detection technologies are mostly based on single solutions, such as vision or single sensors. These single detection solutions have various drawbacks. Vision-based detection solutions are limited by the camera and lose some of their capabilities in dark environments, further impacting their accuracy. Single-sensor solutions, due to sensor limitations, can have significant errors.
[0004] The applicant believes that with the development of smart cockpits, the types of sensors in the cockpit have increased, and sensor data can be transmitted over the network, such as temperature sensors that remotely adjust the temperature inside the vehicle. Existing smart vehicles are generally equipped with corresponding mobile phone apps that can control the vehicle and obtain some vehicle information. At the same time, infrared sensor detection technology is relatively mature and has low power consumption, and can operate for a long time. Millimeter-wave radar can determine the presence of living things by detecting micro-movements of breathing. It is possible to combine the data of several existing sensors, and use a multi-source data fusion detection algorithm to control their operation mode and sequence, while determining the method of interaction with the mobile phone app, to form a multi-sensor-based in-vehicle liveness detection solution to ensure the safety of children or pets in the vehicle. Summary of the Invention
[0005] To solve at least one aspect of the above problems, the present invention provides a method for detecting liveness in a vehicle, comprising: obtaining the status of a driver or passenger, and starting an infrared sensor when the driver or passenger leaves the vehicle; collecting temperature information through the infrared sensor, and outputting a liveness detection result based on the temperature information using an infrared liveness detection method; when the liveness detection result is that a live body is present, starting a millimeter-wave radar, and collecting a respiratory micro-motion signal through the millimeter-wave radar; using a preset liveness detection model to output a model detection result based on the temperature information and the exhalation micro-motion signal; when the model detection result is that a live body is present, outputting the model detection result to a server, and outputting the model detection result to a mobile terminal through the server; receiving a confirmation instruction input by a user through the mobile terminal; obtaining the real-time temperature output by the in-vehicle sensor in response to the confirmation instruction input by the user; evaluating the real-time temperature according to a preset temperature threshold, and starting the vehicle ventilation equipment when the real-time temperature is greater than the preset temperature threshold.
[0006] Preferably, when the model detection result indicates that a living body is present, an in-vehicle image is captured by an onboard camera, and the in-vehicle image is output to a mobile terminal through a server.
[0007] Preferably, in response to a stop instruction input by a user, the infrared sensor and the millimeter wave radar are turned off.
[0008] Preferably, the model detection result is output to the server and the timer is started. When the timer time is greater than the preset time threshold and no user instruction is received, the real-time temperature output by the in-vehicle sensor is obtained, the real-time temperature is evaluated according to the preset temperature threshold, and the vehicle ventilation equipment is started when the real-time temperature is greater than the preset temperature threshold.
[0009] Preferably, when the model detection result indicates that a living body exists, a prompt text message is sent to the user through the server.
[0010] Preferably, the step of obtaining the driver and passenger status also includes: obtaining the vehicle power-off time and the door lock closing time, and when the vehicle power-off time is greater than or equal to a preset threshold and the door lock closing time is greater than or equal to a preset threshold, it is determined that the driver and passenger status is leaving the vehicle.
[0011] Preferably, the training method of the liveness detection model includes: obtaining a live sample data set of a live target, the live sample data set including infrared temperature information and millimeter wave radar respiratory micro-movement information; performing feature extraction and feature calibration on the sample data set; dividing the sample data set into a training set and a test set according to a preset ratio; inputting the training set into a random forest algorithm for training, wherein a random forest classifier is used to predict the type of live organisms; and testing the trained model with a test set until a preset accuracy rate is reached, thereby forming a sub-classification model of the live target; using live sample data sets of different live targets to perform model training separately, thereby forming multiple sub-classification models; and using a voting method to fuse multiple sub-classification models into a classification model.
[0012] On the other hand, an intravital detection device includes a memory and a processor, wherein the memory includes a computer program, and when the computer program is executed by the processor, the computer program executes any of the above methods.
[0013] The in-vehicle liveness detection method and device of the present invention have the following beneficial effects: They utilize a multi-sensor combination and a multi-source data fusion detection algorithm to fuse data from different sensors. The algorithm's results are then linked to the corresponding vehicle mobile app, effectively connecting user and vehicle information. This minimizes the occurrence of dangerous situations while facilitating immediate user awareness and mitigation. In-vehicle liveness detection and emergency notification push capabilities reduce the risk of children or pets being left behind in the vehicle and mitigate this risk through a variety of means. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] To better understand the above and other objects, features, advantages, and functions of the present invention, reference may be made to the embodiments shown in the accompanying drawings. Like reference numerals in the accompanying drawings refer to like components. Those skilled in the art should understand that the accompanying drawings are intended to illustrate preferred embodiments of the present invention by way of illustration and are not intended to limit the scope of the present invention. The components in the drawings are not drawn to scale.
[0015] Figure 1 A flowchart of a method for detecting a living body in a vehicle according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0016] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0017] As used herein, the term "including" and its variations represent open inclusion, i.e., "including but not limited to." Unless otherwise stated, the term "or" means "and / or." The term "based on" means "based at least in part on." The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment." The term "another embodiment" means "at least one additional embodiment." The terms "first," "second," etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.
[0018] In order to at least partially solve one or more of the above problems and other potential problems, the embodiments of the present disclosure provide a method for detecting a living body in a vehicle, such as Figure 1 As shown, it includes: step S1, obtaining the status of the driver and passenger, and starting the infrared sensor when the driver and passenger leave the vehicle.
[0019] Specifically, the in-vehicle liveness detection method is implemented using an in-vehicle liveness detection module installed within the cabin. The in-vehicle liveness detection module can determine the driver and occupant status based on vehicle information, or it can receive driver and occupant status input from a user via a mobile device.
[0020] In some embodiments, the step of obtaining the driver and passenger status also includes: obtaining the vehicle power-off time and the door lock closing time, and when the vehicle power-off time is greater than or equal to a preset threshold and the door lock closing time is greater than or equal to a preset threshold, it is determined that the driver and passenger status is leaving the vehicle.
[0021] The duration of vehicle inactivity is determined by the time the vehicle is powered off, and the door lock closing time is also obtained. The vehicle inactivity time and door lock closing time are evaluated using a preset threshold, such as a 5-minute threshold. When the vehicle power-off time and door lock closing time are greater than or equal to 5 minutes, the in-vehicle liveness detection module is activated to enable the detection algorithm. In some embodiments, the door lock closing time is measured from the time the vehicle is powered off. In other embodiments, the starting point of the door lock closing time is the door lock closing moment immediately following the last time the vehicle was powered off.
[0022] The in-vehicle living body detection function module activates the infrared sensor and the millimeter-wave radar in response to the received information that the driver or passenger has left the vehicle.
[0023] Step S2: collecting temperature information through an infrared sensor, and outputting a liveness detection result based on the temperature information using an infrared liveness detection method.
[0024] Specifically, the infrared sensor is used to collect temperature information inside the vehicle, and to determine whether there is a living body inside the vehicle based on the vehicle-borne temperature information using the existing infrared living body detection method.
[0025] Step S3: When the liveness detection result indicates that a live body is present, the millimeter-wave radar is activated to collect respiratory micro-movement signals.
[0026] Specifically, because infrared sensors consume little energy and can operate continuously, activating the millimeter-wave radar based on infrared liveness detection results can save operating costs and also reduce computing costs. The millimeter-wave radar is used to collect micro-breathing signals within the vehicle cabin.
[0027] Step S4: using a preset liveness detection model to output a model detection result based on the temperature information and the exhalation micro-motion signal.
[0028] Specifically, the in-vehicle liveness detection module also includes a liveness detection module. The preset liveness detection model is a classifier model trained by machine self-learning methods. By inputting temperature information and exhalation micro-movement signals into the liveness detection module, the liveness detection model outputs a model detection result. For example, children, pets, and the elderly, etc.
[0029] In some embodiments, the preset liveness detection model includes an infrared liveness detection model and a millimeter wave liveness detection model. The infrared liveness detection model and the millimeter wave liveness detection model adopt the neural network model of the existing technology. When the liveness detection results of the infrared liveness detection model and the millimeter wave liveness detection model are both that there is liveness, the output model detection result is that there is liveness.
[0030] In other embodiments, the training method of the liveness detection model includes: obtaining a live sample data set of a live target, the live sample data set including infrared temperature information and millimeter wave radar respiratory micro-movement information; performing feature extraction and feature calibration on the sample data set; dividing the sample data set into a training set and a test set according to a preset ratio; inputting the training set into a random forest algorithm for training, wherein a random forest classifier is used to predict the type of live organisms; and testing the trained model with a test set until a preset accuracy rate is reached, thereby forming a sub-classification model of the live target; using live sample data sets of different live targets to perform model training separately, thereby forming multiple sub-classification models; and using a voting method to fuse multiple sub-classification models into a classification model, i.e., a liveness detection model.
[0031] In another embodiment, the training method of the liveness detection model can be implemented using a Transformer architecture. First, a live sample data set containing infrared temperature information and millimeter-wave radar respiratory micro-motion information is obtained, and the live sample data set includes different live targets. The sample data is preprocessed, and the infrared temperature information and millimeter-wave radar respiratory micro-motion information are converted into feature vectors suitable for processing by the Transformer model through an embedding layer. Then, the multi-head self-attention mechanism is used to perform feature modeling on the sample data to capture the temporal features and spatial dependencies in the sample data. Subsequently, the expression of temporal and spatial information is enhanced through position encoding, and the processed feature vector is input into the encoder of the Transformer model for training to learn higher-order feature representations. Finally, the model verifies its performance through cross-validation and test sets until the preset accuracy is reached, thereby forming a comprehensive liveness detection model.
[0032] Step S5: When the model detection result indicates that a living body exists, the model detection result is output to the server, and the model detection result is output to the mobile terminal through the server.
[0033] Specifically, the server is a cloud server for the connected car, and the mobile device is a mobile phone, smart wearable device, etc. The mobile device is loaded with an application, and the user logs in to the vehicle and connects to the vehicle. When a living person is detected in the cabin, the server sends the model detection results to the mobile device to alert the user.
[0034] In another embodiment, in steps S3-S5, the in-vehicle liveness detection module uses an infrared liveness detection method to output a liveness detection result based on the temperature information. If the liveness detection result indicates the presence of a live person, the millimeter-wave radar is activated to collect respiratory micro-movement signals through the millimeter-wave radar, and the presence of a live person is determined using conventional respiratory micro-movement signal detection methods. If the millimeter-wave radar detection result indicates the presence of a live person, the model detection result is output to the server, and the server then outputs the model detection result to the mobile device.
[0035] Step S6: receiving a confirmation instruction input by the user through the mobile terminal.
[0036] Specifically, the user receives the model detection result through the mobile terminal, verifies it, and then enters the instruction through the mobile terminal. When the user confirms that there is a living body in the vehicle cabin, the user enters the confirmation instruction.
[0037] When the user confirms that there is no living body in the vehicle cabin, a stop command is input. In response to the stop command input by the user, the infrared sensor and the millimeter wave radar are turned off.
[0038] In some embodiments, when the model detection result indicates that a living body is present, an in-vehicle image is captured by an onboard camera, and the in-vehicle image is output to a mobile terminal through a server.
[0039] Specifically, the vehicle-mounted camera collects images inside the vehicle in response to information about the presence of living things, processes the collected images inside the vehicle using an image-sensitive information processing method of the prior art, and outputs the processed images to the mobile terminal through the server.
[0040] In some embodiments, when the model detection result indicates that a living body exists, a prompt text message is sent to the user through the server.
[0041] Step S7 : acquiring the real-time temperature output by the in-vehicle sensor in response to the confirmation instruction input by the user.
[0042] Specifically, the in-vehicle liveness detection function module receives a user confirmation instruction, connects to the vehicle-mounted temperature sensor based on the confirmation instruction, and obtains the real-time temperature through the vehicle-mounted temperature sensor.
[0043] Step S8: Evaluate the real-time temperature according to a preset temperature threshold, and start the vehicle ventilation device when the real-time temperature is greater than the preset temperature threshold.
[0044] Specifically, when the real-time temperature is greater than the preset temperature threshold, the vehicle will turn on the air conditioning and ventilation facilities (only electric vehicles) through remote control to lower the temperature inside the vehicle.
[0045] In some embodiments, the model detection result is output to the server and a timer is started. When the timer time is greater than a preset time threshold and no user instruction is received, the real-time temperature output by the in-vehicle sensor is obtained, the real-time temperature is evaluated according to a preset temperature threshold, and the vehicle ventilation equipment is started when the real-time temperature is greater than the preset temperature threshold.
[0046] Example 1
[0047] like Figure 1 As shown, when the vehicle stops and the doors are locked for more than five minutes, it is determined that the driver and passenger have left the vehicle.
[0048] After determining that the driver or occupant has indeed left the vehicle, the infrared sensor and millimeter-wave radar begin operating. The infrared sensor detects rapidly changing temperatures to determine the presence of living beings. However, the convection of hot and cold air can also cause the sensor to believe that a living being is present, so the millimeter-wave radar is used for fusion. The millimeter-wave radar detects microscopic breathing movements to determine whether a person is alive. The data obtained by the millimeter-wave radar is fused with the data from the infrared sensor to determine whether there are children or pets left in the vehicle.
[0049] If a child or pet is confirmed to have been left behind in the vehicle, the system will upload the information to the cloud via the internet. The cloud will then push the information to the vehicle owner via a mobile app or send a text message to remind them to confirm the situation. If a camera in the vehicle can capture an image of the relevant location (inside the vehicle only), the image will be sent to the owner for confirmation.
[0050] If there is an image, the detection will stop if the owner confirms that there are no children or pets left in the car.
[0051] If there is no image, or the owner has not confirmed, the vehicle will use the temperature sensor inside the vehicle. When the temperature reaches a certain set threshold (that is, when it may cause danger or harm to children or pets in the vehicle), the vehicle will remotely control the air conditioning and ventilation facilities (only electric vehicles) to lower the temperature inside the vehicle.
[0052] On the other hand, an intravital detection device includes a memory and a processor, wherein the memory includes a computer program, and when the computer program is executed by the processor, the computer program executes any of the above methods.
[0053] This invention utilizes a multi-sensor, multi-source data fusion solution, effectively resolving the issue of single-vision detection methods being affected by lighting conditions. It can also effectively detect the presence of living organisms in locations with poor lighting, such as the trunk. Furthermore, the invention combines multiple sensors to improve detection accuracy. By integrating the car and mobile phone, the invention can effectively and promptly alert the owner.
[0054] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand this document.
Claims
1. A method for detecting a living body in a vehicle, characterized in that: include: Get the status of the driver and passengers, and activate the infrared sensor when the driver and passengers leave the vehicle; Collect temperature information through an infrared sensor, and output a liveness detection result based on the temperature information using an infrared liveness detection method; When the liveness detection result indicates that a live body is present, the millimeter wave radar is started to collect a breathing micro-motion signal; Using a preset liveness detection model to output a model detection result based on the temperature information and the exhalation micro-motion signal; When the model detection result indicates that a living body exists, the model detection result is sent to a server, and the model detection result is sent to a mobile terminal through the server; Receiving a confirmation instruction input by the user through the mobile terminal; Responding to the confirmation instruction input by the user, acquiring the real-time temperature output by the in-vehicle sensor; The real-time temperature is evaluated according to a preset temperature threshold, and the vehicle ventilation device is started when the real-time temperature is greater than the preset temperature threshold.
2. The method according to claim 1, characterized in that When the model detection result is that a living body exists, the in-vehicle image is collected by the vehicle camera, and the in-vehicle image is output to the mobile terminal through the server.
3. The method according to claim 2, characterized in that In response to a stop command input by the user, the infrared sensor and the millimeter wave radar are turned off.
4. The method according to claim 3, characterized in that The model detection result is output to the server and a timer is started. When the timer time is greater than a preset time threshold and no user instruction is received, the real-time temperature output by the in-vehicle sensor is obtained, the real-time temperature is evaluated according to a preset temperature threshold, and the vehicle ventilation equipment is started when the real-time temperature is greater than the preset temperature threshold.
5. The method according to claim 1, characterized in that When the model detection result indicates that a living body is present, a prompt SMS is sent to the user through the server.
6. The method according to claim 1, characterized in that The step of obtaining the driver and passenger status also includes: obtaining the vehicle power-off time and the door lock closing time, and when the vehicle power-off time is greater than or equal to a preset threshold and the door lock closing time is greater than or equal to a preset threshold, it is determined that the driver and passenger status is leaving the vehicle.
7. The method according to claim 1, characterized in that The training method of the living body detection model includes: Acquire a live sample data set of a live target, where the live sample data set includes infrared temperature information and millimeter wave radar breathing micro-motion information; Perform feature extraction and feature calibration on the sample data set; Divide the sample data set into training set and test set according to the preset ratio; The training set is input into the random forest algorithm for training, wherein the random forest classifier is used to predict the type of living organisms, and the trained model is tested with the test set until a preset accuracy rate is reached, thereby forming a sub-classification model of the living organism target; Use live sample data sets of different live targets to train models separately, thereby forming multiple sub-classification models; Use voting method to merge multiple sub-classification models into a classification model.
8. A liveness detection device in a vehicle, characterized in that: A memory and a processor, wherein the memory comprises a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 7 is executed.