Methods, devices, computer equipment, and storage media for detecting objects falling from inside vehicles.

Abstract

This application relates to a method, device, computer equipment, storage medium, and computer program product for detecting objects falling from inside a vehicle. The method includes: activating an in-vehicle object falling detection function when the vehicle is in motion; if an object is detected falling from the vehicle, detecting the location of the falling object and acquiring an image of the in-vehicle environment; acquiring driver behavior change data based on the in-vehicle environment image; if the driver behavior change data indicates abnormal behavior, generating and pushing an object falling alarm message based on the location of the falling object. This method can accurately detect objects falling from inside a vehicle.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, device, computer equipment, storage medium, and computer program product for alarming objects falling inside a vehicle. Background Technology

[0002] With the development of artificial intelligence (AI) technology, technologies that use AI to assist in vehicle driving are emerging in large numbers. This means that AI can identify the situation inside the vehicle and issue warnings when abnormalities occur; this technology can reduce the risk of driving hazards, ensuring driver safety and enabling them to drive safely.

[0003] In traditional technologies, when an object falls from inside a vehicle during driving, the detection and perception of this situation are typically achieved using camera sensors, auxiliary LiDAR, or millimeter-wave radar sensors to regulate driving. However, this technology cannot accurately detect and perceive falling objects, thus failing to provide accurate warnings. Summary of the Invention

[0004] Therefore, it is necessary to provide an accurate method, device, computer equipment, computer-readable storage medium, and computer program product for detecting objects falling from inside vehicles, in order to address the aforementioned technical problems.

[0005] Firstly, this application provides a method for detecting objects falling inside a vehicle. The method includes:

[0006] When the vehicle is in motion, the in-vehicle object falling detection function is activated.

[0007] If an object is detected falling inside the vehicle, the location of the falling object is detected, and an image of the vehicle's interior environment is acquired.

[0008] Based on the in-vehicle environment image, obtain driver behavior change data;

[0009] If the driver behavior change data indicates abnormal behavior, then an object falling alarm message is generated and pushed based on the location of the object falling inside the vehicle.

[0010] In one embodiment, the object inside the vehicle is an object that has a temperature difference with the interior environment of the vehicle;

[0011] If an object is detected falling from inside the vehicle, the method for detecting the location of the falling object includes:

[0012] If an object is detected falling inside the vehicle, the infrared monitoring results inside the vehicle are obtained.

[0013] Based on the in-vehicle infrared monitoring results, the in-vehicle temperature distribution results are obtained;

[0014] Based on the temperature distribution inside the vehicle, the location where the object fell inside the vehicle was detected.

[0015] In one embodiment, the driver behavior change data includes driver head position change data;

[0016] The step of obtaining driver behavior change data based on the in-vehicle environment image includes:

[0017] An object detection algorithm is used to extract the driver's human body region image from the in-vehicle environment image;

[0018] Determine the head detection feature points in the driver's body region image;

[0019] Based on the head detection feature points, obtain the head detection feature point change data, and based on the head detection feature point change data, determine the driver's head posture change data.

[0020] Based on the head posture change data, driver behavior change data is obtained.

[0021] In one embodiment, the driver behavior change data includes driver eye change data;

[0022] The step of obtaining driver behavior change data based on the in-vehicle environment image includes:

[0023] A facial detection algorithm is used to extract the driver's facial image from the in-vehicle environment image, and then the driver's eye image is extracted from the driver's facial image;

[0024] Identify eye detection feature points in the driver's eye image;

[0025] Based on the eye detection feature points, obtain the change data of the eye detection feature points, and determine the driver's eye change data based on the change data of the eye detection feature points.

[0026] Based on the eye change data, driver behavior change data is obtained.

[0027] In one embodiment, it further includes:

[0028] When the vehicle is stationary and an object is detected falling, the location of the falling object inside the vehicle is detected;

[0029] Based on the location where the object fell inside the vehicle, an alarm message is generated and pushed out.

[0030] In one embodiment, it further includes:

[0031] Based on the in-vehicle environment image, obtain the image of the rear seat occupants and extract the facial images from the image of the rear seat occupants.

[0032] Based on the facial images of the rear seat occupants in the vehicle images, identify the facial expressions of the rear seat occupants.

[0033] When the facial expression of a rear-seat occupant in the vehicle indicates crying, an alarm message is generated and pushed based on the location where the object fell.

[0034] In one embodiment, recognizing the facial expressions of the rear-seat occupants based on their facial images in the vehicle rear-seat occupant image includes:

[0035] Obtain the initial convolutional neural network model, the training set of human face images, and the expression labels corresponding to the training set of human face images;

[0036] A facial feature localization algorithm is used to locate features in the training set of facial images to obtain the facial features corresponding to the training set of facial images.

[0037] Based on the facial features and the expression labels, the initial convolutional neural network model is trained to obtain a convolutional neural network model.

[0038] A facial feature localization algorithm is used to locate the facial features in the images of the rear seat occupants of the vehicle, and the feature-localized facial images are input into the convolutional neural network model to recognize the facial expressions of the rear seat occupants of the vehicle.

[0039] Secondly, this application also provides a vehicle interior object falling alarm device. The device includes:

[0040] The monitoring activation module is used to activate the in-vehicle object falling detection function when the vehicle is in motion;

[0041] The in-vehicle detection module is used to detect the location of the object falling inside the vehicle and acquire an image of the in-vehicle environment if an object is detected falling inside the vehicle.

[0042] The data analysis module is used to acquire driver behavior change data based on the in-vehicle environment image;

[0043] The alarm recognition module is used to generate and push an object falling alarm message based on the location of the falling object inside the vehicle if the driver behavior change data indicates abnormal behavior.

[0044] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0045] When the vehicle is in motion, the in-vehicle object falling detection function is activated.

[0046] If an object is detected falling inside the vehicle, the location of the falling object is detected, and an image of the vehicle's interior environment is acquired.

[0047] Based on the in-vehicle environment image, obtain driver behavior change data;

[0048] If the driver behavior change data indicates abnormal behavior, then an object falling alarm message is generated and pushed based on the location of the object falling inside the vehicle.

[0049] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0050] When the vehicle is in motion, the in-vehicle object falling detection function is activated.

[0051] If an object is detected falling inside the vehicle, the location of the falling object is detected, and an image of the vehicle's interior environment is acquired.

[0052] Based on the in-vehicle environment image, obtain driver behavior change data;

[0053] If the driver behavior change data indicates abnormal behavior, then an object falling alarm message is generated and pushed based on the location of the object falling inside the vehicle.

[0054] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0055] When the vehicle is in motion, the in-vehicle object falling detection function is activated.

[0056] If an object is detected falling inside the vehicle, the location of the falling object is detected, and an image of the vehicle's interior environment is acquired.

[0057] Based on the in-vehicle environment image, obtain driver behavior change data;

[0058] If the driver behavior change data indicates abnormal behavior, then an object falling alarm message is generated and pushed based on the location of the object falling inside the vehicle.

[0059] The aforementioned safe driving method, device, computer equipment, storage medium, and computer program product activate the in-vehicle object falling detection function when the vehicle is in motion. If an object is detected falling inside the vehicle, the system detects the object's location and acquires an image of the in-vehicle environment. Based on the in-vehicle environment image, it acquires driver behavior change data. If the driver behavior change data indicates abnormal behavior, it generates and pushes an object falling alarm message based on the object's location. Throughout this process, the object falling detection function is activated while the vehicle is in motion, and when an object is detected falling, the system identifies the object's location and, combined with the driver behavior change data acquired through the vehicle environment image, accurately pushes the object falling location when the driver behavior change data is abnormal. Attached Figure Description

[0060] Figure 1 This is a diagram illustrating the application environment of a vehicle interior object falling alarm method in one embodiment.

[0061] Figure 2 This is a flowchart illustrating a method for detecting objects falling from inside a vehicle in one embodiment.

[0062] Figure 3 This is a flowchart illustrating a vehicle interior object falling alarm method in another embodiment;

[0063] Figure 4 This is a schematic diagram of a vehicle interior pressure sensor and infrared monitor in a specific application example.

[0064] Figure 5 This is a schematic diagram of an in-vehicle camera in another specific application example;

[0065] Figure 6 This is a structural block diagram of a vehicle interior object falling alarm device in one embodiment;

[0066] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] The vehicle interior object falling alarm method provided in this application embodiment can be applied to, for example... Figure 1The application environment shown is as follows. Within the vehicle, there is a monitor 102 and a vehicle controller 104, which communicate with each other. The monitor 102 includes pressure sensors, infrared sensors, vision sensors, etc. A data storage system can store the data that the vehicle controller 104 needs to process. The data storage system can be integrated into the vehicle controller 104, or it can be located in the cloud or on another networked vehicle controller.

[0069] When the vehicle is in motion, the vehicle controller 104 acquires vehicle motion status messages. When the vehicle motion status messages indicate that the vehicle is in motion, the control monitor 102 activates the in-vehicle object falling detection function. First, the pressure sensor in the control monitor 102 monitors for objects falling inside the vehicle. If the monitor 102 detects an object falling, it sends the object falling message to the vehicle controller 104. The vehicle controller 104 then activates other monitors 102 on the vehicle, such as infrared monitors and vision sensors, to detect the location of the falling object and acquire images of the in-vehicle environment. Based on the in-vehicle environment images, the vehicle controller 104 acquires driver behavior change data. If the driver behavior change data indicates abnormal behavior, it generates and pushes an object falling alarm message based on the location of the falling object. The object falling alarm message can be pushed to a terminal. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc.

[0070] In one embodiment, such as Figure 2 As shown, a method for detecting objects falling inside a vehicle is provided, which can be applied to... Figure 1 Taking the vehicle controller 104 as an example, the following steps are included:

[0071] S200: When the vehicle is in motion, the in-vehicle object falling detection function is activated.

[0072] Specifically, various monitors are present on the vehicle. When the vehicle is in motion, it sends a motion status message to the vehicle controller. The vehicle controller receives the message and, when it indicates that the vehicle is in motion, activates the in-vehicle object falling detection function, controlling the on-vehicle monitors to detect falling objects. Furthermore, these monitors can be pressure sensors, which obtain pressure change data based on the deformation or change of an object under pressure. Pressure sensors include strain sensors, magnetic sensors, piezoresistive sensors, and pressure sensors. Moreover, more than one pressure sensor can be installed inside the vehicle to achieve full coverage of the pressure sensor monitoring range on the vehicle's floor.

[0073] If the S400 detects an object falling inside the vehicle, it will detect the location of the falling object and acquire an image of the vehicle's interior environment.

[0074] Specifically, when an object falls onto the floor inside the vehicle, the pressure sensor on the floor inside the vehicle acquires pressure change data and uploads the pressure change data to the vehicle controller. When the vehicle controller receives the pressure change data, it confirms that an object has fallen inside the vehicle. At this time, it can detect the location where the object fell inside the vehicle and acquire an image of the interior environment.

[0075] Furthermore, the in-vehicle environment image can be acquired using other monitors within the vehicle, such as vision sensors, including cameras, and the vision sensor may include, but is not limited to, one.

[0076] The S600 acquires driver behavior change data based on images of the in-vehicle environment.

[0077] Specifically, when a vision sensor is installed in the front of the vehicle, it can acquire images of the driver and the surrounding environment during driving. Based on these images, the driver's behavior can be identified to obtain driver behavior change data. Furthermore, the driver behavior change data includes driver eye movement changes and head posture changes. The eye movement changes include the eye's gaze angle and its dynamic changes, as well as the geometric movement characteristics of the eyeballs. The head posture changes include changes in the driver's head position and direction.

[0078] S800: If the driver's behavior change data indicates abnormal behavior, then an object falling alarm message will be generated and pushed according to the location of the object falling inside the vehicle.

[0079] Specifically, driver behavior change data is analyzed, and a legal threshold for driver behavior change data is preset. When the driver behavior change data exceeds this threshold, it is considered to represent abnormal behavior. At this time, an object falling alarm message is generated and pushed according to the location of the object falling inside the vehicle. In other words, in this embodiment, pushing an object falling alarm message requires three conditions to be met: the vehicle is in motion, an object has fallen inside the vehicle, and the driver behavior change data is abnormal. When these conditions are met, an alarm message can be pushed. The alarm message includes alarm voice, alarm SMS, etc. The alarm message content can be "Your object has now fallen under the driver's seat. Please drive safely or pull over."

[0080] In the aforementioned method for detecting objects falling from inside the vehicle, the in-vehicle object falling detection function is activated when the vehicle is in motion. If an object is detected falling, the location of the falling object is detected, and an image of the in-vehicle environment is acquired. Based on the in-vehicle environment image, driver behavior change data is obtained. If the driver behavior change data indicates abnormal behavior, an object falling alarm message is generated and pushed based on the location of the falling object. Throughout the process, the object falling detection function can be activated while the vehicle is in motion, and when an object is detected falling, the location of the falling object is identified. Combined with the driver behavior change data acquired through the vehicle environment image, the accurate push of the object falling location is achieved when the driver behavior change data is abnormal.

[0081] In one embodiment, the method of using pressure sensors to monitor objects falling inside the vehicle is as follows:

[0082] 1. Strain Sensor: A strain sensor measures the pressure on an object when it is dropped by using the elastic deformation of a metal or other material. When a strain sensor is subjected to pressure, it will undergo a small strain, and the strain sensor calculates the pressure value by measuring the strain.

[0083] 2. Magnetic Sensor: A magnetic sensor uses changes in a magnetic field to measure the pressure exerted when an object falls. When an object is subjected to pressure, the magnetism of the magnetic material changes, and the magnetic sensor can determine the pressure value by detecting these changes in the magnetic field.

[0084] 3. Piezoresistive Sensor: A piezoresistive sensor uses changes in resistance to measure the pressure exerted when an object is dropped. Inside the sensor is a special resistive element; when pressure is applied, the resistance of this element changes, and the pressure information is obtained by measuring this change in resistance.

[0085] 4. Piezoelectric Sensor: A piezoelectric sensor uses the piezoelectric effect to measure pressure. Some materials undergo charge separation when subjected to pressure, and the pressure value is obtained by detecting the change in charge.

[0086] In one embodiment, the object inside the vehicle is an object that has a temperature difference with the vehicle's interior environment;

[0087] If an object is detected falling from inside the vehicle, the location of the falling object will be detected as follows:

[0088] If an object is detected falling inside the vehicle, the infrared monitoring results inside the vehicle are obtained; based on the infrared monitoring results inside the vehicle, the temperature distribution inside the vehicle is obtained; based on the temperature distribution results inside the vehicle, the location where the object fell inside the vehicle is detected.

[0089] Specifically, there are many types of objects that can fall inside a vehicle. To more accurately detect the location of a falling object, it can be categorized based on the temperature difference between the object and the vehicle's interior environment. When an object has a significant temperature difference with the environment, its location can be determined by analyzing the temperature distribution between the object and the environment. Furthermore, to determine this temperature distribution, an infrared detector can be used to monitor the vehicle's interior, obtaining infrared monitoring results, and then the temperature distribution within the vehicle can be calculated.

[0090] Furthermore, the steps for using an infrared monitor to monitor the in-vehicle environment and obtain the in-vehicle infrared monitoring results include:

[0091] 1. To more accurately determine whether an object has fallen from inside the vehicle and to avoid errors in the pressure signals detected by the pressure sensors, an infrared detector is first used to reconfirm whether an object has fallen inside the vehicle. Specifically, if an object has fallen inside the vehicle, it will usually emit infrared radiation. The infrared detector can detect the infrared radiation emitted by the object and determine the presence or absence of the object based on changes in the amount of radiation.

[0092] 2. After confirming the presence of a fallen object inside the vehicle, an infrared detector emits an infrared beam to the vehicle floor. When the infrared beam reaches the object on the floor, the object reflects some of the infrared light back. After receiving the beam reflected by the object, the infrared detector can determine the object's location by analyzing parameters such as light intensity and time delay.

[0093] In addition, some infrared detectors can also perform infrared imaging, that is, generate thermal images or infrared images based on the infrared radiation intensity at different locations of an object. By analyzing the distribution of hot spots in the image, the location of objects that have fallen into a non-living vehicle can be identified.

[0094] In this embodiment, by using an infrared monitor to accurately detect objects falling inside the vehicle, the vehicle interior temperature distribution can be obtained through the infrared monitoring results obtained by the infrared monitor, and the location of objects falling inside the vehicle can be accurately detected based on the vehicle interior temperature distribution results.

[0095] In one embodiment, when the object that falls from inside the vehicle has a small or no temperature difference with the vehicle's interior environment, the following method can be used to detect the location where the object fell:

[0096] 1. Passive Infrared Technology. Passive infrared technology utilizes the property of objects to absorb and radiate heat. Even if an object does not radiate significant heat, it can be detected by the temperature difference between the object and its surrounding environment.

[0097] 2. Other auxiliary technologies. The ability to detect cold objects can be improved by combining infrared detectors with other sensors or technologies. For example, a visible light camera can be used in conjunction with an infrared detector for dual detection, or ultrasonic or radar sensors can be used to enhance the detection range and accuracy of cold objects.

[0098] In this embodiment, the location of the falling object is determined by detecting whether the object has a small or no temperature difference with the vehicle's interior environment. This allows for location detection of all falling objects inside the vehicle, resulting in a more comprehensive detection of objects falling inside the vehicle.

[0099] In one embodiment, driver behavior change data includes driver head position change data; such as Figure 3 As shown, S600 includes:

[0100] The S620 uses an object detection algorithm to extract the driver's human body region image from the in-vehicle environment image.

[0101] Specifically, since the in-vehicle environment image is a series of changing frames, the driver's behavioral changes can be detected and measured in real time based on the in-vehicle environment image acquired by the visual sensor. When the driver's behavioral changes are in the form of head pose changes, a pose tracking algorithm can be used to obtain the driver's head pose changes based on the in-vehicle environment image. In this case, preprocessing of the in-vehicle environment image is first required, i.e., determining the driver's body region image from the in-vehicle environment image. Methods for determining the driver's body region image include using object detection algorithms. Object detection algorithms include deep learning-based models, Haar cascade classifiers, etc. Among them, the Haar cascade classifier refers to the Adaboost cascade face detection classifier based on Haar features, which can accurately detect the driver's body region image.

[0102] S640, determine the head detection feature points in the driver's body region image.

[0103] Specifically, since the driver's body region image is a large-scale image, specific points can be selected from the driver's body region image as detection feature points, making the monitoring of driver change data more accurate. This application requires determining the driver's head pose change data; therefore, specific head detection feature points can be selected from the head image of the driver's body region image. Furthermore, when selecting head detection feature points, a feature point detection algorithm can be used for keypoint detection. In practical applications, feature point detection algorithms typically include SIFT (Scale Invariant Feature Transform) and HOG (Histogram of Oriented Gradient) algorithms to obtain head detection feature points. Alternatively, deep learning-based methods can also be used for keypoint detection, including OpenPose for human pose recognition and HRNet (High-Resolution Network) for high-resolution network modeling.

[0104] S660 acquires head detection feature point change data based on head detection feature points, and determines the driver's head posture change data based on the head detection feature point change data.

[0105] Specifically, after determining the head detection feature points, these feature points are used as the characteristic points for changes in the driver's head. Head detection feature point change data is acquired, including changes in the position and orientation of the detection feature points. Based on this data, the driver's head pose change data is determined. Further, acquiring the head detection feature point change data can involve obtaining pose change angles or pose change matrices. Methods for obtaining pose change angles or pose change matrices include using mathematical models such as rigid body motion models and joint connection models. Additionally, machine learning algorithms can be used to classify or regress the head detection feature point change data to determine the driver's head pose change data.

[0106] Furthermore, in practical applications, the posture tracking algorithm is a continuous process. It can obtain the driver's head posture change data by taking frame-by-frame images of the in-vehicle environment; or it can select the first few frames of in-vehicle environment images to build a model or rule system to track the driver's head posture and continuously update the head posture change data.

[0107] S680 obtains driver behavior change data based on head position change data.

[0108] Specifically, when the driver's behavior change data is head position change data, the driver's behavior change data can be obtained based on the head position change data.

[0109] In this embodiment, by employing a target detection algorithm, the driver's human body region image can be accurately extracted from the in-vehicle environment image. This avoids interference from complex backgrounds on the driver's head pose detection. Furthermore, the driver's head pose change data is obtained by acquiring head detection feature point change data, rather than directly acquiring the driver's head pose change data. This makes the process of acquiring the driver's head pose change data more efficient and accurate.

[0110] In one embodiment, driver behavior change data includes driver eye change data;

[0111] Based on in-vehicle environment images, driver behavior change data is obtained, including:

[0112] A facial detection algorithm is used to extract the driver's facial image from the in-vehicle environment image, and then extract the driver's eye image from the driver's facial image. Eye detection feature points are determined in the driver's eye image. Based on the eye detection feature points, change data of the eye detection feature points are obtained, and based on the change data of the eye detection feature points, the driver's eye change data is determined. Based on the eye change data, the driver's behavior change data is obtained.

[0113] Specifically, when the driver behavior change data is the driver's eye change data, the driver's eye change data is obtained by using an eye-tracking algorithm based on the in-vehicle environment image.

[0114] First, the in-vehicle environment image is preprocessed, specifically to extract the driver's eye image. The extraction method involves using a face detection algorithm to locate the facial image region, and then employing a method based on human eye features to determine the eye region image. Further, the face detection algorithms include the Haar cascade classifier and the HOG-SVM (Histogram of Oriented Gradient-Support Vector Machine) algorithm. HOG-SVM is a machine learning method that uses the SVM algorithm to classify HOG features, thereby achieving the object detection task. The method based on human eye features uses a circular or elliptical model to approximate the shape of the eyeball and utilizes color information or texture features to distinguish the eyes from other regions.

[0115] Secondly, the eye detection feature points in the eye image are identified. These feature points include the pupil, eyeball, and cornea. Based on these feature points, data on their changes are obtained. For example, by analyzing features such as the shape, color, and texture of the eyeball, information about eye movement and gaze direction can be extracted. Specifically, mathematical models (such as geometric or statistical models) can be used to calculate the gaze direction, which is then used as eye change data. Alternatively, machine learning algorithms can be used for classification or regression to obtain eye change data. Based on this eye change data, driver behavior change data can then be derived.

[0116] In practical applications, eye-tracking algorithms are a continuous process. They can obtain driver eye change data by taking frames of in-vehicle environment images one by one; or they can select the first few frames of in-vehicle environment images to build a model or rule system to track the driver's eyes and continuously update the eye change data.

[0117] In this embodiment, by employing a facial feature detection algorithm and a method based on human eye features to extract eye images from the in-vehicle environment image, interference from other areas in the in-vehicle environment image on the monitoring of eye change data can be avoided. Furthermore, eye change data of the driver can be indirectly obtained by detecting eye feature points, making the process of obtaining driver eye change data more efficient and accurate.

[0118] In one embodiment, it also includes:

[0119] When the vehicle is stationary and an object is detected falling, the location of the falling object inside the vehicle is detected; based on the location of the falling object inside the vehicle, an alarm message is generated and pushed.

[0120] Specifically, when the vehicle is stationary, regardless of whether a driver is inside, there is no need to monitor driver behavior data. When a pressure sensor detects an object falling inside the vehicle, the control sensor detects the object's position and location. Based on the object's location, an alarm message is generated and pushed out; the alarm message could be something like, "Your object has now fallen under the driver's seat." Furthermore, when the object inside the vehicle has a temperature difference from the interior environment, the sensor can be an infrared detector.

[0121] In this embodiment, by analyzing the falling of objects when the vehicle is stationary, an accurate alarm for falling objects inside the vehicle can be achieved when the vehicle is stationary.

[0122] In one embodiment, it also includes:

[0123] Based on the in-vehicle environment image, obtain the image of the rear seat occupant and extract the face image from the rear seat occupant image; based on the face image of the rear seat occupant, identify the facial expression of the rear seat occupant; when the facial expression of the rear seat occupant indicates crying, generate and push an alarm message according to the location of the object falling.

[0124] Specifically, when the vehicle is in motion, i.e., while driving, if an object is detected falling, the system can control visual sensors to acquire images of the in-vehicle environment. Furthermore, there is more than one type of visual sensor. When the visual sensor is located in the front row, it monitors the driver and the surrounding environment. When the visual sensor is located in the rear row, it acquires images of the rear-seat occupants. The system analyzes these images, using facial expression analysis algorithms to determine if the occupants are crying. If the facial expression indicates crying, an alarm message is generated and pushed based on the object's location. The alarm message could be something like, "Your object is located under the middle rear seat; please don't worry about losing it."

[0125] Furthermore, it can recognize the facial expressions of rear-seat occupants not only when the vehicle is in motion, but also when the vehicle is stationary.

[0126] In this embodiment, by analyzing the facial expressions of the rear-seat occupants, accurate vehicle object falling alarms can be pushed based on data on changes in driver behavior and the facial expressions of the rear-seat occupants.

[0127] In one embodiment, recognizing facial expressions of rear-seat occupants based on facial images in images of rear-seat occupants includes:

[0128] The process involves obtaining an initial convolutional neural network (CNN) model, a training set of facial images, and corresponding expression labels. A facial feature localization algorithm is then used to locate features in the training set of facial images, yielding the corresponding facial features. Based on these facial features and expression labels, the initial CNN model is trained to obtain the final CNN model. Finally, a facial feature localization algorithm is used to locate facial features in images of rear-seat occupants, and the localized facial images are input into the CNN model to recognize the facial expressions of rear-seat occupants.

[0129] Specifically, based on facial images, deep learning models and facial feature localization algorithms can be used to recognize facial expressions. The deep learning model used in this application is a convolutional neural network (CNN) model. The specific training steps of the CNN model include:

[0130] 1. Data Collection and Preparation: Obtain the initial convolutional neural network model and collect facial images containing different expressions. Label these images and associate each facial expression with its corresponding label to ensure the diversity and representativeness of the dataset. This involves obtaining a training set of human face images and the corresponding expression labels for those images.

[0131] 2. Facial Feature Localization: Facial feature localization algorithms are used to detect and locate key feature points in facial images within a training set, obtaining the facial features corresponding to the training set, such as eyes, eyebrows, nose, and mouth. In practical applications, facial feature localization algorithms include Dlib, MTCNN (Multi-task Cascaded Convolutional Networks), and OpenCV (Open Source Computer Vision Library).

[0132] 3. Model Training: Facial features and expression labels are input into a convolutional neural network (CNN). The CNN extracts feature vectors from the facial features and trains the initial CNN model based on these feature vectors and expression labels, resulting in a new CNN model. The CNN models used in this application include VGG (Visual Geometry Group), ResNet (Residual Network), and Inception. Furthermore, common deep learning frameworks such as TensorFlow and PyTorch can be used to build and train the model.

[0133] 4. Model evaluation and optimization: A validation set is also needed to evaluate the trained model. Based on the evaluation results, parameter tuning and optimization are performed to improve the model's accuracy and generalization ability.

[0134] Convolutional neural network (CNN) models can output the facial expressions corresponding to the input face image. Therefore, a facial feature localization algorithm is used to locate the facial features in the image of the rear-seat occupant of the vehicle, resulting in a feature-localized face image. This feature-localized face image is then input into the CNN model, which outputs the facial expressions of the rear-seat occupant.

[0135] In this embodiment, a convolutional neural network model and a facial feature localization algorithm are used to identify the facial expressions of passengers in the back seat of a vehicle, making the recognition process more efficient and accurate.

[0136] In one embodiment, such as Figure 4 As shown, the vehicle has four seats: two in the front and two in the back. Each seat has pressure sensors A1, B1, C1, and D1, and infrared monitors A2, B2, C2, and D2 installed on the floor. Figure 5 As shown, the vehicle seats are divided by the A-pillar, B-pillar, and C-pillar. The visual sensors are cameras. Visual sensor A3 is installed on the A-pillar on the left side of the driver in the front seat, and visual sensor B3 is installed on the B-pillar next to the rear seat. The installation location of the visual sensors is not limited; they can also be installed next to the C-pillar. Taking a rear passenger including a child playing with a toy as an example, the methods for detecting objects falling inside the vehicle include:

[0137] 1. When the vehicle is in motion:

[0138] When a child's toy falls onto the floor under the middle rear seat, the pressure sensor on the floor detects the pressure change data and sends it to the vehicle controller. Since the toy that the child dropped is a toy with a temperature difference from the car's interior environment, the vehicle controller triggers the infrared monitor to detect the state and location of the object that fell into the car, and triggers camera A to monitor the driver's body area and camera B to monitor the rear seat occupants.

[0139] The vehicle controller acquires images of the driver's body area and images of the rear seat occupants.

[0140] The vehicle controller uses images of the driver's body area to detect and measure in real time the geometric and movement features of the driver's eyelids and eyeballs, the gaze angle and its dynamic changes, and changes in head position and direction. It detects changes in the driver's head posture or eyes. If the changes exceed the legal threshold, the driver's behavior changes are considered abnormal. At this time, an alarm message is triggered to the driver: "Your object has now fallen under the middle rear seat. Please drive safely or pull over."

[0141] The vehicle controller identifies the facial expressions of rear-seat occupants, such as children, based on the images of the rear-seat occupants. If the child's facial expression indicates crying, an alarm message is generated and pushed based on the location of the dropped object: "Your toy is under the middle seat in the back row. Please don't worry about losing it." This alarm message helps the child stop crying.

[0142] Therefore, a necessary condition for sending an alarm message during vehicle operation is the detection of an object falling inside the vehicle. Additionally, an object-falling alarm message can be sent when driver behavior data is abnormal, or when a rear-seat passenger's facial expression indicates crying. Abnormal driver behavior data includes abnormal changes in the driver's head posture or eye movements.

[0143] 2. When the vehicle is stationary:

[0144] When a child's toy falls onto the floor under the middle rear seat, a pressure sensor detects the fall. The vehicle controller then activates an infrared sensor to acquire the infrared monitoring results inside the vehicle. Based on these results, the system obtains the temperature distribution inside the vehicle and determines the location of the fallen object. Finally, based on the location of the fallen object, an alert message is generated and pushed to the system: "Your object has now fallen under the middle rear seat."

[0145] At this time, the vehicle controller can also control camera B to monitor the images of the rear seat occupants. If the child's facial expression in the rear seat occupant image indicates crying, an alarm message will be generated and pushed according to the location where the object fell: "Your toy is under the middle seat in the back row. Please don't worry about losing it." This will allow the child to stop crying after receiving the alarm message.

[0146] Therefore, when the vehicle is stationary, the necessary condition for pushing an alarm message is the detection of an object falling inside the vehicle. Once an object falling inside the vehicle is detected, an alarm message can be pushed. If, after detecting an object falling inside the vehicle, a facial expression indicating crying is also detected in the image of a rear-seat occupant, an alarm message will be pushed again.

[0147] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0148] Based on the same inventive concept, this application also provides a vehicle interior object falling alarm device for implementing the above-mentioned vehicle interior object falling alarm method. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more vehicle interior object falling alarm device embodiments provided below can be found in the limitations of the vehicle interior object falling alarm method above, and will not be repeated here.

[0149] In one embodiment, such as Figure 6As shown, a vehicle interior object falling alarm device is provided, including: a monitoring activation module 200, an in-vehicle detection module 400, a data analysis module 600, and an alarm recognition module 800, wherein:

[0150] The monitoring activation module 200 is used to activate the in-vehicle object falling detection function when the vehicle is in motion.

[0151] The in-vehicle detection module 400 is used to detect the location of the object falling inside the vehicle and acquire an image of the in-vehicle environment if an object is detected falling inside the vehicle.

[0152] The data analysis module 600 is used to acquire driver behavior change data based on images of the in-vehicle environment;

[0153] The alarm recognition module 800 is used to generate and push an object falling alarm message based on the location of the object falling inside the vehicle if the driver's behavior change data indicates abnormal behavior.

[0154] In one embodiment, the object inside the vehicle is an object with a temperature difference from the vehicle interior environment; the in-vehicle detection module 400 is also used to acquire the in-vehicle infrared monitoring result if the in-vehicle object is detected to have fallen; acquire the in-vehicle temperature distribution result based on the in-vehicle infrared monitoring result; and detect the location where the in-vehicle object fell based on the in-vehicle temperature distribution result.

[0155] In one embodiment, the driver behavior change data includes driver head pose change data; the data analysis module 600 is further configured to extract the driver's human body region image from the in-vehicle environment image using a target detection algorithm; determine head detection feature points in the driver's human body region image; obtain head detection feature point change data based on the head detection feature points, and determine the driver's head pose change data based on the head detection feature point change data; and obtain driver behavior change data based on the head pose change data.

[0156] In one embodiment, driver behavior change data includes driver eye change data; the data analysis module 600 is further configured to extract the driver's facial image from the in-vehicle environment image using a facial detection algorithm, and extract the driver's eye image from the driver's facial image; determine eye detection feature points in the driver's eye image; obtain eye detection feature point change data based on the eye detection feature points, and determine the driver's eye change data based on the eye detection feature point change data; and obtain driver behavior change data based on the eye change data.

[0157] In one embodiment, the vehicle in-vehicle object falling alarm device further includes a vehicle stationary object falling module, which is used to detect the falling location of the object inside the vehicle when the vehicle is stationary and an object is detected falling; and generate and push an alarm message based on the falling location of the object inside the vehicle.

[0158] In one embodiment, the in-vehicle object falling alarm device further includes a rear seat occupant analysis module. The rear seat occupant analysis module is used to acquire images of the rear seat occupants based on images of the in-vehicle environment and extract facial images from the images of the rear seat occupants; to identify facial expressions of the rear seat occupants based on the facial images of the rear seat occupants; and to generate and push an alarm message when the facial expression of the rear seat occupant indicates crying, based on the location where the object fell.

[0159] In one embodiment, the rear-seat occupant analysis module is further configured to acquire an initial convolutional neural network model, a training set of facial images, and expression labels corresponding to the training set of facial images; use a facial feature localization algorithm to perform feature localization on the training set of facial images to obtain facial features corresponding to the training set of facial images; train the initial convolutional neural network model based on the facial features and expression labels to obtain a convolutional neural network model; use a facial feature localization algorithm to perform feature localization on the facial images in the rear-seat occupant images of the vehicle, and input the feature-localized facial images into the convolutional neural network model to recognize the facial expressions of the rear-seat occupants of the vehicle.

[0160] The various modules in the aforementioned vehicle in-vehicle object falling alarm device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0161] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores data such as images of the vehicle's interior environment. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a method for detecting objects falling inside a vehicle.

[0162] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0163] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0164] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0165] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0166] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0167] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0168] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0169] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for issuing an alarm for objects falling inside a vehicle, characterized in that, The method includes: When the vehicle is in motion, the in-vehicle object falling detection function is activated; the in-vehicle object refers to an object that has a temperature difference with the in-vehicle environment; If an object is detected falling inside the vehicle, the infrared monitoring results inside the vehicle are obtained; based on the infrared monitoring results inside the vehicle, the temperature distribution inside the vehicle is obtained; based on the temperature distribution results inside the vehicle, the location where the object fell inside the vehicle is detected; and an image of the interior environment of the vehicle is obtained. Based on the in-vehicle environment image, obtain driver behavior change data; If the driver behavior change data indicates abnormal behavior, then an object falling alarm message is generated and pushed based on the location of the object falling inside the vehicle.

2. The method according to claim 1, characterized in that, The driver behavior change data includes driver head position change data; The step of obtaining driver behavior change data based on the in-vehicle environment image includes: An object detection algorithm is used to extract the driver's human body region image from the in-vehicle environment image; Determine the head detection feature points in the driver's body region image; Based on the head detection feature points, obtain the head detection feature point change data, and based on the head detection feature point change data, determine the driver's head posture change data. Based on the head posture change data, driver behavior change data is obtained.

3. The method according to claim 1, characterized in that, The driver behavior change data includes driver eye change data; The step of obtaining driver behavior change data based on the in-vehicle environment image includes: A facial detection algorithm is used to extract the driver's facial image from the in-vehicle environment image, and then the driver's eye image is extracted from the driver's facial image; Identify eye detection feature points in the driver's eye image; Based on the eye detection feature points, obtain the change data of the eye detection feature points, and determine the driver's eye change data based on the change data of the eye detection feature points. Based on the eye change data, driver behavior change data is obtained.

4. The method according to claim 1, characterized in that, Also includes: Based on the in-vehicle environment image, obtain the image of the rear seat occupants and extract the facial images from the image of the rear seat occupants. Based on the facial images of the rear seat occupants in the vehicle images, identify the facial expressions of the rear seat occupants. When the facial expression of a rear-seat occupant in the vehicle indicates crying, an alarm message is generated and pushed based on the location where the object fell.

5. The method according to claim 4, characterized in that, The step of recognizing facial expressions of rear-seat occupants based on their facial images in the vehicle rear-seat occupant images includes: Obtain the initial convolutional neural network model, the training set of human face images, and the expression labels corresponding to the training set of human face images; A facial feature localization algorithm is used to locate features in the training set of facial images to obtain the facial features corresponding to the training set of facial images. Based on the facial features and the expression labels, the initial convolutional neural network model is trained to obtain a convolutional neural network model. A facial feature localization algorithm is used to locate the facial features in the images of the rear seat occupants of the vehicle, and the feature-localized facial images are input into the convolutional neural network model to recognize the facial expressions of the rear seat occupants of the vehicle.

6. A vehicle interior object falling alarm device, characterized in that, The device includes: The monitoring activation module is used to activate the in-vehicle object falling detection function when the vehicle is in motion; the in-vehicle object is an object that has a temperature difference with the in-vehicle environment; The in-vehicle detection module is used to acquire in-vehicle infrared monitoring results if an object is detected falling inside the vehicle; acquire in-vehicle temperature distribution results based on the in-vehicle infrared monitoring results; detect the location where the object fell based on the in-vehicle temperature distribution results; and acquire an in-vehicle environment image. The data analysis module is used to acquire driver behavior change data based on the in-vehicle environment image; The alarm recognition module is used to generate and push an object falling alarm message based on the location of the falling object inside the vehicle if the driver behavior change data indicates abnormal behavior.

7. The apparatus according to claim 6, characterized in that, The driver behavior change data includes driver head position change data; the data analysis module is also used to extract the driver's human body region image from the in-vehicle environment image using a target detection algorithm; The system identifies head detection feature points in the driver's body region image; based on these head detection feature points, it acquires head detection feature point change data and determines the driver's head pose change data; and based on the head pose change data, it acquires the driver's behavior change data.

8. The apparatus according to claim 6, characterized in that, The driver behavior change data includes driver eye change data; the data analysis module is also used to extract the driver's facial image from the in-vehicle environment image using a facial detection algorithm, and to extract the driver's eye image from the driver's facial image; and to determine eye detection feature points in the driver's eye image; Based on the eye detection feature points, obtain the change data of the eye detection feature points, and determine the driver's eye change data based on the change data of the eye detection feature points; based on the eye change data, obtain the driver's behavior change data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images