Methods for detecting and locating living beings in a vehicle interior and vehicle

Abstract

Method for detecting and localizing living beings in a vehicle interior, wherein at least one camera (1) and one radio sensor (2) each detect at least one section of the vehicle interior, sensor data generated by the camera (1) and the radio sensor (2) are processed by a computing unit (3), and the issuing of a warning and / or the control of a vehicle function depending on a living being detected in the vehicle interior can be initiated by the computing unit (3), wherein: - the computing unit (3) examines the sensor data generated by the radio sensor (2) for the presence of a breathing pattern and / or a heartbeat, locates a breathing pattern and / or heartbeat in the vehicle interior if one is detected, and marks the respective location as a region of interest (4.1); - the computing unit (3) in the sensor data generated by the camera (1) for the respective region of interest (4.1) determines correlating areas (5) and examines these areas (5) for the presence of an adult, a child, and an animal using image recognition methods; and- issues a warning and / or activates the vehicle function if at least one child and / or animal but no adult is detected in the vehicle interior, and suppresses the issuing of the warning and / or the activation of the vehicle function if at least one adult is detected in the vehicle interior, characterized in that- the computing unit (3) marks an installation position of the radio sensor (2) as a further region of interest (4.2) and deactivates the radio sensor (2) if the computing unit (3), by processing the sensor data generated by the camera (1), detects an object or living being in the further region of interest (4.2).

Description

The invention relates to a method for detecting and locating living things in a vehicle interior in the manner defined in more detail in the preamble of claim 1 and to a vehicle in the manner defined in more detail in the preamble of claim 2.In the everyday life, children or animals may forget or be intentionally left behind in the vehicle interior. For example, a dog owner would like to make a short purchase but would not like to take his dog. Especially in the case of strong solar radiation in summer and cold temperatures in winter, life-threatening situations can hereby arise, since the vehicle interior can heat up or cool down sharply, whereby a corresponding child or pet threatens to overheat or undercool. Thus, there is a need for means and methods for alerting vehicle users of children or animals left in the vehicle.Such methods and means are known, for example, from EP 3 855 212 A1. The publication describes a vehicle whose interior is monitored by means of at least one camera, a radar sensor and a thermal sensor. Fusing sensor data provided by the corresponding sensors improves the accuracy of detection of living beings in the vehicle interior, thus reducing the number of false positive and false negative detections. The detection and localization of living things in the vehicle interior based on camera images is possible by using image recognition algorithms that examine the camera images for the presence of corresponding living things. By means of the radar sensor, movements in the vehicle interior can be detected, as a result of which heart beats and / or breathing patterns, but also general movements, can be detected. The thermosensor can be embodied as a thermographic camera, whereby surface temperatures can be determined. In particular, if the vehicle heats up or cools down, but the surface temperature in the area where a living being is present remains substantially constant, the corresponding living being can thus easily be located. If a corresponding living being is detected in the vehicle interior, then by outputting visual and / or acoustic indications, the passers in the vicinity of the vehicle can be called for assistance or else warning messages can be sent to the vehicle owner or a rescue service. A vehicle function can likewise be activated, for example the automatic opening of the windows or activation of an air conditioning system. The corresponding emergency measure to be initiated by the vehicle can be adapted to the respective situation. If a corresponding living being has, for example, an unusually high breathing rate or pulse rate or threatens to overheat or undercool, the rescue service can be called directly, for example.Furthermore, WO 2022 / 268 313 A1 discloses an interior monitoring system for a vehicle and a method for detecting life signs in the interior of a vehicle. The interior monitoring system allows detection of living beings in the interior of the vehicle by fusing the sensor data of a motion sensor, an optical sensor and an acoustic sensor.In addition, DE 10 2022 004 382 A1 discloses a method for controlling measures for an occupant remaining in a vehicle. With the aid of the method, infants left in the vehicle interior can be protected from overheating. Thus, warnings are issued by the vehicle when the presence of an infant is detected and the vehicle interior is excessively heated. The warnings can alert the vehicle-guiding person or passersby in the environment of the vehicle, which can then be used to aid. In addition, the windows of the vehicle can be automatically opened or the air conditioner can be turned on.The object of the present invention is to specify an improved method for detecting and locating living things in a vehicle interior, which method operates reliably at a reduced outlay, in the sense that living things present or absent are detected reliably and the emission of unnecessary false alarms is at least reduced.According to the invention, this object is achieved by a method for detecting and locating living beings in a vehicle interior having the features of claim 1. Advantageous embodiments and refinements and a vehicle for carrying out the method are evident from the claims dependent thereon.A method of the generic type for detecting and locating living beings in a vehicle interior, wherein at least one camera and one radio sensor each capture at least one section of the vehicle interior, sensor data generated by the camera and the radio sensor are processed by a computing unit, and the outputting of a warning and / or activation of a vehicle function as a function of a living being detected in the vehicle interior can be initiated by the computing unitthe computer unit examines the sensor data generated by the radio sensor for the presence of a breathing pattern and / or a heartbeat, locates the breathing pattern and / or heartbeat in the vehicle interior when a detected breathing pattern and / or heartbeat is detected, and marks a respective location as the region of interest;the computer unit determines areas correlating to the respective region of interest in the sensor data generated by the camera and examines these areas for the presence of an adult, a child and an animal by means of image recognition methods; andoutputting the warning and / or activating the vehicle function if at least one child and / or animal but no adult is detected in the vehicle interior and suppressing the outputting of the warning and / or activating the vehicle function if at least one adult is detected in the vehicle interior.According to the invention, it is provided that the computing unit marks the installation position of the radio sensor as a further region of interest and deactivates the radio sensor if the computing unit recognizes an object or living being in the further region of interest while processing the sensor data generated by the camera.By sensor fusion of sensor data generated by the camera and the radio sensor, reliable detection and localization of living beings in the vehicle interior is ensured. The computing unit does not have to monitor the entire vehicle interior in the camera images, but only those regions which were marked as relevant taking into account the sensor data generated by the radio sensor. This reduces the effort involved in carrying out the method according to the invention, since it is not necessary to evaluate the entire camera images, but only the regions which correspond to the regions of interest. In addition, the issue of unnecessary alarms is reliably prevented. If an adult is present in the vehicle interior, it is not a child or animal who is forgotten in the vehicle interior. Thus, no warning has to be issued or no vehicle function such as lowering the windows or activating the air conditioning system has to take place. If, on the other hand, the computing unit recognizes the at least one animal and / or at least one child present in the vehicle interior, but no adult, it is accordingly a left-behind child or animal, so that said warning should be output and / or said vehicle function should be carried out.Since not all image areas for detecting adults, children and animals have to be processed by the computing unit, and the computing unit only has a limited computing power, the sensing time with which the computing unit determines whether or not said living beings are present in the vehicle interior decreases.The combination of camera and radio sensor is particularly suitable for monitoring the vehicle interior. Thus, living being detection solely on the basis of camera images can meet limits. If, for example, dolls or magazines are carried along in the vehicle interior, which show photographs of people, a corresponding image recognition algorithm can hold a doll and / or a person depicted on a photograph for a living being present in the vehicle interior. However, this can be reliably avoided by taking into account the sensor data generated by the radio sensor. Thus, only in the regions of interest in which a heart beat and / or a breathing pattern has been detected are the detection of living beings enabled. This means that the computing unit takes into account only living beings detected within a region of interest as such.The radio sensor may be, for example, a radar sensor, an UltraBreitband (UWB) sensor, or a Wi-Fi sensor. It is also possible for a plurality of such sensors, also in combination, to be installed. By means of radar, movements can be detected in a particularly simple manner. By analyzing a Wi-Fi signal strength, movements and thus also heartbeats and breathing patterns in an examination room can likewise be detected. For this purpose, a corresponding Wi-Fi sensor comprises in particular a plurality of Wi-Fi antennas which are arranged distributed on the vehicle.The method according to the invention is carried out in particular when the vehicle is at a standstill, since an adult is generally present during the journey and thus does not threaten any risk. If no living things are detected in the vehicle interior, then no warning must be issued either and no vehicle function must be activated either. The corresponding sensors can then be deactivated, for example after a defined waiting time such as five minutes, started after the detection of the absence of all living beings. After predetermined time intervals, such as every other five minutes, a new vehicle interior detection may be performed to detect potentially entering animals. After deactivation of the sensors, these can also be reactivated as a function of an event, for example after an unlocking signal for the vehicle doors has been received or a door opening and door closing process has been detected.If an object or living being is present in the environment around the radio sensor, this object or living being can block the detection of the vehicle interior by the radio sensor. In this case, the radio sensor cannot detect breathing patterns and / or heartbeats either. The radio sensor is then switched off, so that less energy has to be consumed for monitoring the vehicle interior. In addition, the radio wave emission in the vehicle interior is reduced, which optionally prevents interference with further radio technologies.This problem occurs in particular when the radio sensor is a radar sensor. If, on the other hand, a UWB sensor and / or a Wi-Fi sensor is used, this problem can be reduced or even avoided.In a vehicle having at least one camera and radio sensor that captures at least one section of the vehicle interior and a computing unit, the camera, the radio sensor and the computing unit are configured according to the invention to carry out a method described above. The vehicle may be any vehicle such as a car, truck, transporter, bus or the like. The camera and the radio sensor can be arranged in any desired manner in the vehicle interior. The respective sensors are preferably arranged in such a way that they can best detect the respective potential whereabouts of living beings in the vehicle interior. This prevents living things from being overlooked due to masking. The computing unit may be an onboard central computer, a vehicle subsystem controller, a telematics unit, or the like. Individual functions carried out by the computing unit can also be distributed over a plurality of computing units. For example, a first computing unit can process the sensor data and a second computing unit can serve to cause the warning and / or control of the vehicle function. The camera can be designed as an infrared camera, which allows improved image recognition in the case of darkness.An advantageous embodiment of the vehicle provides that at least one radio sensor is arranged at the height of the rear in the vehicle interior with respect to the vehicle longitudinal axis and in particular its detection region is oriented in the direction of the vehicle front. This allows a particularly comprehensive monitoring of the vehicle interior by the radio sensor. For example, the radio sensor can be integrated into the headliner, so that monitoring of the vehicle interior from the bird's eye view is possible.Such a radio sensor can be blocked when a person sits down on the middle seat in the rear. Thus, the area to be monitored by the camera in this case, i.e. the respective region of interest, is located on the spatial volume in which the head of the vehicle occupant will normally be located. A plurality of such spatial volumes and thus regions of interest for persons of different sizes can be defined.Preferably, at least one camera is arranged at the height of the dashboard in the vehicle interior with respect to the vehicle longitudinal axis and in particular its capture region is oriented toward the rear. For reliable differentiation between adults, children and animals, it is necessary to analyze the faces of the living beings by means of image recognition algorithms. Adults and children typically look towards the front of the vehicle in the direction of the longitudinal axis of the vehicle. Thus, facial analysis is possible when the corresponding camera is disposed in the front area of the vehicle and oriented toward the rear. For example, the camera can be integrated into the dashboard itself or also into the headliner or the interior mirror.Due to this advantageous arrangement of the radio sensor and camera, reliable monitoring of the vehicle interior is possible. If the radio sensor and camera are installed at different installation positions in the vehicle and, in particular, their detection direction are aligned with one another, it is possible to prevent that non-monitored regions remain in the vehicle interior. Thus, the number of sensors required can also be reduced. Accordingly, only one radio sensor and one camera are preferably required for providing the method according to the invention.Furthermore, a sensor fusion can take place with further sensors already located in the vehicle. Thus, for example, it is possible to distinguish, taking into account a pressure sensor integrated into a vehicle seat, whether a child or an adult has taken place. For differentiation, the height of the pressure signal (correlated with the weight of the person) is then evaluated.It is also conceivable that a warning cascade is adapted in the case of a living being detected in the vehicle interior, depending on whether a child or animal has been left behind in the vehicle interior. Thus, in the event of an emergency, for example, a warning message could always be sent to the vehicle owner and the air conditioning system could be activated automatically. In addition, a rescue service is then alerted to a child left in the vehicle interior, but not to an animal.Further advantageous embodiments of the method according to the invention for detecting and locating living beings in a vehicle interior and of the vehicle according to the invention also result from the exemplary embodiment which is described in more detail below with reference to the figure.FIG. 1 shows a schematic illustration of the detection ranges of a radio sensor and of a camera for detecting a vehicle interior.FIG. 1 shows a top view of a vehicle 6 according to the invention. The vehicle 6 comprises at least one camera 1 and at least one radio sensor 2 for detecting the vehicle interior. In FIG. 1, the detection range 1E of the camera 1 is shown at the upper left and the detection range 2E of the radio sensor 2 at the upper right. FIG. 1 shows a combined illustration at the bottom, in which the two detection regions 1E and 2E are placed one above the other. In the exemplary embodiment shown, the radio sensor 2 is a radar sensor. If, on the other hand, a UWB sensor and / or a Wi-Fi sensor is used, it can also happen that, in these sensors, the detection region 2E occupies the entire vehicle interior and no invisible regions remain.The arrangement of camera 1 and radio sensor 2 on vehicle 6 shown in the figure allows comprehensive detection of the vehicle interior. It is thus ensured that no concealed regions remain which cannot be detected at least by a sensor.Sensor data generated by the camera 1, here in the form of camera images, and sensor data generated by the radio sensor 2 are evaluated by a computing unit 3. Preferably, the at least one camera 1 is installed in the front part of the vehicle 6, in particular at the height of the dashboard, as shown in FIG. 1, and the at least one radio sensor 2 is installed in the region of the rear of the vehicle 6, for example integrated into the roof lining.According to the invention, the computing unit 3 evaluates sensor data generated by the camera 1 and the radio sensor 2 for detecting and locating living things in the vehicle interior. In this case, a sensor fusion is carried out in such a way that breathing patterns and / or the heartbeat of living beings in the vehicle interior are detected by means of the radio sensor 2 and corresponding locations in the vehicle interior are marked as region of interest 4.1. The arithmetic unit 3 determines areas 5 in the camera images that correlate therewith and checks, by using image recognition methods, whether adults, children and animals are present in the respective area 5. If no living being is present in the vehicle interior or if at least one adult is present, it is not a forgotten or left-behind child or animal, so that no warning needs to be issued and no vehicle function needs to be actuated. If, on the other hand, no adult is present in the vehicle interior, but at least one child and / or animal, the computing unit 3 initiates that a warning is output and / or a vehicle function is activated. For example, the horn of the vehicle 6 can be extinguished, the headlights of the vehicle 6 are illuminated, a warning message can be sent to a mobile terminal of the vehicle owner or a rescue service can also be contacted.In the exemplary embodiment shown in the figure, the radio sensor 2 is integrated into the headliner in the region of the rear above the central seat. If objects are now transported in the rear or if a person sits on the middle rear seat, they optionally block the view of the radio sensor 2 on the vehicle interior. The method according to the invention therefore provides for the radio sensor 2 to be deactivated in this case. For this purpose, the computing unit 3 checks whether objects or living beings can be detected in the camera images generated by the camera 1 in a corresponding further region of interest 4.2 in the region of the installation position of the radio sensor 2. For this purpose, the arithmetic unit 3 stores information about which spatial volumes of the vehicle interior most likely correspond to a location region of the head of the vehicle occupant. These spatial volumes are permanently stored as a further region of interest 4.2.As can be seen from FIG. 1, the respective detection ranges 1E and 2E may be restricted. For example, the backrests of the driver seat and the passenger seat at least partially block the view of the camera 1 on the rear as well as the view of the driver seat and the passenger seat for the radio sensor 2 (radar sensor). In order to be able to carry out the method according to the invention, either further radio sensors 2, not shown in detail, can be installed, which also make it possible to detect the driver's seat and the passenger's seat. Additionally or alternatively, as explained in connection with the further region of interest 4.2, for the head-holding boxes of the driver and passenger or regions of interest (not shown) can be stored in the computing unit 3, so that, independently of a radar-based detection, the computing unit 3 examines the camera images for the presence of a driver / passenger. In addition, further cameras (not shown) 1 can be installed, so that the entire rear region can be reliably detected.

Claims

Method for detecting and locating living beings in a vehicle interior, wherein at least one camera (1) and one radio sensor (2) each capture at least one section of the vehicle interior, sensor data generated by the camera (1) and the radio sensor (2) are processed by a computing unit (3) and the outputting of a warning and / or triggering of a vehicle function can be initiated by the computing unit (3) as a function of a living being detected in the vehicle interior, wherein - the computing unit (3) examines the sensor data generated by the radio sensor (2) for the presence of a breathing pattern and / or a heartbeat, locates said sensor data in the vehicle interior in the case of a detected breathing pattern and / or heartbeat and marks a respective location as an area of interest (4.1); the computer unit (3) determines areas (5) correlating with the respective region of interest (4.1) in the sensor data generated by the camera (1) and examines these areas (5) for the presence of an adult, a child and an animal by means of image recognition methods; and - outputs the warning and / or actuates the vehicle function if at least one child and / or animal but no adult is detected in the vehicle interior and suppresses the output of the warning and / or the actuation of the vehicle function if at least one adult is detected in the vehicle interior, characterized in that - the arithmetic unit (3) marks an installation position of the radio sensor (2) as a further region of interest (4.2) and deactivates the radio sensor (2) if the arithmetic unit (3) recognizes an object or living being in the further region of interest (4.2) by processing the sensor data generated by the camera (1).Vehicle (6) having at least one camera (1) and radio sensor (2) which records at least one section of a vehicle interior and having a computing unit (3), characterized in that the at least one camera (1), the at least one radio sensor (2) and the computing unit (3) are configured to carry out a method according to Claim 1.Vehicle (6) according to Claim 2, characterized in that at least one radio sensor (2) is arranged at the height of a rear in the vehicle interior with respect to the vehicle longitudinal axis and in particular its detection region (2E) is oriented in the direction of the vehicle front.Vehicle (6) according to Claim 2 or 3, characterized in that at least one camera (1) is arranged at the height of a dashboard in the vehicle interior with respect to the vehicle longitudinal axis and in particular its capture region (1E) is oriented towards the rear.

Images