Method and system for monitoring a vehicle interior

Abstract

The invention relates to a method for monitoring the interior (2) of a vehicle (1), in particular a motor vehicle, which has a plurality of seats (3a, 3b, 3c, 3d, 3e) and in which at least one recording device (4a, 4b, 4c, 4d) is arranged. For this purpose, a reference image of the interior (2) is first recorded at a first time point by the at least one recording device (4a, 4b, 4c, 4d) (S100). Subsequently, at least one comparison image of the interior (2) is recorded at at least a second time point by the at least one recording device (4a, 4b, 4c, 4d) (S110).The at least one comparison image is compared with the reference image (S120) to determine whether a change has occurred in the vehicle interior (2), and, if it is determined that a change has occurred in the vehicle interior (2), the at least one comparison image is compared with a plurality of calibration images (S130). Subsequently, a vehicle interior situation is determined based on the result of comparing the comparison image with the plurality of calibration images (S140).

Description

Background of the invention

[0001] The present invention relates to a method and a system for monitoring a vehicle interior as well as a computing unit and a computer program for carrying out the method.

[0002] Cameras can be used in vehicles to monitor the vehicle interior, but they have a high power consumption. Disclosure of the invention

[0003] According to the invention, a method and a system for monitoring a vehicle interior, as well as a computer program for implementing the method, are proposed, having the features of the independent patent claims. Advantageous embodiments are the subject of the dependent claims and the following description. The embodiments and advantages described below apply equally to all subject matters of the invention.

[0004] By taking a reference image and one or more comparison images, which are in particular three-dimensional images of the vehicle interior, and comparing them with one another, a vehicle interior situation can be determined, which can include not only the position of a vehicle occupant within the vehicle interior, but also other characteristics of the vehicle occupant. Various measures can be implemented based on the determined vehicle interior situation. Various measures can be implemented based on the determined vehicle interior situation, such as issuing a warning depending on the determined vehicle interior situation or controlling a safety system or vehicle assistance system depending on the determined vehicle interior situation.

[0005] For example, it can be determined whether a vehicle occupant is an adult or a child, especially in a child seat, an animal, shopping, or something similar. In this case, it may not be necessary to wear a seatbelt.

[0006] If a child seat is detected on the front passenger seat facing the opposite direction of travel, the front passenger seat airbag can be automatically deactivated, thereby increasing the child's safety in the event of an accident.

[0007] In the event of an accident, a belt tensioner and / or the airbag can also be activated in the seats occupied by one person, which can further increase the safety of the vehicle occupants.

[0008] Furthermore, movement inside the car can be detected, making it possible to detect, for example, an arm reaching through a window into the vehicle interior, which could indicate a theft attempt. Movement inside the vehicle could also indicate a pet or child left in the car. In both cases, an alarm can be triggered or a window can be opened.

[0009] In one aspect, the invention relates to a method for monitoring a vehicle interior of a vehicle, in particular a motor vehicle, which has a plurality of seats and in which at least one recording device is arranged. In the method, a reference image of the vehicle interior is recorded at a first point in time by the at least one recording device. Later, at least one comparison image of the vehicle interior is recorded at at least a second point in time by the at least one recording device. The at least one comparison image is compared with the reference image to determine whether a change has occurred in the vehicle interior and, if it is determined that a change has occurred in the vehicle interior, the at least one comparison image is compared with a plurality of calibration images.A vehicle interior situation is then determined based on the result of the comparison of the comparison image with the large number of calibration images.

[0010] The calibration images are images of the vehicle interior taken in various situations or configurations. These various situations or configurations can include, in particular, the multiple seats and their occupants. These various situations or configurations can also include, for example, an arm being moved through a window into a vehicle, which can also be used to detect attempted theft.

[0011] The comparison images are compared with the calibration images and a vehicle interior situation is determined based on the result of the comparison of the comparison image with the large number of calibration images.

[0012] When comparing the comparison image with the multitude of calibration images, a probability can be calculated, for example, as to how well the current situation in the comparison image matches the situations captured by the calibration images. The agreement or determination of the result of the comparison of the images can be achieved using a classification, e.g., a support vector machine. If the current comparison image does not match a situation shown in the calibration images exactly or with a predetermined probability, the situation can be assessed / interpolated using additional software intelligence, e.g., artificial intelligence, and a probability for a combination of calibration images can be determined, based on which the vehicle interior situation is determined.

[0013] In one embodiment, a measure is carried out based on the determined vehicle situation. In this case, various elements of the vehicle can be controlled as a measure, for example. For example, in the event of an accident, an airbag or a belt tensioner can only be activated on seats that are actually occupied. Furthermore, it is also possible to activate a heater, in particular seat heating, only on seats that are occupied. Furthermore, if vehicle theft is detected, an alarm can be triggered and the engine prevented from starting. Based on the result, a further measure can also be deactivated an airbag on a seat on which a child seat is placed, or a driver can be warned if it is detected that a child is attempting to open a door while driving.

[0014] Furthermore, comparative images can be used to determine that a child and / or pet has been left inside the vehicle. If such a situation is detected, a warning can be issued or a window can be fully or partially opened to prevent the vehicle interior from becoming too warm. The warning can, in particular, only be issued after a predetermined period of time has elapsed.

[0015] In one embodiment, when determining the vehicle interior situation, at least one occupancy of the plurality of seats and / or at least one property of one or more vehicle occupants and / or at least one movement of one or more vehicle occupants is determined.

[0016] The at least one characteristic of the one or more vehicle occupants is, for example, the type of vehicle occupant. The vehicle occupant can be identified, for example, as an adult, a child (especially in a child seat), a pet (such as a dog or cat), or an object (e.g., shopping).

[0017] The at least one property can in particular also comprise an identity of the vehicle occupant, which can be determined, for example, by means of facial recognition. By comparing this data with existing data, it can be determined that at least one of the vehicle occupants is a person who is authorized to operate the vehicle. In particular, it can also be determined whether a smartphone or transponder assigned to an authorized person (e.g. in a key or key fob (also referred to as a fob or key fob)) is in the vehicle. This makes it possible to detect a theft of the vehicle if, for example, the only vehicle occupants in the vehicle are those unknown to the system.

[0018] The movements of one or more vehicle occupants include, for example, movements of the extremities, the head, or breathing movements. The vehicle situation can therefore also include a life-threatening situation for the occupants, for example, if insufficient breathing movements are detected, or a theft situation if an arm is detected reaching through a window into the vehicle interior.

[0019] In particular, one or more of the specific properties and / or movements can be used to control various components of the vehicle.

[0020] In one embodiment, the reference image, the at least one comparison image, and the plurality of calibration images are three-dimensional images of the vehicle interior. In such images, in particular, all surfaces that reflect a signal from the recording device, e.g., a radar or lidar signal, are recognizable, with a three-dimensionality resulting, for example, from a time-of-flight (TOF) analysis. At least some of the aforementioned images can also be two-dimensional images.

[0021] In contrast to two-dimensional images, three-dimensional images also record information about depth and spatial arrangement, thus particularly information in three dimensions.

[0022] The images can be acquired, in particular, by ultra-wideband (UWB) radars. Related acquisition techniques are described, for example, in "I. Ullman, Novel Concepts for Radar Imaging in Heterogeneous Media, Dissertation, FAU Erlangen-Nuremberg," especially pages 9 and 10, "G. Adamikum, Methods for the Realization of Dual-Orthogonal, Linearly Polarized Antennas for UWB Technology, Dissertation KIT," or "A. Bulavinov, Real-time 3D Imaging Using Pulsed Array Beam Techniques for Ultrasonic Testing Applications; DGZfP Annual Conference 2009 - Tue.2.A.2."

[0023] Three-dimensional or topological images of the vehicle interior can improve the determination of the vehicle interior situation, as more reference points are available for comparison between the individual images. This allows the aforementioned advantages to be achieved in an optimized manner.

[0024] In one embodiment, the first time at which the reference image is recorded is a time at which a door of the vehicle is unlocked or locked. At this time, the vehicle is usually unoccupied, and the recorded reference image can be used as a basis for easily determining the position and properties of the vehicle occupants, since the occupancy can be determined based on the differences between the reference image and the comparison image. In particular, by recording when the vehicle is being locked, it can be achieved that the method can also be carried out in the event of a theft, in particular in which a break-in occurs through a window of the vehicle without the vehicle doors being unlocked.

[0025] In one embodiment, several, in particular two, three, or four, comparison images are taken at second, offset points in time. When determining the vehicle interior situation, at least one comparison image of the several comparison images, in particular the most recent comparison image or all comparison images, is used.

[0026] Comparing only the most recent comparison image can save computing power. However, if multiple comparison images are taken and compared with the calibration images, the vehicle interior situation can be more accurately determined, as multiple comparison points are available, and a temporal progression of the vehicle interior situation can be determined.

[0027] Furthermore, by continuously recording comparative images at different times, changes in the occupant occupancy and characteristics of the vehicle's occupants can be detected, ensuring that this data is always up-to-date. If necessary, the driver can also be alerted to dangerous changes, such as a child attempting to open a door or unfasten their seatbelt while driving. This can increase the safety of all vehicle occupants.

[0028] In one embodiment, the plurality of calibration images are obtained in a calibration process in which images of the vehicle interior are generated in different situations. This provides a plurality of known vehicle interior situations, allowing the current vehicle interior situation to be accurately determined, thereby achieving the aforementioned advantages in a simple, low-computational-intensive, and cost-effective manner.

[0029] In one embodiment, the second point in time is a point in time at which an acceleration has reached and / or exceeded an acceleration threshold. The acceleration of the vehicle is measured, in particular, by an acceleration sensor.

[0030] By starting the comparison recording after a certain acceleration value has been exceeded, it is ensured that the vehicle is already moving and that no load is being added to the vehicle or that children are not being restrained. This allows the computing power of the processing unit to be saved.

[0031] A computing unit according to the invention, e.g. a control unit of a motor vehicle, is designed, in particular in terms of programming, to carry out a method according to the invention.

[0032] A system according to the invention for monitoring a vehicle interior of a vehicle, in particular a motor vehicle having a plurality of seats, comprises at least one recording device arranged in the vehicle interior and the computing unit according to the invention.

[0033] The at least one recording device may comprise a radar device, in particular an ultra-wideband (UWB) radar device, or a lidar device.

[0034] The at least one receiving device can be arranged in a vehicle pillar, in particular the A and / or B and / or C pillar, and / or a roof lining of the vehicle, wherein the at least one receiving device is aligned such that images of the vehicle interior can be taken by it.

[0035] For example, two ultra-wideband radar devices can be installed in the vehicle interior. One ultra-wideband radar device can be located on each A-pillar of the vehicle. It is also conceivable to install four ultra-wideband radar devices in the vehicle interior, with one ultra-wideband radar device installed in each of the A-pillars and one in each of the B-pillars. This can increase the resolution accuracy, especially in the rear seat. Additionally or alternatively, it is also possible to install an ultra-wideband radar device in the roof lining.

[0036] For example, using ultra-wideband radar devices, it is possible to detect a mobile phone. This can further ensure that at least one of the vehicle occupants is authorized to use the vehicle.

[0037] By means of such a system for monitoring a vehicle interior, the advantages previously mentioned in connection with the method can be achieved in a simple and cost-effective manner.

[0038] The implementation of a method according to the invention in the form of a computer program or computer program product with program code for carrying out all method steps is also advantageous, since this entails particularly low costs, in particular if an executing control unit is also used for other tasks and is therefore already present. Finally, a machine-readable storage medium is provided with a computer program stored thereon, as described above. Suitable storage media or data carriers for providing the computer program are, in particular, magnetic, optical and electrical memories, such as hard disks, flash memories, EEPROMs, DVDs, and others. Downloading a program via computer networks (Internet, intranet, etc.) is also possible. Such a download can be wired or cable-based or wireless (e.g. via a WLAN network, a 3G, 4G, 5G or 6G connection, etc.).

[0039] Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawings.

[0040] The invention is illustrated schematically in the drawing using exemplary embodiments and is described below with reference to the drawing. Short description of the drawings Fig. 1 shows a block diagram of a motor vehicle which is designed to carry out an embodiment of the method according to the invention, and Fig. 2 shows a flowchart of an embodiment of the method according to the invention. Embodiment(s) of the invention

[0041] Fig. 1 shows a block diagram of a vehicle 1, in particular a motor vehicle, which is designed to carry out an embodiment of the method according to the invention. Fig. Figure 2 shows a flowchart of an embodiment of the method according to the invention. Fig. 1 and Fig. 2 will be described together below.

[0042] The Fig. The vehicle 1 shown in Figure 1 has five seats 3a, 3b, 3c, 3d, 3e in the vehicle interior 2, two front seats 3a, 3b and three seats 3c, 3d, 3e in the rear seat. Furthermore, four recording devices 4a, 4b, 4c, 4d are arranged in the vehicle interior 2. These recording devices are, in particular, UWB radar devices and can generate three-dimensional images of the vehicle interior 2. Two recording devices 4a, 4b are arranged on the A-pillars and two recording devices 4c, 4d are arranged on the B-pillars of the vehicle 1. The two mounting devices 4a, 4b, which are arranged on the A-pillars, can be used to monitor the front seats 3a, 3b, while the mounting devices 4c, 4d, which are arranged on the B-pillars, can be used to monitor the seats 3c, 3d, 3e on the rear bench.

[0043] In the method for monitoring the vehicle interior 2, which is carried out in a computing unit 7, for example a control unit of the vehicle 1, and in Fig. 2 is shown as a flow chart, in a first step S100, a reference image of the vehicle interior 2 is taken by the recording devices 4a, 4b, 4c, 4d at a first time. The reference image is taken in particular when a door 5 of the vehicle 1 is unlocked or locked.

[0044] Subsequently, in a second step S110, at least one comparison image is recorded by the recording devices 4a, 4b, 4c, 4d at at least a second point in time. If multiple comparison images are recorded, they are recorded, in particular, at different times from one another.

[0045] In one embodiment, for example, the recording of the comparison images can only begin when an acceleration sensor 6 in the vehicle 1 determines that the acceleration of the vehicle 1 reaches and / or exceeds an acceleration threshold. This can be useful for measures that require a moving vehicle, such as the aforementioned seat belt and airbag measures, etc.

[0046] In one embodiment, the recording of the comparison images can also begin while the vehicle is stationary. This can be useful for measures that require a stationary vehicle, such as the aforementioned theft, child, and pet detection measures, etc.

[0047] The comparison images are compared with the reference images in a step S120 and from the comparison it is determined whether a change has occurred in the vehicle interior 2.

[0048] A change can be determined, for example, when the driver and possibly other vehicle occupants have entered the vehicle. However, a change can also be an arm reaching through a window when the vehicle 1 is stationary and the door 5 is locked. The movement of a child or pet when the door 5 is locked can also be detected by comparing a comparison image taken after the door 5 is locked with the reference image taken when the door 5 is locked.

[0049] If it is determined that a change has occurred in the vehicle interior, in step S130, the at least one comparison image is compared with a plurality of calibration images to determine the vehicle interior situation (S140). In this case, for example, the occupancy of seats 3a, 3b, 3c, 3d, 3e and / or characteristics of the vehicle occupants located in one or more of the seats 3a, 3b, 3c, 3d, 3e can be determined. The calibration images can be created, in particular, in a calibration process that was performed, for example, before the first use of the vehicle 1.

[0050] The calibration images are expediently generated in different vehicle interior situations or configurations, e.g. with different numbers of vehicle occupants and vehicle occupants in different seats 3a, 3b, 3c, 3d, 3e.

[0051] Based on the determined vehicle interior situation, a measure can be determined and carried out, for example by the computing unit 4 or another control unit of the vehicle 1 (S150).

[0052] For example, it can be determined for which of the seats 3a, 3b, 3c, 3d, and 3e a seat belt must be used. This means that if shopping is placed on a rear seat or a child seat is installed in the rear of the rear seat, the seat belts of the corresponding seats 3a, 3b, 3c, 3d, and 3e do not need to be used, even if these seats 3a, 3b, 3c, 3d, and 3e are occupied. Furthermore, it can be determined which airbags and belt pretensioners must be activated in the event of an accident. This can increase the safety of the vehicle's occupants.

[0053] Furthermore, it is also possible to control the air conditioning or seat heating of vehicle 1 based on the specific occupancy and characteristics. This eliminates the need to activate the seat heating on seats 3a, 3b, 3c, 3d, and 3e that are unoccupied or occupied by a child seat or shopping.

[0054] An alarm signal can also be issued if an attempted theft of vehicle 1 is detected.

[0055] If a child or pet is discovered in the vehicle 1 when the door 5 is locked, a warning can be issued. This can be, for example, an acoustic warning signal such as in the event of a theft or a message to a mobile phone of the driver of the vehicle 1 or to an emergency call center. Alternatively or additionally, a window of the vehicle 1 can be opened completely or partially in order to reduce the temperature in the vehicle. In particular, the measure can only be carried out after a predetermined time in order to avoid an unnecessary or possibly even undesirable measure being carried out in situations in which the vehicle is only locked with passengers for a short time, for example to pay at a gas station.

Claims

[1] Method for monitoring a vehicle interior (2) of a vehicle (1), in particular a motor vehicle, which has a plurality of seats (3a, 3b, 3c, 3d, 3e) and in which at least one receiving device (4a, 4b, 4c, 4d) is arranged, the method comprising the following steps: Recording (S100) a reference image of the vehicle interior (2) at a first time by the at least one recording device (4a, 4b, 4c, 4d); Recording (S110) at least one comparison image of the vehicle interior (2) at at least a second point in time by the at least one recording device (4a, 4b, 4c, 4d); Comparing (S120) the at least one comparison image with the reference image to determine whether a change has occurred in the vehicle interior (2); if it is determined that a change has occurred in the vehicle interior (2), comparing (S130) the at least one comparison image with a plurality of calibration images; and Determining (S140) a vehicle interior situation based on the result of the comparison of the comparison image with the plurality of calibration images. [2] Method according to claim 1, wherein the reference image, the at least one comparison image and the plurality of calibration images are two-dimensional or three-dimensional images of the vehicle interior (2). [3] Method according to any preceding claim, wherein the method further comprises, after determining (S140) a vehicle situation: Carrying out (S150) an action depending on the determined vehicle situation. [4] Method according to one of the preceding claims, wherein the first time is a time at which at least one door (5) of the vehicle (1) is locked or unlocked. [5] Method according to one of the preceding claims, wherein a plurality of, in particular two, three or four, comparison images are recorded at second points in time offset from one another and in the step of determining (S140) the vehicle interior situation, at least one comparison image of the plurality of comparison images, in particular the most recent comparison image, is used. [6] Method according to one of the preceding claims, wherein during the determination (S140) of the vehicle interior situation at least one of the following is determined: an occupancy of the plurality of seats (3a, 3b, 3c, 3d, 3e), at least one property of one or more vehicle occupants, at least one movement of one or more vehicle occupants. [7] Method according to one of the preceding claims, wherein the plurality of calibration images are obtained in a calibration process in which images of the vehicle interior (2) are generated in different situations. [8] Method according to one of the preceding claims, wherein the second time is a time at which an acceleration of the vehicle (1) reaches and / or exceeds an acceleration threshold value. [9] Method according to the preceding claim, wherein the acceleration of the vehicle (1) is detected by an acceleration sensor (6). [10] Computing unit which is designed to carry out all method steps of a method according to one of the preceding claims. [11] System for monitoring a vehicle interior (2) of a vehicle (1), in particular a motor vehicle, which has a plurality of seats (3a, 3b, 3c, 3d, 3e), the system comprising: at least one receiving device (4a, 4b, 4c, 4d) arranged in the vehicle interior (2), and the computing unit according to claim 10. [12] System according to claim 11, wherein the at least one recording device (4a, 4b, 4c, 4d) comprises a radar device, in particular an ultra-wideband radar device. [13] System according to claim 11 or 12, wherein the at least one receiving device (4a, 4b, 4c, 4d) is arranged in a vehicle pillar, in particular the A and / or B pillar, and / or a roof lining of the vehicle (1) and is aligned such that images of the vehicle interior can be taken by the at least one receiving device (4a, 4b, 4c, 4d). [14] Computer program which causes a computing unit to carry out all the method steps of a method according to one of claims 1 to 9 when executed on the computing unit. [15] A machine-readable storage medium having stored thereon a computer program according to claim 14.

Images