Methods for monitoring the interior of a motor vehicle and motor vehicle

Abstract

Method for monitoring the interior of a motor vehicle (2) with regard to objects (8, 9) placed in the interior (1), wherein the motor vehicle (2) has at least one interior camera (14) for recording image data of the interior (1), at least one radar sensor (15) for recording radar data of the interior (1) and several storage areas (7) to be monitored for objects (8, 9) in the interior (1), wherein the storage areas (7) to be monitored are a parcel shelf (6) and at least an substantially horizontal portion of a dashboard, wherein the method comprises, within the framework of an assistance functionality: - Classification of all objects (8, 9) located in the storage area (7) as potentially dangerous or harmless by means of a classification function which uses as input data both image data or data derived therefrom and radar data or data derived therefrom, wherein the hazard classes are defined differently for several storage areas to be monitored, so that the classification function determines a storage area-specific hazard class for each storage area, - upon fulfillment of a warning condition which indicates at least the presence of at least one object (8, 9) classified as potentially dangerous in at least one of the at least one storage area (7) to be monitored, output of a warning information to indicate the at least one object (8, 9) classified as potentially dangerous located in at least one of the storage area (7) to be monitored.

Description

[0001] The invention relates to a method, particularly a computer-implemented one, for monitoring the interior of a motor vehicle with regard to objects placed in the interior, wherein the motor vehicle has at least one interior camera for recording image data of the interior, at least one radar sensor for acquiring radar data of the interior, and several storage areas for objects in the interior to be monitored, wherein the storage areas to be monitored are a parcel shelf and a portion of the dashboard that extends at least substantially horizontally. The invention also relates to a motor vehicle.

[0002] Inside a vehicle, there are storage areas for items that, in the event of hard braking and / or an accident, could become dislodged and potentially move into the vicinity of an occupant. Such objects can therefore pose a potential hazard. Examples of such storage areas include a predominantly horizontal portion of the dashboard, and especially the parcel shelf, which is located between the rear window and the rear seat backrest.

[0003] Prior art has already proposed assistance functions to increase safety regarding such storage areas.

[0004] DE 10 2019 204 630 A1 discloses a method for displaying information to a user regarding an object located in a motor vehicle. The method checks whether the object poses a danger according to at least one predetermined criterion, and if so, an optical warning is displayed at a spatially corrected projection position. The object is detected using at least one 3D camera. A potentially dangerous position could be on the parcel shelf.

[0005] DE 10 2020 102 053 A1 relates to a system for object monitoring in a vehicle. Using a sensor, in particular an interior camera, an object inside the vehicle is detected and correlated with at least one interior parameter for situation assessment. For example, the motion trajectories of the at least one object can be determined and it can be checked whether these trajectories intersect with vehicle occupants. For example, it can be determined that a hammer lying on the parcel shelf would hit the head of an occupant sitting in the back seat if the vehicle were to brake sharply.

[0006] DE 10 2021 123 129 B4 describes a method for securing a motor vehicle and a motor vehicle itself. Within this framework, object position information concerning a position and object property information concerning a property of several objects located in the passenger compartment are determined using detection devices. Subsequently, danger zone position information and danger zone extent information are determined, based on which the danger zone can be detected. The determination of this information depends on the danger potential of the individual object. The detected objects are categorized into objects with a relatively low danger potential or with a relatively high danger potential. Accordingly, the position or extent of the danger zone located between the airbag and the occupant is also considered.If the hazard condition is met, a visual or acoustic warning signal is issued to the occupant.

[0007] DE 10 2017 211 243 B4 describes a method and a device for monitoring the interior of a motor vehicle. Using sensor units (camera and / or radar sensors), which are located, for example, in the driver's footwell, the headliner, etc., it is possible to determine whether objects are present within their detection range and, if so, whether they are moving, and whether these objects could pose a nuisance or even a danger to the vehicle occupants. In the event of a nuisance caused by the detected object, a warning message is issued via an output unit.

[0008] DE 102 29 815 A1 relates to a system for protecting the cargo area of ​​vehicles. The system comprises several sensors that detect cargo in the vehicle's cargo area and the condition of the cargo area's protective devices. Depending on the detected cargo and the condition of the protective devices, a cargo area status signal is generated, indicating different states, such as a safe driving condition, a potentially hazardous but still safe driving condition, or an unsafe condition. The generated cargo area status signal is displayed by means of a display device.

[0009] DE 10 2021 206 012 A1 describes a method for monitoring the loading area of ​​a vehicle. The method involves detecting an object on the vehicle's loading area using a detection device. First, the detection device determines whether an object on the loading area is in a condition that poses a danger to the vehicle's surroundings. For this purpose, the detection device's measuring points are categorized into different categories. If a critical condition is detected in the corresponding category based on the measured values, a visual and / or audible warning can be issued to the driver or to those in the vehicle's vicinity.

[0010] DE 10 2019 204 630 A1 describes, among other things, a warning device for providing information to a user regarding a specific object located in a motor vehicle. The warning device comprises several interior cameras with which, in particular, loose objects in the vehicle can be detected. Subsequently, a control unit checks whether the object poses a danger according to a predetermined criterion. Possible criteria include the weight, size, geometry, and structure of the object. If the object is classified as dangerous, an optical warning signal is emitted via the warning device.

[0011] The invention is based on the objective of enabling improved safety and robustness in interior monitoring with regard to loose objects in monitoring assistance systems.

[0012] To solve this problem, the invention provides a method, in particular a computer-implemented method, and a motor vehicle with the features of the dependent claims. Advantageous embodiments are described in the sub-claims.

[0013] A procedure of the type mentioned above comprises the following steps within the framework of an assistance function: - Classification of all objects located in the storage area as potentially dangerous or harmless by means of a classification function that uses as input data both image data or data derived therefrom and radar data or data derived therefrom, whereby in the case of several storage areas to be monitored the hazard classes are defined differently, so that the classification function determines a storage area-specific hazard class for each storage area, - upon fulfillment of a warning condition indicating the presence of at least one object classified as potentially dangerous in at least one of the storage areas to be monitored, a warning information is issued to indicate the at least one object classified as potentially dangerous located in at least one of the storage areas to be monitored.

[0014] According to the invention, an assistance function for interior monitoring, which can be provided, for example, by a corresponding assistance system of the motor vehicle, is proposed. This function first classifies objects detected from the image data of the at least one interior camera and the radar data of the at least one interior radar sensor, in particular at least those objects located in storage areas to be monitored, especially the parcel shelf, but preferably all loose objects in the interior that have been detected, as potentially dangerous or harmless. For this purpose, a classification function is used that can classify objects, for example, into one of two hazard classes, namely "potentially dangerous" and "harmless." Harmless objects are those that do not pose a significant risk of injury, such as stuffed animals, hats, and the like.Potentially dangerous objects are items that pose a significant risk of injury if they come into contact with an occupant, such as laptops, water bottles, and the like. It should be noted that if there are multiple storage areas to be monitored, for example, a storage area on the dashboard and a parcel shelf, the hazard classes can be defined differently, so that the classification function can determine a storage area-specific hazard class for each storage area.

[0015] If a potentially dangerous object is detected in a monitored storage area, i.e., if the warning condition is met, a corresponding warning information is issued via a suitable output device of the vehicle to inform the occupants of the vehicle, especially the driver, for example at the start of the journey, about the potential hazard.

[0016] A fundamental aspect of the inventive approach is the use of an indoor radar sensor to compensate for the disadvantages of indoor cameras alone and to allow for a more robust evaluation. For example, indoor cameras have detection problems in darkness that do not exist for the radar sensor. In other words, the indoor camera and the indoor radar (which are expediently registered or are registered) work together in such a way that the sensor data from both systems, i.e., the image data and the radar data, are used for monitoring, in particular detection and, if necessary, tracking, of the same object.

[0017] In this way, under good visibility conditions, the use of the radar sensor increases the robustness and reliability of detection and classification, while under poorer visibility conditions for the interior camera, the radar sensor can still provide information about objects inside the vehicle and thus compensate for the disadvantages of the interior camera.

[0018] In appropriate further training, a trained function can be used as the classification function. Generally speaking, a trained function models cognitive functions that people associate with other human brains. Through training based on training data, also known as machine learning, the trained function is able to adapt to new circumstances, detect patterns, and extrapolate. Parameters of a trained function can be adjusted through training. A trained function can, for example, include a neural network, a support vector machine (SVM), a decision tree, and / or a Bayesian network. The neural network can be a deep neural network, a convolutional neural network (CNN), or a deep CNN. Other types of neural networks can also be used.

[0019] In other words, artificial intelligence can be used to detect potentially dangerous objects. Trained functions are particularly well-suited for evaluating image data and image-like data. In this context, it can also be advantageous to provide at least one radar sensor as an imaging radar, which delivers spatially resolved information comparable to camera images from a three-dimensional camera. By using both image and radar data, a robust basis for the detection and classification of objects inside the vehicle is provided.

[0020] Preferably, the classification function can evaluate not only the position and shape of the object to be classified, as described by the input data, but also its color and texture. The classification function, and in particular the trained classification function, considers not only the geometric shape of the object, but also its color and texture. From this, the artificial intelligence can better assess the potential hazard emanating from the object.

[0021] For example, the classification function can be specifically designed to assign objects to specific object classes, with each specific object class being assigned one of the two hazard classes. Specific object classes could, for instance, group common items in motor vehicles such as laptops, bottles, stuffed animals, bags, and the like, with each of these specific object classes then being assigned one of the two hazard classes: "potentially dangerous" and "non-dangerous." If the specific object class is also output as a result of the classification function, it can be used further, particularly for object-specific customization of the warning information, for example, "Caution! Bottle on the parcel shelf!"

[0022] In a particularly advantageous embodiment of the present invention, it can be provided that objects located outside the at least one storage area to be monitored are also classified using the classification function and subsequently tracked, particularly at least in darkness, at least using radar data, with regard to their placement in at least one of the at least one storage area to be monitored. In particular, with the aid of the at least one interior camera, a more precise distinction between potentially dangerous and harmless objects can thus be made possible in good light conditions, for example, during the boarding of occupants.Since a corresponding registration between the coordinate systems of the at least one interior camera and the at least one interior radar sensor is usually established, which is also useful for joint evaluation using the classification function, tracking can continue even in darkness using the at least one radar sensor, for example, while driving. In other words, in a specific embodiment, when entering the vehicle at night with the interior lighting activated, an object can be detected and classified (potentially dangerous / non-dangerous) using the at least one radar sensor and the at least one interior camera. If the interior lighting is then switched off while driving, the at least one camera can no longer track the previously classified object.The radar sensor can still detect whether an object has been placed in a safety-critical location within a monitored storage area, such as the parcel shelf, while driving in the dark. If an object identified as potentially dangerous is placed in at least one of the monitored storage areas while driving in the dark, a warning message can be issued accordingly. This configuration therefore employs a two-step process: first, an object is robustly classified according to its hazard potential based on image data from at least one interior camera; then, in a second step, the object classified as potentially dangerous is tracked by the radar sensor (and preferably also by the interior camera), i.e., its position is monitored. Thus, the two sensor types work together optimally to increase the robustness and safety of the assistance function.

[0023] A further, particularly advantageous embodiment of the present invention provides that, in cases of insufficient image data due to darkness, objects that cannot be clearly classified by the classification function are categorized as potentially dangerous, and / or the warning condition is met even for objects that cannot be clearly classified in at least one of the at least one storage area to be monitored. Thus, if an object that was not present in the light and therefore could not be reliably classified is placed on a storage area to be monitored in darkness, this will nevertheless be detected by the radar sensor. Since the object cannot be reliably classified, it is still considered potentially dangerous to increase safety. Therefore, for example, if an object is taken from a backpack in darkness and placed on the parcel shelf, the at least one interior camera is of no use.In this case, at least one radar sensor still detects the added object on the parcel shelf and informs the occupants about this object as part of the warning information, even without successfully carrying out the assessment of the hazard potential.

[0024] An ultra-wideband radar sensor can advantageously be used as the radar sensor. Such ultra-wideband radar sensors are already frequently installed in the interiors of motor vehicles to detect the presence of children. Such an ultra-wideband radar sensor can also advantageously be used within the scope of the present invention.

[0025] The warning information can be displayed visually and / or audibly. For example, a recognizable acoustic signal can indicate an object in a monitored storage area, or a corresponding voice announcement can be made, for example, via a loudspeaker. A visual display of the warning information is particularly useful, for example, on a vehicle display, which can specifically show the affected storage area to be monitored. If the classification function also identifies a specific object class, this can be used for further visualization or more generally to tailor the warning information.

[0026] In a further advantageous embodiment of the invention, the warning condition can be checked at least during and / or immediately after the vehicle is started and / or a journey begins. In such a case, the driver is informed directly at the beginning of an operating phase or upon commencement of the journey that a potential safety hazard may exist due to an object, for example, on the parcel shelf. The driver can then immediately increase safety if necessary, for example, by removing the object in question. However, it is particularly advantageous, especially with regard to newly detected and / or moving objects, to also check the warning condition while the vehicle is in motion.Therefore, if a potentially hazardous object is added to a monitored storage area at a later time, for example, because it was placed there by an inmate, security can be immediately enhanced by issuing a corresponding warning. This can occur, for instance, if an object initially detected and classified by light, which is being tracked, is only later moved to one of the monitored storage areas.

[0027] The assistance functionality can be activated and / or deactivated based on user input. It is therefore conceivable that the assistance functionality (and thus the output of warning information) can be deactivated via a user interface in the vehicle, for example, a central touchscreen display. When activated, the assistance functionality remains active until it is deactivated by the vehicle user. To increase safety, it may be advisable to have the assistance functionality activated as the default setting upon delivery of the vehicle.

[0028] As already mentioned and explained, the storage area to be monitored is at least one parcel shelf and a portion of the dashboard that runs at least essentially horizontally.

[0029] Other potentially hazardous storage areas, such as side storage surfaces in the doorway, can also be taken into account.

[0030] In addition to the method, the invention also relates to a motor vehicle comprising at least one interior camera for recording image data of the interior, at least one radar sensor for acquiring radar data of the interior, at least one storage area for objects in the interior to be monitored, in particular at least one parcel shelf, an output device, and a control device configured to carry out the method according to the invention. All aspects relating to the method according to the invention can be applied analogously to the motor vehicle according to the invention and vice versa, so that the aforementioned advantages can also be obtained with the motor vehicle.

[0031] The control unit, which may in particular comprise at least one control device, preferably has at least one processor and / or at least one storage medium. Functional units can be formed by hardware and / or software to carry out steps of the method according to the invention. For example, the control unit may have at least one interface for receiving image data and radar data, a classification unit for applying the classification function, a monitoring unit for monitoring compliance with the warning condition, and an output unit for outputting the product information. Further functional units, for example a tracking unit, may also be provided accordingly.

[0032] Further advantages and details of the present invention will become apparent from the exemplary embodiments described below and from the drawings. These show: Fig. 1 a flowchart of an embodiment of the method according to the invention, Fig. 2. a section of a motor vehicle with a parcel shelf as the area to be monitored, Fig. 3 a schematic explanation of a tracking situation, and Fig. 4 a schematic diagram of a motor vehicle according to the invention.

[0033] Fig. Figure 1 shows a flowchart of an embodiment of the inventive method for implementing an assistance function for monitoring the interior of a motor vehicle with regard to objects placed in the vehicle's interior. In this instance, the vehicle has several interior cameras for recording image data of the interior and several radar sensors for acquiring radar data of the interior. A parcel shelf is provided in the interior as the storage area to be monitored for objects.

[0034] In step S1, as soon as the vehicle is put into operation, i.e., an operating phase begins, image data is captured by the interior cameras and radar data by the radar sensors and made available via corresponding interfaces to a control unit that implements the assistance functionality. After an evaluation of detected objects, in step S2, the loose objects located in the interior of the vehicle, even those outside the storage area to be monitored, are classified by a trained classification function, which in this case also determines a specific object class to which a hazard class is assigned.Two hazard classes are used: "potentially dangerous" for objects that could pose a safety risk during hard braking and / or accidents originating from the monitored storage area, and "non-hazardous" for objects that should pose no or only a very minor risk, such as stuffed animals. The classification function utilizes not only the geometric shape of the objects, as described by radar data combined with image data, but also the color and texture provided by the image data. It is assumed that even during nighttime operation, the vehicle's interior lighting is active, at least when entering the vehicle, allowing useful camera data regarding color and texture to be collected.Thus, at the beginning of a vehicle's operating phase, a hazard class can be assigned to each detectable object. However, for objects that only become visible in darkness, a reliable determination of the hazard class using the classification function may not be possible.

[0035] Nevertheless, to increase safety, the objects will at least temporarily be marked as "potentially dangerous".

[0036] In step S3, the objects previously detected and classified as "potentially dangerous" are tracked, as far as possible, using radar and camera data. This means their current position is determined, particularly from their previous position, to ascertain whether any objects subsequently enter the storage area to be monitored, in this case, the parcel shelf.

[0037] In step S4, it is then checked whether a warning condition is met. In this case, the warning condition is always met if an object classified as potentially dangerous is detected in the storage area being monitored. If this is the case, in step S5, a warning message is issued to the occupants visually and / or audibly via an output device in the vehicle. The warning message can specify the particular object class and the storage area being monitored, for example, "Attention! Laptop on the parcel shelf!" or a corresponding visual representation on a display, such as a diagram. If the warning condition is not met, the system returns to step S1, as monitoring continues throughout the entire operational phase, especially while driving.This means that a warning can be given both at the beginning of the operational phase, especially when the journey starts, but also if a potential hazard situation arises at a later time due to a potentially dangerous object in the storage area to be monitored.

[0038] Fig. Figure 2 schematically depicts an initial situation, for example, at the start of a journey, in the interior 1 of a partially indicated motor vehicle 2, which is schematically bounded here by a vehicle roof 3, a rear window 4, and a rear seat 5. A parcel shelf 6, which serves as the storage area 7 to be monitored, extends between the rear window 4 and the backrest of the rear seat 5, forming a further boundary. A potentially dangerous object 8, such as a water bottle, is already placed on the parcel shelf 6 at the start of the journey. This object is robustly detected and classified in steps S1 and S2 using image and radar data, so that a warning message can be issued according to step S5 immediately at the beginning of the operational phase, i.e., after the motor vehicle 2 has started or at the start of a journey.

[0039] Fig. Figure 3 shows a potential situation during night driving in the same scenario. Upon entering and starting the vehicle, i.e., at the beginning of the operational phase, the interior lighting 10 of the vehicle 2 is initially active, allowing an object 9 located outside the storage area 7 to be reliably detected and classified, as the image data displays color and texture. Even after the interior lighting is switched off, the object 9, classified as potentially dangerous, can still be tracked thanks to the radar data in step S3.If, during the journey, for example by an occupant, the object 9 is moved from the first position 11 (shown in light) according to arrow 12 to a second position 13 (shown in dark) on the parcel shelf 6 as the storage area 7 to be monitored, as determined by the tracking in step S3, the fulfillment of the warning condition is recognized in step S4 and a corresponding warning information can also be issued during the journey in step S5, which enables the occupants to remove the object 9 from the parcel shelf 6 again for safety.

[0040] Even situations where an object, for example after being taken out of a bag, only appears in the dark during a journey, can be handled by the procedure described here, since it is detected based on the radar data and, even if a reliable classification as potentially dangerous or harmless is not possible by the trained classification function, it is classified as potentially dangerous for safety reasons and a warning can be issued accordingly.

[0041] Fig. Figure 4 shows a schematic diagram of the motor vehicle 2 according to the invention, in which, in addition to the storage area to be monitored (which is only indicated), two interior cameras 14 and two radar sensors 15 are shown by way of example. A central display 17 can be used as the output device 16, which also allows for the basic activation or deactivation of the assistance functionality. The motor vehicle also has a control unit 18, which is used to carry out the described method according to [reference to relevant section]. Fig. 1 is designed and can, for example, include at least one control unit. The control unit 18 can have functional units in addition to the interfaces, for example a classification unit, a monitoring unit, a tracking unit, and an output unit.

[0042] It should also be noted here that the radar sensors 15 are designed as ultra-wideband radar sensors. They also serve to detect children inside the interior 1.

Claims

[1] Method for monitoring the interior of a motor vehicle (2) with regard to objects (8, 9) placed in the interior (1), wherein the motor vehicle (2) has at least one interior camera (14) for recording image data of the interior (1), at least one radar sensor (15) for recording radar data of the interior (1) and several storage areas (7) to be monitored for objects (8, 9) in the interior (1), wherein the storage areas (7) to be monitored are a parcel shelf (6) and a portion of an instrument panel extending at least substantially horizontally, wherein the method comprises, within the framework of an assistance functionality: - Classification of all objects (8, 9) located in the storage area (7) as potentially dangerous or harmless by means of a classification function which uses as input data both image data or data derived therefrom and radar data or data derived therefrom, wherein the hazard classes are defined differently for several storage areas to be monitored, so that the classification function determines a storage area-specific hazard class for each storage area, - upon fulfillment of a warning condition which indicates at least the presence of at least one object (8, 9) classified as potentially dangerous in at least one of the at least one storage area (7) to be monitored, output of a warning information to indicate the at least one object (8, 9) classified as potentially dangerous located in at least one of the storage area (7) to be monitored. [2] Method according to claim 1, characterized by , that a trained function is used as the classification function. [3] Method according to claim 1 or 2, characterized by , that the classification function evaluates not only the position and shape of an object to be classified (8, 9) as described by the input data, but also the color and texture of the object to be classified (8, 9). [4] Method according to any of the preceding claims, characterized by , that the classification function assigns objects (8, 9) to specific object classes, with each specific object class being assigned one of the two hazard classes. [5] Method according to any of the preceding claims, characterized by, that objects (8, 9) located outside the at least one storage area (7) to be monitored are also classified using the classification function and subsequently tracked, particularly at least in darkness, at least using radar data with regard to their placement in at least one of the storage areas (7) to be monitored. [6] Method according to any of the preceding claims, characterized by , that if the image data is insufficient due to darkness, objects (8, 9) that cannot be clearly classified by the classification function are classified as potentially dangerous and / or the warning condition is also met in at least one of the at least one storage area (7) to be monitored, even if objects (8, 9) cannot be clearly classified. [7] Method according to any of the preceding claims, characterized by , that an ultra-wideband radar sensor is used as the radar sensor (15). [8] Method according to any of the preceding claims, characterized by that the warning information is displayed visually and / or audibly. [9] Method according to any of the preceding claims, characterized by , that the warning condition is checked at least at and / or immediately after the start of the motor vehicle (2) and / or the start of a journey and / or, in particular with regard to newly detected and / or moving objects (8, 9), is also checked during a journey of the motor vehicle (2). [10] Method according to any of the preceding claims, characterized by that the assistance functionality can be activated and / or deactivated based on user input. [11] Motor vehicle (2) comprising at least one interior camera (14) for recording image data of an interior (1) of the motor vehicle (2), at least one radar sensor (15) for recording radar data of the interior (1), at least one storage area (7) to be monitored for objects (8, 9) in the interior (1), an output device (16) and a control device (18) which is configured to carry out a method according to one of the preceding claims.

Images