Method for monitoring at least one occupant of a motor vehicle, wherein the method serves in particular to monitor and detect possible hazardous situations for at least one occupant.

Abstract

Method for monitoring at least one occupant (2) of a motor vehicle (1), comprising at least the following process steps: a) receiving data from at least one interior detection system (10) and b) automatically determining the state of the at least one occupant (2) from the data acquired in step a); wherein in step b) it is cyclically checked whether a visual appearance of the at least one occupant (2) changes, wherein the visual appearance of the at least one occupant (2) is checked at least using a visual appearance model.

Description

State of the art

[0001] Driver monitoring camera systems (DMC) and occupant monitoring cameras (OMC) are particularly well-known for monitoring vehicle occupants. Such systems can be used, for example, for (partially) autonomous or fully automated driving. In particular, these systems are used to analyze the driver's reaction time. Disclosure of the invention

[0002] This document presents a particularly advantageous method for monitoring at least one occupant of a motor vehicle. The dependent claims describe particularly advantageous further developments of the method.

[0003] In step a) of the described procedure, data from at least one indoor sensor is received.

[0004] The at least one interior sensing system preferably comprises at least one interior camera directed at the occupants of the vehicle. Video signals can be generated by the at least one interior camera, which can be received as data according to step a). The at least one interior camera is preferably a DMC camera. The interior camera can also be referred to as a vehicle interior monitoring camera. The interior sensing system preferably further comprises a processor, such as a CPU (Central Processing Unit) or a SoC (System On Chip).

[0005] The data recorded by the at least one interior sensor are preferably transmitted by the at least one interior camera and / or by the processor and received by a control unit, which is specifically designed and configured to carry out the described method. The control unit is preferably located in the motor vehicle. Alternatively, however, it is also possible for the data according to step a) to be received by a computer system outside the motor vehicle. In particular, a computer system can be provided that is capable of receiving and processing data from a multitude of motor vehicles. The data can be exchanged directly between the at least one interior sensor and the data receiver (i.e., in particular, the control unit and / or the computer system).However, it is also possible that the data passes through intermediate stations, for example, for distributing signals to multiple different receivers and / or for signal amplification. Such intermediate stations can be used, for instance, to increase the range between indoor detection and the receiver.

[0006] In step b) of the described procedure, the condition of at least one occupant is automatically determined from the data recorded in step a).

[0007] If the condition of occupants in a large number of vehicles is monitored according to the described procedure, a correspondingly large amount of data must be analyzed. Transmitting and receiving video data from a vehicle, or even from multiple or numerous vehicles, requires immense bandwidth. In this case, in particular, it is intended that the recording of the condition of at least one occupant is automated. This is generally considerably more practical than transmitting video data that is subsequently analyzed.

[0008] The condition recorded here refers specifically to the inmate's vital status. In some variations, the recorded condition may also be the inmate's emotional state or include information regarding the inmate's emotional state.

[0009] The automated determination of the condition of at least one occupant means, in particular, that the condition is assessed using technical means based on predefined criteria. This is preferably done by a software algorithm, which is preferably implemented in the control unit or computer system. The mere viewing of a video image by a person should not be considered an automated evaluation. Even a large amount of data can be analyzed and pre-sorted based on the predefined criteria. In the event of anomalies, a human (referred to here as an operator) preferably assesses the situation in addition to the automated evaluation. Such an operator is preferably located in a control center outside the vehicle and is preferably a trained professional.The remaining, less conspicuous situations are preferably filtered out by the software algorithm.

[0010] Preferably, the data received according to step a) is processed in real time in step b). This can be achieved, in particular, by performing DMC basic image processing and / or head / eye tracking. The image processing can also include image preprocessing with, for example, error correction and contrast enhancement, as well as segmentation and feature extraction and / or classification. Preferably, a dedicated CPU or SOC in the vehicle has sufficient computing capacity for the described data processing steps.

[0011] Software preferably used in the described method for image signal processing preferably includes, in particular, a CV algorithm (CV: computer vision). When using the CV algorithm, training sequences can be employed, for example, via deep learning algorithms, especially using support vector machines. The use of other (image) data, which can be provided, for example, via an interface from a smart device, can also be used to train the CV algorithms.

[0012] In a preferred embodiment, the method further comprises the following process step: c) Transmitting at least one signal describing the state of the at least one occupant as determined according to step b) via a radio link.

[0013] In this embodiment, it is preferred that the data according to step a) is received via a control unit within the vehicle. In this case, the at least one signal is preferably generated by the control unit. The signal thus generated can, in particular, be transmitted via a mobile communication connection. For this purpose, the signal output by the control unit is preferably directed to a mobile communication transmitter of the vehicle and transmitted from there via the mobile communication connection.

[0014] The receiver of the at least one signal could in particular be a server which has appropriate receiving devices and which is designed and equipped for receiving and processing such signals, in particular from a large number of motor vehicles.

[0015] The signal transmitted according to step c) preferably encodes the result of the state determination according to step b). This means, in particular, that step c) does not merely consist of continuously transmitting raw data from the at least one indoor sensor (i.e., regardless of a state determined according to step b).

[0016] In particular, the at least one signal transmitted according to step c) may contain information about whether the condition determined according to step b) shows any anomalies that might indicate a dangerous situation, for example, so that an assessment by a human or the operator described above is required.

[0017] It is also possible for the data in step a) to be received by a computer system outside the vehicle. For this purpose, the raw data generated by the interior sensing system can be transmitted to the computer system, for example, via a mobile network connection. Such a transmission does not constitute transmission within the meaning of step c). The data can then be processed in the computer system. The signal generated according to step c) can then be generated by the computer system and transmitted to the server.

[0018] In addition to the signal transmitted according to step c), which describes the state of at least one occupant, raw data can also be transmitted. In particular, the following data can be transmitted: - a picture signal of a still image of the occupants or the vehicle interior and - a video signal of the occupants or the vehicle interior.

[0019] In a further preferred embodiment of the method, in step b) the health status of the at least one occupant is determined.

[0020] The described procedure can therefore also be referred to as an "Automatic Visual Emergency Status Report" (AVES). It is thus a method for the visual assessment and transmission of injury and vital status information of injured occupants of a motor vehicle using interior vehicle monitoring.

[0021] Parameters of an occupant's vital status can be determined, in particular, by non-contact pulse and respiratory rate measurement using the so-called "Eulerian Video Magnification." This involves analyzing video data by amplifying minor fluctuations in intensity values ​​of significant image areas across successive video frames.

[0022] The health status of an occupant as determined in step b) may include, in particular, injuries (such as contusions, lacerations, and / or fractures), especially injuries to the face and head (for example, in the form of a traumatic brain injury or injuries to the mouth / oral cavity / tongue / teeth, nose, or ears). Injuries to the ears or nose, and internal injuries (especially head and brain injuries), may be detected, for example, by leakage of fluids such as blood or ventricular fluid. Furthermore, the health status as determined according to step b) preferably includes information on injuries to the visible extremities and upper body (such as to the neck, nape, spine, shoulders, or arms).In particular, the signal transmitted according to step c) (including CV result data) can be sent to an emergency call center. Furthermore, an emergency call with vital status information can be transmitted directly and ad hoc via Wi-Fi Direct to other road users / pedestrians within a radius of up to 500 meters, who can then provide initial first aid to the injured. Wi-Fi Direct also offers the possibility, in the event of a poor mobile network connection, of transmitting an emergency call via an ad-hoc network connected to several nearby pedestrians via Wi-Fi Direct. As soon as a participant in the ad-hoc network has sufficient mobile network reception, they can then place an emergency call via the mobile network.

[0023] Alternatively or in addition to transmitting the at least one signal according to step c), the condition of the at least one occupant, determined according to step b), is preferably displayed on a vehicle display. This allows information obtained according to the described procedure to be communicated to emergency services and / or first responders on site. Specifically, the display can show vital status information of the occupants for initial care. Depending on the condition determined according to step b), instructions for first responders can also be displayed. Furthermore, the display can indicate that an emergency call has been made and / or when the arrival of emergency services is expected. The display can be, for example, a central display or a head-up display. The display can be considered a human-machine interface (HMI).It is also possible to display the described information as an HMI on a smart device (such as a smartphone) connected to the vehicle's on-board computer.

[0024] The present embodiment of the described method offers the particular advantage that it can detect after an accident whether one or more of the occupants require medical assistance. This can be especially beneficial if occupants are unconscious, so severely injured, and / or trapped in the vehicle that they cannot independently call for help. Furthermore, crucial information about the occupants' health can be transmitted to rescue workers even before they arrive at the scene of the accident. In particular, after a vehicle accident, important information about the condition, and if applicable, the extent of injuries and vital functions of the occupants can be transmitted to rescue workers, thus providing valuable information for recovery and rescue operations. This allows rescue efforts to be coordinated much more effectively and carried out more efficiently.In particular, valuable time can be saved before initial treatment and rescue of the victims.

[0025] Alternatively or in addition to the health status of the at least one occupant, the condition of the at least one occupant determined according to step b) may also include a threat situation for the at least one occupant.

[0026] In addition to detecting injuries after accidents, the described procedure can also detect violence against the occupants or threatening situations for the occupants (both external threats and threats among themselves). In such a case, an emergency call can be made, for example, according to step c).

[0027] Furthermore, the described procedure can detect unauthorized entry into the vehicle. In such cases, an emergency call can be placed. This can prevent, or at least detect early on, the theft of the vehicle or items from within it. Image data of the intruders can also be stored or transmitted to the emergency response center for future investigations.

[0028] In a further preferred embodiment of the method, an alarm signal is issued if the condition of the at least one occupant determined according to step b) represents an emergency situation.

[0029] The alarm signal can be, in particular, a visual and / or audible signal. Preferably, the alarm signal is emitted audibly via a loudspeaker in the vehicle, so that other road users can be alerted to the emergency situation.

[0030] In another preferred embodiment of the method, in step b) it is cyclically checked whether a visual appearance of the at least one occupant changes.

[0031] The condition of at least one occupant can be determined, in particular, from their outward appearance. External injuries, in particular, are immediately recognizable. For example, body posture can indicate whether the occupant is conscious.

[0032] In a further preferred embodiment of the method, the visual appearance of the at least one occupant is checked at least using an appearance model.

[0033] For particularly reliable visual detection of injuries, a personal appearance model is preferably created for one or more of the occupants and updated cyclically. The appearance model can be considered, in particular, as a representation of the normal state. The appearance model preferably includes at least one head model.

[0034] The appearance model preferably includes information about the external appearance of various body parts of the occupant. This information is preferably assigned to the corresponding body parts. For example, the appearance model can include the information that an occupant's arm is free of any discernible injuries in a first cycle. In subsequent cycles, it can be checked whether any injuries have occurred on this arm. Information relating to the arm is then assigned to it. This preferably occurs regardless of the arm's location within the vehicle. Thus, the appearance model can detect changes in the external appearance of individual body parts of the occupant even when the occupant moves from one cycle to the next.

[0035] During the cyclical update of the appearance model, the current appearance of the respective occupant is preferably compared with the appearance created in one or more previous cycles. A significant change from one cycle to the next may indicate an injury to the occupant in question.

[0036] In another preferred embodiment of the method, the execution of step b) is triggered upon detection of a motor vehicle accident.

[0037] If an occupant is injured in an accident, not only can the injury be detected as a result of the accident according to step b), but the accident itself as the cause of the injury can also be detected directly, for example, via the vehicle's acceleration sensors. In the present embodiment, the condition of at least one occupant is checked in the event of a detected accident. This means that the accident is recognized as the cause of potential injuries, and in response, it is checked whether the occupants have been injured. Furthermore, the system can request the occupants to provide a status update on their current condition after the accident. This can be done via a visual or audible human-machine interface (HMI).

[0038] The control unit is preferably connected to sensors in the vehicle that can detect an accident. From the information obtained in this way, an accident signal can be generated, which can also be transmitted to the server via the mobile network connection. This signal can also include information about the type and severity of the accident.

[0039] In another preferred embodiment of the method, the interior lighting of the motor vehicle is intensified when an accident of the motor vehicle is detected.

[0040] The occupants can be visually detected using at least one interior sensor. This is more effective the better the interior is illuminated. For this purpose, interior lighting is provided, particularly in this embodiment. Sufficient interior illumination for the at least one interior sensor is especially advantageous in the event of an accident, enabling the detection of injuries. Accordingly, the interior lighting is intensified in this embodiment when an accident is detected, for example, via acceleration sensors.

[0041] In a further preferred embodiment of the method, the interior of the motor vehicle is illuminated at least in step a) with infrared interior lighting.

[0042] Infrared interior lighting can achieve homogeneous illumination of the interior that is almost or even completely independent of the ambient light conditions. Preferably, infrared interior lighting uses a wavelength in the range of 940 nm [nanometers]. At this wavelength, the infrared interior lighting is invisible to the occupants and therefore cannot dazzle or otherwise disturb them.

[0043] The infrared interior lighting preferably comprises one or more infrared emitters, which can be arranged at various positions within the vehicle, particularly near the interior sensor. Preferably, one infrared emitter is positioned in front of the driver and one in front of the front passenger, each directed towards them. The infrared emitters preferably illuminate so-called regions of interest (ROIs). These regions of interest can be particularly sensitive areas of the occupants, such as the face and / or upper body. Visible extremities of the occupants are also preferably illuminated by the interior lighting.

[0044] The interior lighting can be regulated and pulsed, especially for optimal illumination of the ROI, in relation to a variable distance of an occupant's head to the camera.

[0045] As a further aspect, a control unit is presented which is set up to monitor at least one occupant of a motor vehicle according to the described procedure.

[0046] The special advantages and design features described earlier for the process are applicable and transferable to the control unit, and vice versa.

[0047] In a preferred embodiment, the control unit is configured to determine data regarding the health status of at least one occupant and forward it to a server.

[0048] Furthermore, a computer program is presented that is configured to execute all steps of the described procedure. Additionally, a machine-readable storage medium on which the described computer program is stored is presented. The special advantages and design features described earlier for the procedure and the control unit are applicable and transferable to the computer program and the machine-readable storage medium.

[0049] Further details of the invention and an exemplary embodiment, to which the invention is not limited, are explained in more detail with reference to the drawings. They show schematically: Fig. 1: A representation of a motor vehicle with two occupants and an interior camera, Fig. 2: a representation of an arrangement for processing data related to indoor air quality monitoring Fig. 1. be recorded, Fig. 3: a first example of an injured inmate, Fig. 4: a second example of an injured inmate, and Fig. 5: a flowchart of the described procedure.

[0050] Fig. 1 shows a motor vehicle 1 with two occupants 2 , located in an interior 7 of the motor vehicle 1 are located. Via an initial interior camera 3 and a second interior camera 4 will the inmates 2 Captured. With infrared emitters 9 will the interior 7 (for the inmates) 2 illuminated (not visible).

[0051] In Fig. Figure 2 shows how the interior cameras work. 3 , 4 out of Fig. 1. The data captured will be processed. The interior cameras 3 , 4 are part of an indoor recording 10 , which also has a CPU 8and the infrared emitters 9 includes the interior cameras. 3 , 4 and the infrared emitters 9 are on the interior 7 of the motor vehicle 1 directed. From the indoor recording 10 Data is sent out that originates from a control unit. 11 be received. This concerns the step a) of the described procedure. In the control unit 11 A state of the two occupants is automatically determined. 2 determined from the received data. This corresponds to step b) of the described procedure. The control unit 11 is connected to a mobile phone transmitter 5 connected, with which a signal is transmitted via radio link to an emergency call center 6 can be transmitted. The signal describes the communication with the control unit. 11 determined condition of the occupants 2 Furthermore, the control unit 11 to a display 12connected via which the control unit 11 The information obtained can be displayed.

[0052] In the Fig. 3 and Fig. 4 are two examples of inmates 2 with various injuries shown. The injuries are visually identifiable and can therefore be seen via the first interior camera. 3 (the occupants shown in this example) 2 (is aligned) are recorded. The two infrared emitters are also shown. 9 , which also targeted the occupants 2 are directed. In Fig. The inmate has 3 2 Injuries to both eyes, upper body, and left arm. Additionally, blood is leaking from the nose and left ear. Fig. The inmate has 4 2 Two damaged teeth and an injury to the left eye. Additionally, blood is coming from the nose.

[0053] Fig. Figure 5 shows a flowchart of the procedure described here, which is particularly applicable to the device (for example, the device according to Fig. 2) can be carried out. The procedural steps can be identified. a) , b) and c) how they are carried out according to the procedure.

Claims

[1] Method for monitoring at least one occupant (2) of a motor vehicle (1), comprising at least the following procedural steps: a) Receiving data from at least one indoor sensor (10) and b) Automated determination of the condition of at least one occupant (2) from the data recorded in step a). [2] The method of claim 1, further comprising the following method step: c) Transmitting at least one signal describing the state of the at least one occupant (2) as determined according to step b) via a radio link. [3] Method according to any of the preceding claims, wherein in step b) a health status of the at least one occupant (2) is determined. [4] Method according to one of the preceding claims, wherein in step b) it is cyclically checked whether a visual appearance of the at least one occupant (2) changes. [5] Method according to claim 4, wherein the visual appearance of the at least one occupant (2) is checked at least using an appearance model. [6] Method according to any of the preceding claims, wherein the execution of step b) is triggered upon detection of an accident involving the motor vehicle (1). [7] Method according to one of the preceding claims, wherein the interior lighting of the motor vehicle is intensified in the event of a detected accident of the motor vehicle (1). [8] Method according to one of the preceding claims, wherein an interior (7) of the motor vehicle is illuminated at least in step a) with an infrared interior lighting. [9] Control unit (11) configured to monitor at least one occupant (2) of a motor vehicle (1) according to a method according to one of the preceding claims. [10] Control unit (11) according to claim 9, wherein the control unit (11) is configured to determine data regarding the health status of the at least one occupant (2) and forward it to a server. [11] Computer program which is configured to perform all steps of the method according to any one of claims 1 to 8. [12] Machine-readable storage medium on which the computer program according to claim 11 is stored.

Images