OCCUPANT MONITORING SYSTEM FOR A VEHICLE

DE502019012451D1Active Publication Date: 2024-11-28ROBERT BOSCH GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE502019012451
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2018-10-24
Filing Date
2019-09-04
Publication Date
2024-11-28
Estimated Expiration
2039-09-04

AI Technical Summary

Technical Problem

Existing occupant monitoring systems in vehicles, particularly in those with autonomous driving functions, face challenges in accurately determining the position, posture, and physical condition of occupants, especially in dynamic situations, due to limitations in sensor reliability and privacy concerns with camera systems.

Method used

An occupant monitoring system utilizing ultrasonic sensors and microphones to detect breathing noises and evaluate physical condition, with an evaluation and control unit processing raw signals to determine characteristic resonance frequencies, stress levels, and posture, allowing for safer automated driving and airbag deployment decisions.

Benefits of technology

This solution provides a reliable and privacy-respecting method for monitoring occupants, enabling safer automated driving by accurately assessing driver stress and physical state, and optimizing airbag deployment, while being cost-effective and resistant to environmental interference.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The invention is based on an occupant monitoring system for a vehicle according to the preamble of independent patent claim 1.

[0002] In order to avoid injuries caused by the airbag in so-called out-of-position situations, occupant protection systems known from the state of the art use occupant monitoring systems with at least one interior sensor to determine the position and posture of occupants.

[0003] This makes it possible to determine whether an occupant is within the airbag's deployment area before the airbag is deployed. Such interior sensor technology usually includes optical sensors or cameras to record the occupant's current position and posture. Particularly in vehicles with autonomous driving functions, the driver can be allowed to assume many different positions and perform activities during automated driving. For example, the driver can use virtual reality glasses, operate electronic devices such as a smartphone, tablet, laptop, etc., make phone calls without a hands-free system (with the smartphone to their ear), read (a book, letter, newspaper, etc.), paint, eat, and / or drink. In addition, the backrest can be adjusted toward the sleeping position, the seat can be moved far back, or their feet can be pulled up on the seat.

[0004] From DE 10 2014 211 501 A1 an occupant monitoring system according to the preamble of claim 1 is known.

[0005] DE 10 2008 002 232 A1 discloses an adaptive method and a device suitable for implementing this method for determining the distance and / or speed of an object relative to a vehicle. Ultrasonic pulses are emitted, reflected by the object, and received again as an echo. The emission of a second ultrasonic pulse is triggered by the reception of the echo of a previously emitted first ultrasonic pulse, thereby gaining valuable time in dynamic situations. Disclosure of the invention

[0006] The occupant monitoring system for a vehicle with the features of independent patent claim 1 has the advantage that ultrasonic sensors and / or microphones of an interior sensor system can detect the breathing sounds of the occupants and evaluate them to determine the physical condition of each occupant. Furthermore, the position and activity of each occupant can be determined using the ultrasonic sensors and / or microphones.

[0007] Turbulent wind noise, such as that produced by breathing, generates ultrasonic signals in a frequency range that can be detected by conventional ultrasonic sensors and / or microphones. Breathing produces many characteristic resonant frequencies similar to those produced by speaking.

[0008] Embodiments of the present invention provide an occupant monitoring system for a vehicle having at least one autonomous driving function. The occupant monitoring system for a vehicle comprises at least one sensor system for detecting a current driving state of the vehicle, an interior sensor system for detecting an interior state, and at least one evaluation and control unit. The interior sensor system comprises a plurality of ultrasonic sensors and / or microphones distributed throughout the vehicle interior, wherein the evaluation and control unit receives and further processes raw signals from the ultrasonic sensors and / or microphones. Based on the raw signals from the ultrasonic sensors and / or microphones, the evaluation and control unit determines breathing sounds of the occupants, which have characteristic resonant frequencies.

[0009] In this case, the evaluation and control unit can be understood to be an electrical device, such as a control unit, in particular an airbag control unit, which processes or evaluates detected sensor signals. The evaluation and control unit can have at least one interface, which can be embodied in hardware and / or software. In a hardware embodiment, the interfaces can, for example, be part of a so-called system ASIC, which contains a wide variety of functions of the evaluation and control unit. However, it is also possible for the interfaces to be separate integrated circuits or to consist at least partially of discrete components. In a software embodiment, the interfaces can be software modules, which are present, for example, on a microcontroller alongside other software modules.Also advantageous is a computer program product with program code stored on a machine-readable medium such as a semiconductor memory, a hard disk memory or an optical memory and used to carry out the evaluation when the program is executed by the evaluation and control unit.

[0010] The sensor technology for detecting a current driving state of the vehicle is understood below to mean sensor devices which, for example, support automatic driving of the vehicle or detect the surroundings of the vehicle for crash detection.

[0011] In contrast to interior camera systems, the ultrasonic sensors and / or microphones are insensitive to dirt and dust and cannot be blinded by other vehicles. Even creative patterns on seat covers or, for example, printed clothing have no influence on breathing detection. Furthermore, ultrasonic sensors are extremely cost-effective. Automated driving can be made more secure because the breathing rate determined from breathing sounds can be used to determine the driver's stress level and thus the driver's physical state of health. This allows the evaluation and control unit to decide whether control can be transferred to the driver or whether the vehicle should be parked safely. Furthermore, the use of ultrasonic sensors and / or microphones in conjunction with breathing sound detection eliminates the need for seat occupancy sensors.In contrast to the use of cameras, the occupants do not feel their privacy is disturbed by the use of ultrasonic sensors.

[0012] The measures and further developments listed in the dependent claims enable advantageous improvements to the occupant monitoring system for a vehicle specified in independent patent claim 1.

[0013] It is particularly advantageous that first ultrasonic sensors and / or first microphones for detecting the occupants' breathing sounds can be arranged in areas of the interior with low air turbulence. Furthermore, second ultrasonic sensors and / or second microphones can be provided for detecting background noise, whereby the evaluation and control unit can use the raw signals from the second ultrasonic sensors and / or the second microphones to compensate for background noise when evaluating the breathing sounds.

[0014] In a further advantageous embodiment of the occupant monitoring system, the evaluation and control unit can convolve the raw signals from the ultrasonic sensors and / or the microphones with themselves and / or calculate them using autocorrelation in order to extract the characteristic resonance frequencies and respiratory frequencies of the individual occupants from the breathing sounds. Furthermore, by evaluating the characteristic resonance frequencies and respiratory frequencies, the evaluation and control unit can determine the number of occupants and the physical health status of the individual occupants. The stress level of the occupants, in particular the driver, can also be detected and evaluated based on the breathing sounds. Not only the respiratory frequency and volume flow or the measured noise intensity can be used to determine the stress level, but also the characteristic resonance frequencies. The different resonance frequencies arise because the mucous membranes in the nose contract when tensed (e.g.due to adrenaline) and swell when relaxed, thereby influencing the resonance volume. The evaluation and control unit can receive and evaluate raw data from the ultrasonic sensors installed in the interior in order to detect and localize the breathing sounds. Wind noise from open windows and noises when driving on wet roads and puddles, as well as ultrasonic noises from the engine compartment are also recorded as noise by the ultrasonic sensors and / or microphones and overlay the breathing sounds. The evaluation and control unit can therefore advantageously also use raw data from external ultrasonic sensors and / or microphones and / or from ultrasonic sensors and / or microphones in the engine compartment for data fusion in order to be able to record and extract the breathing sounds even better.Data fusion with ultrasonic sensors and / or microphones from the engine compartment and the surrounding area naturally also improves object detection.

[0015] In a further advantageous embodiment of the occupant monitoring system, the evaluation and control unit can transmit the physical health status of the individual occupants together with an emergency call to an emergency call center after an accident.

[0016] In a further advantageous embodiment of the occupant monitoring system, the evaluation and control unit can determine the position and posture of each occupant by fusing and evaluating the raw signals. Furthermore, during an automated journey, the evaluation and control unit can decide, based on the driver's posture and physical health, whether control of the vehicle can be returned to the driver or whether the vehicle should be safely parked automatically.

[0017] In a further advantageous embodiment of the occupant monitoring system, the evaluation and control unit can use echolocation to calculate the positions of the faces of individual occupants. Based on the position and posture of each occupant, a higher-level occupant protection system can decide during an accident which protective measures are activated to protect the occupants. This allows, for example, airbags to be deployed in the event of an accident only if the deployed airbag does not pose a risk to the occupants based on the determined physical health status, posture, and activity of the occupants.

[0018] In a further advantageous embodiment of the occupant monitoring system, the interior sensor system can comprise at least one additional interior sensor, the signals of which the evaluation and control unit can use to check the plausibility and / or optimize the evaluation results of the raw signals from the ultrasonic sensors. The at least one additional interior sensor can be designed, for example, as a seat position sensor or as an optical sensor or lidar system. Since seat adjustment has a significant influence on the position of the occupants and their posture, the information from electrically adjustable seats can be used to check the plausibility or optimize object detection. The distance between the face and the ultrasonic sensors or microphones has a significant influence on the attenuation of the measured noise level.In addition, echolocation can be used to detect the position of the occupants' faces in order to compensate for the measured intensity of the noise level.

[0019] By fusing the sensor signals, the evaluation and control unit in embodiments of the invention can detect the seats occupied by people in the vehicle, their height, and their posture. The evaluation and control unit can then decide which protective measures are triggered in the event of an accident. During automated driving, the evaluation and control unit can decide, based on the detected posture or a detected driver state, whether control can be returned to the driver before the automated driving process ends, or whether the vehicle should be safely parked.If the evaluation and control unit cannot return control of the vehicle to the driver, for example, because the driver's hands are not on the steering wheel, the driver is under high stress, or the driver's feet are not in the footwell, the evaluation and control unit can terminate the journey by stopping at the side of the road or on the hard shoulder. Before doing so, the evaluation and control unit can, for example, issue at least one, preferably several, visual and / or acoustic warning signals.

[0020] An embodiment of the invention is illustrated in the drawing and explained in more detail in the following description. In the drawing, identical reference numerals designate components or elements that perform identical or similar functions. Short description of the drawing

[0021] Fig. 1 shows a schematic block diagram of an embodiment of an occupant monitoring system according to the invention for a vehicle. Embodiments of the invention

[0022] As from Fig. 1As can be seen, the illustrated embodiment of an occupant monitoring system 1 according to the invention for a vehicle comprises at least one sensor system 5 for detecting a current driving state of the vehicle, an interior sensor system 20 for detecting an interior state, and at least one evaluation and control unit 10. The interior sensor system 20 comprises a plurality of ultrasonic sensors US1, US2 and / or microphones M1, M2 arranged distributed throughout the vehicle interior, wherein the evaluation and control unit 10 receives and further processes raw signals from the ultrasonic sensors US1, US2 and / or the microphones M1, M2. The evaluation and control unit 10 determines breathing noises of the occupants, which have characteristic resonance frequencies, based on the raw signals from the ultrasonic sensors US1, US2 and / or the microphones M1, M2.In the illustrated embodiment, the vehicle 1 comprises a driver assistance system 3, which has at least one autonomous driving function AF1, AF2 for carrying out a fully automatic or at least a partially automatic driving operation, and a higher-level occupant protection system (not further specified), to which at least one occupant protection means SM1, SM2, SM3 is assigned.

[0023] In the illustrated embodiment, the interior sensor system 20 comprises first ultrasonic sensors US1 and first microphones M1 for detecting the breathing sounds of the occupants, which are arranged in areas of the interior with low air turbulence, and second ultrasonic sensors US2 and second microphones M2 for detecting background noise. The evaluation and control unit 10 uses the raw signals from the second ultrasonic sensors US2 and the second microphones M2 to compensate for background noise when evaluating the breathing sounds. In one embodiment not shown, the interior sensor system 20 comprises only ultrasonic sensors US1, US2. In another embodiment not shown, the interior sensor system 20 comprises only microphones M1, M2. In another embodiment not shown, the interior sensor system 20 comprises ultrasonic sensors U1 for detecting breathing sounds and microphones M2 for detecting background noise.In yet another embodiment not shown, the interior sensor system 20 comprises microphones M1 for detecting breathing noises and ultrasonic sensors U1 for detecting background noises.

[0024] Suitable mounting locations for the ultrasonic sensors U1, U2 and / or the microphones M1, M2 are locations that are sufficiently large to accommodate the housing of the ultrasonic sensors U1, U2 and / or the microphones M1, M2, so that the sensor surface can lie flat on the surface of the interior and the remaining components, except for the sensor surface, are hidden invisibly behind the paneling. In addition, the ultrasonic sensors U1, U2 and / or the microphones M1, M2 are mounted so that moving parts of the interior (seats, steering wheel, etc.) and body parts of occupants are as far away as possible. Since the first ultrasonic sensors U1 and / or the first microphones M1 can be disturbed by other air turbulence when detecting breathing sounds, the first ultrasonic sensors U1 and / or the first microphones M1 are also mounted away from ventilation slots and window openings.In addition, when the sensor surfaces are in flat contact with the interior trim, they are aligned as closely as possible towards the occupants' body parts. With this in mind, installation locations near an interior mirror and in the area of ​​interior lighting are particularly suitable. The A-pillar, B-pillar, and C-pillar can also be used as installation locations for the second ultrasonic sensors U2 and / or the second microphones M2 in order to detect air turbulence caused by open windows as background noise. Mounting the first ultrasonic sensors U1 and / or the first microphones M1 in the dashboard, as far away from the ventilation slots as possible, is also advantageous. It is also conceivable here to point the ultrasonic sensors U1, U2 towards a windshield so that it reflects the sound onto the driver and front passenger, thus increasing the distance to the occupants and air turbulence.

[0025] For object detection inside the vehicle, the same ultrasonic sensors U1, U2 can generally be used as outside the vehicle. However, the range of the ultrasonic sensors U1, U2 is significantly shorter inside the vehicle, which is why performance losses are possible and therefore the ultrasonic sensors U1, U2 can also be installed invisibly behind the paneling. In general, shorter echo signals are preferable due to the shorter distances in the interior, as the ultrasonic sensors U1, U2 require a certain decay time. Therefore, objects that are too close to the ultrasonic sensors U1, U2 cannot be detected by a single ultrasonic sensor U1, U2. However, it is possible to detect objects with increased intensity using cross echoes, since in this case the echo from one ultrasonic sensor U1, U2 is received by another ultrasonic sensor U1, U2 and the decay time is therefore irrelevant.

[0026] In the illustrated embodiment, the evaluation and control unit 10 convolves the raw signals from the ultrasonic sensors US1, US2 and / or the microphones M1, M2 with themselves to extract the characteristic resonance frequencies and respiratory rates of the individual occupants from the breathing sounds. Additionally or alternatively, the evaluation and control unit 10 can calculate the raw signals from the ultrasonic sensors US1, US2 and / or microphones M1, M2 using autocorrelation. In this way, resonance frequencies in the ultrasonic range can be detected particularly well. By evaluating the characteristic resonance frequencies and respiratory rates, the evaluation and control unit 10 in the illustrated embodiment determines the number of occupants and the physical health status of the individual occupants. Furthermore, the occupants can be identified by pattern recognition of the resonance frequencies, similar to using a fingerprint.Therefore, it is also possible to verify access authorization to the vehicle based on breathing sounds. The driver's stress level can also be determined based on breathing sounds. Not only the breathing rate and volume flow, or the measured noise intensity, but also the characteristic resonance frequencies can be used to determine the stress level.

[0027] In the illustrated embodiment, the evaluation and control unit 10 transmits the physical health status of the individual occupants along with an emergency call to an emergency call center after an accident. This allows the emergency control center to obtain an initial impression of the possible health status or physical condition of the individual occupants.

[0028] In addition, the evaluation and control unit determines the position and posture of the individual occupants by fusing and evaluating the raw signals. Based on the driver's posture and physical state of health, the evaluation and control unit 10 decides during an automated journey whether control of the vehicle can be returned to the driver or whether the vehicle should be safely parked automatically. To improve the determination of the position and / or posture of the occupants, the evaluation and control unit 10 in the illustrated embodiment calculates the positions of the faces of the individual occupants using echolocation. Based on the position and posture of the individual occupants, the higher-level occupant protection system 1 decides during an accident which protective devices SM1, SM2, SM3, such as belt tensioners, airbags, etc., should be activated to protect the occupants.

[0029] In the illustrated embodiment, the interior sensor system 20 comprises additional interior sensors IS1, IS2, whose signals are used by the evaluation and control unit 10 to check the plausibility and / or optimize the evaluation results of the raw signals from the ultrasonic sensors US1, US2 and / or the microphones M1, M2. Since seat adjustment also has a significant influence on the position of the occupants and their posture, the additional interior sensors IS1, IS2 in the illustrated embodiment provide information from electrically adjustable seats to check the plausibility or optimize object detection. The distance between the face and the ultrasonic sensors U1, U2 and / or the microphones M1, M2 has a significant influence on the attenuation of the measured noise level. Therefore, the detected positions of the occupants' faces are additionally used to compensate for the measured intensity of the noise level.Additionally or alternatively, optical sensors such as cameras or a lidar system can be used as additional interior sensors IS1, IS2 to determine the position and posture of individual occupants. However, the development of algorithms for robust object detection is very expensive due to the high susceptibility of the cameras to interference. The use of lidar is also very complex and therefore expensive. Optical methods generally have the disadvantage that dust is generated when the airbags are deployed, which can settle on the lenses and impair visibility and object detection. In addition, the lenses must be regularly cleaned of dust and dirt to ensure functionality. Furthermore, optical systems have problems with black clothing.

Claims

1. Occupant monitoring system (1) for a vehicle that has at least one autonomous driving function (AF1, AF2), having at least one sensor system (5) for detecting a current driving state of the vehicle, an interior sensor system (20) for detecting an interior state and at least one evaluation and control unit (10), characterized in that the interior sensor system (20) comprises multiple ultrasonic sensors (US1, US2) and / or microphones (M1, M2) arranged in the vehicle interior in a distributed manner, wherein the evaluation and control unit (10) receives raw signals of the ultrasonic sensors (US1, US2) and / or of the microphones (M1, M2) and processes said signals further, wherein the evaluation and control unit (10) takes the raw signals of the ultrasonic sensors (US1, US2) and / or of the microphones (M1, M2) as a basis for determining breathing sounds from the occupants, which have characteristic resonant frequencies, and wherein the evaluation and control unit (10) convolutes the raw signals of the ultrasonic sensors (US1, US2) and / or of the microphones (M1, M2) with itself and / or accounts for said signals by autocorrelation in order to extract the characteristic resonant frequencies and breathing frequencies of the individual occupants from the breathing sounds, the evaluation and control unit (10) evaluating the characteristic resonant frequencies and breathing frequencies in order to determine a number of occupants and a physical state of health of the individual occupants.

2. Occupant monitoring system (1) according to Claim 1, characterized in that first ultrasonic sensors (US1) and / or first microphones (M1) for capturing the breathing sounds of the occupants are arranged in areas with low turbulence in the interior, and there is provision for second ultrasonic sensors (US2) and / or second microphones (M2) for capturing noise, wherein the evaluation and control unit (10) uses the raw signals of the second ultrasonic sensors (US2) and / or of the second microphones (M2) to compensate for noise when evaluating the breathing sounds.

3. Occupant monitoring system (1) according to Claim 1 or 2, characterized in that the evaluation and control unit (10) relays the physical state of health of the individual occupants to an emergency centre together with an emergency call following an accident.

4. Occupant monitoring system (1) according to one of Claims 1 to 3, characterized in that the evaluation and control unit (10) determines the position and body posture of the individual occupants by fusing and evaluating the raw signals.

5. Occupant monitoring system (1) according to Claims 1 or 2 and 4, characterized in that during an automated journey the evaluation and control unit (10) takes the body posture and the physical state of health of a driver as a basis for deciding whether control of the vehicle is transferred back to the driver or whether the vehicle is safely parked in an automated manner.

6. Occupant monitoring system (1) according to Claim 4 or 5, characterized in that the evaluation and control unit (10) uses echolocation to compute positions of faces of the individual occupants.

7. Occupant monitoring system (1) according to Claims 1 to 6, characterized in that the interior sensor system (20) comprises at least one further interior sensor (IS1, IS2), the signals of which are used by the evaluation and control unit (10) to check the plausibility of and / or optimize the evaluation result pertaining to the raw signals of the ultrasonic sensors (US1, US2) and / or of the microphones (M1, M2).

8. Occupant monitoring system (1) according to Claim 7, characterized in that the at least one further interior sensor (IS1, IS2) is a seat position sensor or an optical sensor or a lidar system.

9. Occupant protection system comprising an occupant monitoring system (1) according to one of Claims 4 to 8, wherein during an accident the occupant protection system takes the position and body posture of the individual occupants as a basis for deciding which protective means (SM1, SM2, SM3) are activated to protect the occupants.