Method for calculating cardiac activity parameters of a vehicle occupant, computing unit and computer program

Abstract

The present invention relates to a method for calculating cardiac activity parameters of one or more vehicle occupants (30), a computing unit (500), and a computer program for carrying out this method. The method comprises determining a cardiac activity signal (100) of one or more vehicle occupants (30) using a camera (10), determining a respiratory activity signal (100) of the vehicle occupant(s) (30) using a radar sensor (20), adjusting the determined cardiac activity signal (100) of the vehicle occupant(s) (30) as a function of the determined respiratory activity signal (200), and determining cardiac activity parameters of the vehicle occupant(s) (30) as a function of the adjusted cardiac activity signal (300).

Description

The present invention relates to a method for calculating cardiac activity characteristics of a vehicle occupant, to a computing unit and to a computer program for carrying out the same.BACKGROUND OF THE INVENTIONMonitoring the health and general physical state of occupants of a vehicle, in particular of a driver, can contribute to increasing the safety in road traffic.For example, information regarding the state of health of the driver can be used by driver assistance systems in order to cause an emergency stop of the vehicle, for example, in emergency situations. This makes it possible to protect both the vehicle occupants and also potential victims of an accident that are not involved, and to avoid serious accidents.In this context, WO 2023 / 031339 A1 describes a method for determining vital functions of a vehicle occupant, with cardiogram signals being determined on the basis of magnetic field sensors integrated in the vehicle seat, by means of which cardiogram signals the vital functions can be calculated. In this case, however, it is necessary to mount additional sensors, namely the seat-integrated magnetic field sensors, in the vehicle interior. In contrast, it would be advantageous to be able to determine vital functions (heart activity and breathing activity) of vehicle occupants with already present sensors with sufficient accuracy in order to be able to infer therefrom, for example, a physical state of the driver.Already existing sensors are, for example, interior cameras, which are usually used for object detection and for detecting persons in / around the vehicle. In addition, radar sensors may be mounted in the vehicle interior, e.g., on the roof lining, which may be used, for example, to detect persons / animals enclosed in the parked vehicle (e.g., child-presence detection-CPD).A cardiac activity of a vehicle occupant can be detected, for example, photoplethysmographically with an interior camera. In photoplethysmography (PPG), light is usually radiated into tissue and at the same time the light intensity re-radiated from this tissue is measured by means of a photosensor. The basic principle of photoplethysmography is based on a dependence of the reflection and transmission properties of the tissue on the blood volume, which in turn changes with the heart beat. Light transmission through the tissue decreases during the heart beat phase (systole) and increases during the relaxation phase (diastolic).Methods in which a camera is used as a photosensor for recording a PPG signal are known under the names "Remote Photoplethysmography (rPP)" or "Imaging Photoplethysmography (iPGG)" and make it possible to obtain information about its heart activity, for example by detecting a change in the blood volume by the heart beat (perfusion rhythm) in a face region of a subject.If such information is detected by a vehicle occupant, it can be used to realize or improve new or existing vehicle functionalities (e.g. classification of vehicle occupants, detection of critical (health) states of the vehicle occupants, etc.).Disclosure of the InventionAccording to the invention, a method for calculating cardiac activity characteristic values of a vehicle occupant and a computing unit and a computer program for carrying them out are proposed. Advantageous embodiments are the subject matter of the dependent claims and of the following description.The invention makes it possible to determine cardiac activity characteristics of a vehicle occupant with increased accuracy from a camera-based cardiac activity signal. For this purpose, interference signals are identified by means of a radar sensor, which result e.g. from a breathing activity of the vehicle occupant, and are removed from the camera-based heart activity signal before the heart activity characteristics are determined.A heart activity signal is to be understood as any signal that allows conclusions to be drawn about a person's heart activity. Expediently, signals which document an activity of the blood circulation of a person (e.g. properties of pulse waves which are caused by a heart action / a heart beat) are also intended to fall under the term "heart activity signal".In the method according to the invention, a cardiac activity signal of a vehicle occupant is determined by means of a camera. The camera can be, in particular, an interior camera of the vehicle. The interior camera can be, for example, a color camera, such as an RGB camera. A near infrared (NIR) camera and a combined color infrared (RGB-IR) camera are also possible. Any other type of camera suitable for detecting a cardiac activity signal of a vehicle occupant may also be used. The camera can be installed, for example, on a rear-view mirror or in the region of the rear-view mirror (e.g. above / below the latter) in the vehicle interior. Any other position suitable for detecting a cardiac activity signal of a vehicle occupant is also possible.The cardiac activity signal determined by means of the camera can be, in particular, a photoplethysmographic signal (PPG signal), which can be determined in a known manner, for example from recordings of one or more face regions (region of interest, ROI) of the vehicle occupant. Suitable ROIs can be, for example, regions of forehead and cheeks of the vehicle occupant. It is also possible to determine skin regions of the vehicle occupant (e.g., palms and / or neck regions) other than ROI.For determining the PPG signal, for example, a video sequence with a specific number of images of the vehicle occupant can be recorded by means of the interior camera and the ROI can be identified in each image. The perfusion rhythm can then be determined in a spatially resolved manner from image pixels of the ROI, wherein an average value of the pixel values in the ROI can be formed in order to increase the signal-to-noise ratio of the PPG signal. When using an RGB camera, its red, blue and green color channels can be evaluated individually and spatially averaged. In particular, the green color channel can be used for determining the PPG signal, since at its wavelength the blood components oxy- and deoxyhemoglobin have a high absorption coefficient, so that the resulting PPG signal has a higher amplitude than that of the other color channels.If an NIR camera is used, recorded brightness values can be evaluated and a PPG signal can be derived therefrom.The described determination of the PPG signal as a heart activity signal can be carried out, for example, in a computing unit of the interior camera. This can be a separate arithmetic unit or an arithmetic unit integrated in a vehicle control device. For example, the vehicle control device may include one or more function blocks configured to determine one or more ROIs from a video sequence received from the interior camera and extract the PPG signal therefrom. The cardiac activity signal of the vehicle occupant can expediently be ascertained continuously. It is also possible for the heart activity signal to be ascertained in a specific period of time and / or depending on the environmental conditions of the vehicle / vehicle occupant.However, it has been found that the PPG signal recorded by means of an interior camera can be impaired to a varying extent by different breathing activities of the respective vehicle occupant. Thereby, rhythmic movements in the ROI can be triggered by respiration, which can lead to fluctuations in the detected light intensity, which are not attributable to a heart activity. In particular, a breathing rate (fundamental frequency of a periodic breathing activity signal) and its harmonics can generate disturbing frequencies which can have a negative effect on the detection of the heart activity.Since this can lead to misinterpretations with respect to the heart activity of the vehicle occupant, a breathing activity signal of the vehicle occupant is now determined by means of a radar sensor, in addition to the heart activity signal, which radar sensor can likewise be arranged in the interior of the vehicle. The radar sensor may be, for example, a continuous wave radar (FMCW) or an ultra-wideband radar (UWB). The radar sensor can detect, for example, chest movements of the vehicle occupant and determine the breathing activity signal therefrom. For example, a respiration frequency / respiration rate and a respiration depth can be deduced on the basis of a frequency and an amplitude of the chest movements. When using a UWB sensor, it is possible to use, for example, the impulse response of the transmitted signal to determine the chest movements, while an FMCW sensor can determine the chest movements, for example by means of Doppler frequency. The respiratory activity signal can be determined analogously to the cardiac activity signal of the vehicle occupant. In other words, the respiratory activity signal can be ascertained whenever the cardiac activity signal is also ascertained in order to be able to adapt / correct the latter in a suitable manner.The described determination of the breathing activity signal can be carried out, for example, in a computing unit of the radar sensor. This can be a separate arithmetic unit or an arithmetic unit integrated in a vehicle control device. For example, the vehicle control device may comprise one or more function blocks which are configured to determine a breathing activity signal from the received radar signal.The specific heart activity signal, in particular the PPG signal, is then adapted as a function of the specific breathing activity signal before one or more heart activity characteristics are determined as a function of the adapted heart activity signal. In particular, by comparing the respiratory activity signal with the cardiac activity signal, the breathing-based disruptive oscillations on the latter can be determined and removed or at least attenuated.According to one embodiment, a heart rate (heart rate) of the vehicle occupant can be determined as a heart activity characteristic. For this purpose, for example, an FFT analysis of an adapted PPG signal can be carried out. A further heart activity characteristic value can be, for example, a heart rate variability (heart rate variability) of the vehicle occupant, wherein a distance between successive regular heart beats is evaluated. The distance between two successive heartbeats can be determined on the basis of two successive amplitudes of the time-based PPG signal. Furthermore, in a known manner, an oxygen saturation and / or a blood pressure of the vehicle occupant can be determined from the PPG signal as a heart activity characteristic value.The heart activity characteristic values determined in this way can be sent, for example, to control device functions in the vehicle control device, which can draw conclusions from this, for example, about a state of health of the vehicle occupant, in particular of the driver, and can process these further or intervene in critical situations. The control device functions can comprise driver assistance functions, for example.According to an embodiment, the determined cardiac activity signal may be adjusted by filtering it such that a fundamental frequency of the respiratory activity signal and / or its harmonics are removed therefrom. For example, parameterizable filters can be used to adapt the cardiac activity signal, by means of which filters oscillation disturbingly superimposing the cardiac activity signal can be suppressed.According to one embodiment, a band elimination filter or a comb filter can be used for this purpose, which allows the interfering spectral components of the breathing activity to be suppressed. A band rejection filter can reject frequencies that are between its two cutoff frequencies and pass frequencies on either side of this range. As a result, signals in a specific frequency range can be suppressed in a targeted manner by means of a band rejection filter. A comb filter is characterized by a plurality of filter frequencies at the same distance and can therefore be used for frequency selection and for removing harmonics.According to one embodiment, one or more parameters of the band elimination filter and / or of the comb filter can be determined as a function of the determined breathing activity signal. For example, the cut-off frequencies of the band-stop filter and / or the filter frequencies of the comb filter can be determined / defined as a function of the breathing activity signal. Alternatively or additionally, attenuation and / or edge steepness of the filters can be adjusted as a function of the breathing activity signal.According to one embodiment, a power density spectrum of the respiratory activity signal can be determined in order to establish the filter parameters. For this purpose, for example, an FFT analysis (fast Fourier transform) of the respiratory activity signal can be carried out. The power density spectrum can be determined in particular continuously (e.g. at a predetermined update frequency). The power density spectrum can be determined, for example, in the computing unit of the radar sensor and / or in the vehicle control unit.According to one embodiment, one or more respiratory activity signal characteristics can be determined from the power density spectrum of the respiratory activity signal, depending on which the parameters of the band elimination filter or of the comb filter are determined / fixed. This can also be carried out, for example, in the computing unit of the radar sensor and / or in the vehicle control unit. In particular, an amplitude of the fundamental frequency of the respiratory activity signal and / or amplitudes of harmonics of the fundamental frequency can serve as one or more respiratory activity signal characteristics. In this way, for example, a damping of the band elimination filter can be adjusted depending on a breathing depth of the vehicle occupant in the corresponding frequency range, which is characterized by the respective amplitude. For this purpose, the amplitude of the fundamental frequency of the breathing activity signal and / or the amplitudes of its harmonics can be transmitted from the computing unit of the radar sensor to the computing unit of the interior camera, which determines filter parameters for adapting the heart activity signal therefrom. Alternatively, the determination of the respiratory activity signal characteristics and the setting of the filter parameters can take place in the vehicle control unit.In this way, the breathing-related disturbing oscillations on the camera-based heart activity signal can be effectively removed or at least reduced, so that one or more heart activity characteristics, such as the heart rate, can be determined correctly and with high accuracy.A computing unit according to the invention, e.g. a control device of a motor vehicle, which can preferably be a central computer or vehicle computer, is configured, in particular by programming, to carry out a method according to the invention, as described above.The implementation of a method according to the invention in the form of a computer program or computer program product with program code for carrying out all method steps is also advantageous since this causes particularly low costs, in particular if an executing control device is also used for further tasks and is therefore present in any case. Finally, a machine-readable storage medium is provided with a computer program stored thereon, as described above. Suitable storage media or data carriers for providing the computer program are, in particular, magnetic, optical and electrical memories, such as hard disks, flash memories, EEPROMs, DVDs, among others. Download of a program via computer networks (Internet, intranet, etc.) is also possible. Such a download can be effected in a wired or wired or wireless manner (e.g. via a WLAN network, a 3G, 4G, 5G or 6G connection, etc.).A vehicle having a vehicle interior, a camera, a radar sensor and a computing unit is also proposed.Further advantages and embodiments of the invention will become apparent from the description of the accompanying drawings.The invention is schematically illustrated in the drawings on the basis of exemplary embodiments and is described below with reference to the drawings. Identical elements are provided with identical reference symbols and are therefore not described a number of times, if this is not necessary.Brief Description of the DrawingsFIG. 1 shows schematically and exemplarily functional blocks of a computing unit for performing a method according to an embodiment of the invention. FIGS. 2 aand 2 bschematically show a breathing activity signal and a heart activity signal of a vehicle occupant, respectively, and an adjustment of the heart activity signal based on a breathing activity signal characteristic of the breathing activity signal according to an embodiment of the invention. FIGS. 3 aand 3 bschematically show a respiratory activity signal and a cardiac activity signal of a vehicle occupant, respectively, and an adaptation of the cardiac activity signal based on further respiratory activity signal characteristics of the respiratory activity signal according to a further embodiment of the invention.Embodiment(s) of the InventionFIG. 1 shows schematically and exemplarily functional blocks 50 to 55 of a computing unit 500 for carrying out a method according to an embodiment of the invention. In particular, the arithmetic unit 500 serves for determining an accurate heart activity signal of any vehicle occupant, for example a driver 30, from which suitable heart activity characteristics can be determined / estimated, from which it is possible to infer a state of the vehicle occupant. The cardiac activity characteristic values serve as input variables of control device functions in the form of driver assistance functions 55 in the arithmetic unit 500, so that they can intervene in the event of a critical state of the vehicle occupant. The term "control device function" is intended to include all functions which process the determined heart activity or its characteristic values further, e.g. information systems, warning systems, intervening systems, extended e, etc.The computer unit 500 shown, which can be a vehicle control unit or vehicle computer 500 in particular, comprises a camera control unit 50 having an evaluation unit 51 with an adaptation unit 52. The evaluation unit 51 is connected to one or more cameras 10 via a signal line 11, which can be a cable- or radio-based signal line 11. The camera 10 is in the present case an interior camera 10, which is mounted in a vehicle interior not shown in detail, and records a plurality of facial images 10 aof a driver 30 (here, for example, located behind a steering wheel 31) as vehicle occupants (indicated by the solid arrow between the driver 30 and the camera 10). The camera 10 can be, for example, an infrared camera and / or color camera, in particular an RGB camera, an NIR camera or a combined RGB and infrared camera.The plurality of facial images 10 acan be, for example, a video sequence with a specific number of facial images 10 aof the driver 30. The video sequence can be sent via the signal line 11 to the evaluation unit 51 of the camera control unit 50, which can determine / identify one or more face regions (region of interest, ROI) of the driver 30, for example from each facial image 10 a. Suitable ROIs can be, for example, regions of forehead and cheeks of the driver 30.From image pixels of the ROI, the evaluation unit 51 can then determine a perfusion rhythm of the driver 30 as a photoplethysmographic (PPG) signal as a heart activity signal in a spatially resolved manner. The PPG signal of the driver 30 can expediently be ascertained continuously. It is also possible for the PPG signal to be ascertained in a specific period of time and / or depending on the environmental conditions of the vehicle and of the driver 30.In order to increase the signal-to-noise ratio of the PPG signal, an average value of the pixel values in the ROI can be formed. In particular, the evaluation unit 51 can evaluate and spatially average a red, blue and green color channel of the RGB camera 10 individually. The green color channel can expediently be used for determining the PPG signal in the evaluation unit 51, since at its wavelength the blood constituents oxy- and deoxyhemoglobin have a high absorption coefficient, so that the resulting PPG signal has a higher amplitude than that of the other color channels.If an infrared camera is used, recorded brightness values can be evaluated and a PPG signal can be derived therefrom. However, due to the breathing action of the driver 30, light reflections in the ROI may generate oscillations in / on the particular PPG signal that are not due to perfusion rhythm. In particular, a fundamental frequency of the breathing activity and its harmonic can interfere with the PPG signal.Since these may lead to misinterpretations with respect to the cardiac activity of the driver 30, a respiratory activity signal of the driver 30 is determined in the arithmetic unit 500 in addition to the PPG signal. For this purpose, the computing unit 500 shown contains a radar control unit 54, which is connected via a further signal line 21 to a radar sensor 20, which is likewise arranged in the vehicle interior. The signal line 21 can likewise be a cable- or radio-based signal line 21. The radar sensor may be, for example, a frequency modulated continuous wave radar (FMCW) or an ultra-wideband radar (UWB).Radar sensor 20 detects chest movements of driver 30 (indicated by three circles of different diameters in the chest area of driver 30 and two dashed arrows between driver 30 and radar sensor 20) in order to draw conclusions about the breathing activity thereof or to ascertain a breathing activity signal of driver 30 therefrom. The radar sensor 20 can, for example, detect a frequency and an amplitude of the chest movements, from which the radar control unit 54 can calculate a breathing frequency / breathing rate and a breathing depth. When using a UWB sensor, the impulse response of the transmitted signal can be used, for example, to determine the chest movements of the driver 30, while an FMCW sensor can determine the chest movements, for example, by means of Doppler frequency. The radar control unit 54 determines the breathing signal in parallel with the driver's PPG signal acquired by the camera control unit 51 and sends it to the adjustment unit 52 of the camera control unit 51, compares the two signals and corrects the PPG signal based on the breathing signal. For example, the PPG signal can be corrected in the adaptation unit 51 by means of parameterizable filters (e.g. comb filters and / or band rejection filters). For this purpose, for example, a power density spectrum of the breathing activity signal can be determined, on the basis of which a fundamental frequency of the breathing activity and its harmonics and the respective amplitudes can be determined. This can be done, for example, either in the radar control unit 54 or in the adaptation unit 51. Based on these variables, the cut-off frequencies of the band-stop filter, for example, and its edge slope and attenuation, can be set in the adaptation unit 51. In this way, breathing-action-based oscillations can be removed from the PPG signal, as shown below in FIGS. 2a to 3b.The corrected / adjusted PPG signal is then sent from the camera control unit 51 to a calculation unit 53 of the calculation unit 500, which determines heart activity characteristics from this, such as heart rate, heart rate variability, oxygen saturation and / or blood pressure. The cardiac activity characteristic values are output by the calculation unit 53 to control device functions, such as driver assistance functions 55 of the calculation unit 500, which can then intervene in a driving activity of the driver 30 based on the obtained cardiac activity characteristic values.In contrast to the illustration shown, the described function blocks 50-55 can also be contained in a plurality of processing units / control units which are separate from one another.FIGS. 2 aand 2 bschematically show in each case a power density spectrum of a respiratory activity signal (AT) 200 and a PPG signal as a cardiac activity signal 100 of a vehicle occupant, e.g. of the driver 30, over the frequency f and an adaptation of the cardiac activity signal 100 based on at least one respiratory activity signal characteristic 201 of the respiratory activity signal 200 according to one embodiment of the invention.In particular, FIG. 2 ashows a spectrum of the breathing activity signal 200 of the driver 30 determined by the radar control unit 54 by means of the radar sensor 20, in which a fundamental frequency f 1 of the breathing activity and its harmonics f 2, f 3 can be identified (see FIG. 2 a above). Furthermore, FIG. 2 ashows a spectrum of the PPG signal 100 determined by the camera control unit 51 before its adaptation / correction by the adaptation unit 52 (see FIG. 2 aon the bottom).It is clear that the PPG signal 100 has high amplitudes at the frequencies f 1, f 2, f 3 which are attributable to the breathing activity of the driver 30. If these amplitudes are not removed from the PPG signal 100, cardiac activity characteristics of the driver 30, e.g., its heart rate, may be incorrectly calculated.Therefore, in the present exemplary embodiment according to FIG. 2 b, a breathing activity signal characteristic 201 of the breathing activity signal 200 is determined and an amplitude 201 of the fundamental frequency f 1 is determined therefrom as a breathing activity signal characteristic 201 (see FIG. 2 b above). On the basis of the amplitude 201 of the breathing activity signal 200 at the fundamental frequency f 1 it is then possible, for example, to define / determine attenuation, edge steepness and cut-off frequencies of a band elimination filter, with which the PPG signal 100 can be filtered.This results in an adjusted / corrected PPG signal 300 in which the amplitude no longer occurs at the frequency f 1 (see FIG. 2 bdownstream). As a result, the cardiac activity characteristics of the driver 30, e.g. the heart rate thereof, can be determined with higher accuracy.FIGS. 3 aand 3 bschematically show a power density spectrum of the breathing activity signal 200 and of the PPG signal 100 of a vehicle occupant, e.g. of the driver 30, over the frequency f and an adaptation of the PPG signal 100 based on further breathing activity signal characteristics 201 of the breathing activity signal 200 according to a further embodiment of the invention.In FIG. 3 b, further breathing activity signal characteristics 201 of the breathing activity signal 200 are determined; in particular, in contrast to FIG. 2 b, the amplitudes 201 of the harmonics f 2, f 3 are additionally determined as breathing activity signal characteristics 201 (see FIG. 3 b above). In this case, a comb filter can be used, for example, to eliminate the disturbing oscillations in the PPG signal 100 and its attenuation, frequency and harmonic ratio can be determined / adjusted based on the breathing activity signal characteristic 201.This results in an adjusted / corrected PPG signal 300 as a heart activity signal, in which the amplitudes no longer occur at the frequencies f 1, f 2, f 3 (see FIG. 3 bb below). As a result, the cardiac activity characteristics of the driver 30, e.g. the heart rate thereof, can be determined with an even greater accuracy.References included in the specificationThis list of documents cited by the applicant has been produced in an automated manner and is only included for the better information of the reader. The list is not part of the German patent application or utility model application. The DPMA does not take any adhesion for any faults or omissions.Patent Literature citedWO 2023 / 031339 A1

[0004]

Claims

Method for determining cardiac activity characteristics of at least one vehicle occupant (30) in a vehicle interior by means of a camera (10) and a radar sensor (20), in particular a vehicle, comprising the steps of - determining a cardiac activity signal (100) of at least one vehicle occupant (30) by means of the camera (20); - determining a breathing activity signal (200) of the vehicle occupant (30) by means of the radar sensor (20); - adjusting the determined cardiac activity signal (100) of the vehicle occupant (30) as a function of the determined breathing activity signal (200); and - determining at least one cardiac activity characteristic of the vehicle occupant (30) as a function of the adjusted cardiac activity signal (300).Method according to claim 1, wherein for adjusting the determined cardiac activity signal (100), a fundamental frequency (f 1) of the respiratory activity signal (200) and / or its harmonic (f 2, f 3) is determined.Method according to claim 2, wherein, for adapting the determined cardiac activity signal (100), it is filtered in such a way that the fundamental frequency (f 1) of the respiratory activity signal (200) and / or its harmonics (f 2, f 3) are removed from the cardiac activity signal (100).Method according to one of the preceding claims, wherein a band-stop filter and / or a comb filter is / are used for adjusting the determined cardiac activity signal (100).Method according to claim 4, wherein one or more parameters of the band elimination filter and / or the comb filter is / are determined depending on the determined breathing activity signal (200).The method of any preceding claim, wherein a power density spectrum of the respiratory activity signal (200) is determined.Method according to claims 5 and 6, wherein the parameters of the band rejection filter and / or the comb filter are determined depending on one or more respiratory activity signal characteristics (201) of the power density spectrum of the respiratory activity signal (200).The method of claim 7, wherein an amplitude (201) of the fundamental frequency (f 1) of the respiratory activity signal (200) and / or amplitudes of harmonics (f 2, f 3) of the fundamental frequency (f 1) are determined as the one or more respiratory activity signal characteristics (201).The method according to any of the preceding claims, wherein the at least one heart activity characteristic is selected from a heart rate, a heart rate variability, an oxygen saturation and a blood pressure of the vehicle occupant (30).Arithmetic unit (500) which is configured to carry out all method steps of a method according to one of the preceding claims.A computer program that causes a computing unit (500) to perform all method steps of a method according to any one of claims 1 to 9 when executed on the computing unit (500).A machine readable storage medium having stored thereon a computer program according to claim 11.Vehicle having a vehicle interior, a camera, a radar sensor and a computing unit according to Claim 10.

Images