Driver heart rate monitoring method, driving risk early warning method, medium, controller and vehicle

Abstract

The invention relates to a driver heart rate monitoring method, a driving risk early warning method, a medium, a controller and a vehicle. The method comprises the steps that a face video of a vehicle driver is collected; based on the face video, determining a face region of interest for heart rate calculation; and processing the facial region of interest through a deep learning network to obtain a heart rate value of the driver. By collecting the face video of the driver and applying the face video to follow-up determination of the heart rate value, compared with the heart rate value measured through hardware equipment connected with the driver in the related technology, direct contact with the driver can be avoided, and then the driving process of the driver can be prevented from being limited. And the face region of interest is processed through the deep learning network to obtain the heart rate value of the driver, so that the accuracy of the heart rate value can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a driver heart rate monitoring method, a driving risk warning method, a medium, a controller, and a vehicle. Background Art

[0002] With the continuous development of technology and the continuous improvement of living standards, the demand for private cars is gradually increasing, which in turn causes increasing pressure on road traffic. However, if the driver has an underlying disease or abnormal driving conditions such as fatigue, it may cause a traffic accident, which may even endanger the driver's life. Summary of the Invention

[0003] The purpose of the present disclosure is to provide a driver heart rate monitoring method, a driving risk warning method, a medium, a controller and a vehicle to solve the technical problems existing in the related technologies.

[0004] In order to achieve the above objectives, in a first aspect, the present disclosure provides a driver heart rate monitoring method, comprising: Collect facial video of the vehicle driver; determining a facial region of interest for heart rate calculation based on the facial video; The facial region of interest is processed through a deep learning network to obtain the driver's heart rate value.

[0005] Optionally, the processing the facial region of interest through a deep learning network to obtain the driver's heart rate value includes: extracting a remote photoplethysmography signal of the facial region of interest through a deep learning network; performing smoothing and denoising processing on the remote photoplethysmography signal to obtain a remote photoplethysmography sub-signal; The heart rate value is determined based on the remote photoplethysmography sub-signal.

[0006] Optionally, extracting the remote photoplethysmography signal in the facial region of interest by using a deep learning network comprises: Extracting physiological feature signals in the facial region of interest through an encoder in a deep learning network, wherein the physiological feature signals are used to represent pulsation signals around capillaries in the driver's face; The physiological characteristic signal is input into a decoder in the deep learning network to obtain the remote photoplethysmography signal.

[0007] Optionally, a multi-level feature fusion module is provided in each layer structure of the encoder, and different multi-level feature fusion modules are used to extract physiological feature signals at different depths in the facial region of interest; Each layer structure of the decoder is provided with a multi-scale enhancement module corresponding to the multi-level feature fusion module, each multi-scale enhancement module is used to connect to the corresponding multi-level feature fusion module, and each multi-scale enhancement module is used to enhance the physiological feature signal transmitted by the corresponding multi-level feature fusion module.

[0008] Optionally, it also includes: Obtaining a sampling frequency of the driver's pulse signal; Determining the heart rate value according to the remote photoplethysmography sub-signal comprises: performing a fast Fourier transform on the remote photoplethysmography sub-signal to obtain a spectrum of the remote photoplethysmography sub-signal, and determining a spectrum peak in the spectrum; Calculating the frequency of the remote photoplethysmography sub-signal in the frequency domain according to the spectrum peak, the number of transformation points in the fast Fourier transform, and the pulse signal sampling frequency; The heart rate value is calculated according to the frequency.

[0009] Optionally, determining a facial region of interest for heart rate calculation based on the facial video includes: Extracting a plurality of key points on the driver's face in the facial video, wherein each key point represents a position point of the driver's facial region, and each key point has a corresponding position number; A facial region of interest for heart rate calculation is determined according to the position numbers corresponding to the multiple key points.

[0010] Optionally, extracting a plurality of key points on the driver's face in the facial video includes: A plurality of key points on the driver's face in the facial video are extracted using a face detector MediaPipe.

[0011] Optionally, determining a facial region of interest for heart rate calculation according to the position numbers corresponding to the multiple key points includes: Determine the key point whose position number is within a first preset range as the target key point, wherein the first preset range is a preset range of position numbers corresponding to key points located in the cheek area and forehead area of the face; or Determining the key point whose position number is not within a second preset range as the target key point, wherein the second preset range is a preset range of position numbers corresponding to key points located in the eyebrow area, the eye area, and the mouth area of the face; The facial area corresponding to the target key point is determined as the facial region of interest for heart rate calculation.

[0012] Optionally, collecting the facial video of the vehicle driver includes: The driver's facial video is collected by a camera module installed above the dashboard of the vehicle.

[0013] In a second aspect, the present disclosure further provides a driving risk warning method, comprising: Determining a heart rate value of a vehicle driver, wherein the heart rate value is obtained by any one of the driver heart rate monitoring methods provided in the first aspect of the present disclosure; When the heart rate value is not within a preset heart rate threshold range, a voice warning message is played to the driver via a sound playing device in the vehicle to remind the driver that there is a driving risk; and / or, When the heart rate value is not within a preset heart rate threshold range, a light warning is issued through the interior lights of the vehicle to remind the driver that there is a driving risk, wherein the preset heart rate threshold is the driver's heart rate value during normal driving.

[0014] In a third aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods provided in the first aspect of the present disclosure and the steps of the method provided in the second aspect of the present disclosure.

[0015] In a fourth aspect, the present disclosure further provides a controller, comprising: a memory having a computer program stored thereon; A processor is used to execute the computer program in the memory to implement the steps of any one of the methods provided in the first aspect of the present disclosure and the steps of the method provided in the second aspect of the present disclosure.

[0016] In a fifth aspect, the present disclosure further provides a vehicle, comprising the controller provided in the fourth aspect of the present disclosure.

[0017] Through the above technical solution, a facial region of interest (ROI) for heart rate calculation is determined based on a captured facial video of the vehicle driver. This region of interest is then processed using a deep learning network to obtain the driver's heart rate value. By capturing the driver's facial video and subsequently determining the heart rate value, compared to related techniques that measure the heart rate value through hardware devices connected to the driver, direct contact with the driver can be avoided, thereby preventing any restrictions on the driver's driving process. Furthermore, by processing the facial region of interest using a deep learning network to obtain the driver's heart rate value, the accuracy of the heart rate value can be improved.

[0018] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following detailed description, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings: Figure 1 is a schematic diagram illustrating a method for monitoring a driver's heart rate according to an exemplary embodiment of the present disclosure.

[0020] Figure 2 FIG. 1 is a schematic diagram illustrating the arrangement of a camera module according to an exemplary embodiment of the present disclosure.

[0021] Figure 3 is a schematic diagram illustrating multiple key points according to an exemplary embodiment of the present disclosure.

[0022] Figure 4 is a schematic diagram showing frequencies according to an exemplary embodiment of the present disclosure.

[0023] Figure 5 The figure is a flowchart illustrating a method for monitoring a driver's heart rate according to an exemplary embodiment of the present disclosure.

[0024] Figure 6 is a schematic diagram illustrating a driving risk warning method according to an exemplary embodiment of the present disclosure.

[0025] Figure 7 The figure is a flowchart illustrating a driving risk warning method according to an exemplary embodiment of the present disclosure.

[0026] Figure 8 FIG. 1 is a schematic diagram showing a driving risk warning device according to an exemplary embodiment of the present disclosure.

[0027] Figure 9 FIG. 4 is a schematic diagram showing a driver heart rate monitoring device according to an exemplary embodiment of the present disclosure.

[0028] Figure 10FIG. 1 is a schematic diagram showing a driving risk warning device according to an exemplary embodiment of the present disclosure.

[0029] Figure 11 is a block diagram of a vehicle according to an exemplary embodiment. DETAILED DESCRIPTION

[0030] The following describes the specific embodiments of the present disclosure in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present disclosure and are not intended to limit the present disclosure.

[0031] Conventional vehicles do not have equipment that can monitor the driver's heart rate, making it impossible to determine whether the driver is in an abnormal driving state.

[0032] Related technologies use electrodes or pulse wave photoelectric sensors installed on the vehicle's steering wheel to collect physiological signals from the driver's palms or fingers while they grip the steering wheel, thereby obtaining the driver's heart rate. Alternatively, a heart rate transmitter is attached to the driver's chest to collect the driver's heart rate, or a watch or bracelet is worn on the driver's wrist to monitor the driver's heart rate.

[0033] However, the inventors found that when monitoring the driver's heart rate using the methods in the relevant technology, the measurements are all performed through hardware devices directly connected to the driver. However, the hardware devices directly connected to the driver may restrict the driver's driving behavior, thereby causing safety hazards to the driver during driving.

[0034] In view of this, the present disclosure provides a driver heart rate monitoring method, a driving risk warning method, a medium, a controller and a vehicle to solve the technical problems existing in the related technologies.

[0035] like Figure 1 As shown, Figure 1 is a schematic diagram showing a driver heart rate monitoring method according to an exemplary embodiment of the present disclosure, with reference to Figure 1 ,include: S101: Collecting facial video of the vehicle driver; S102: Determine a facial region of interest for heart rate calculation based on the facial video; S103: Processing the facial region of interest through a deep learning network to obtain the driver's heart rate value.

[0036] The above technical solution determines a facial region of interest (ROI) for heart rate calculation based on a captured facial video of the vehicle driver. This region of interest is then processed using a deep learning network to obtain the driver's heart rate. By capturing the driver's facial video and subsequently using it to determine the heart rate, compared to related techniques that measure heart rate using hardware devices connected to the driver, direct contact with the driver can be avoided, thereby preventing any restrictions on the driver's driving. Furthermore, obtaining the driver's heart rate by processing the facial region of interest using a deep learning network can improve the accuracy of the heart rate value.

[0037] In order to enable those skilled in the art to better understand the driver heart rate monitoring method provided by the present disclosure, the above steps are described in detail with examples below.

[0038] For example, a facial video may be a facial image of a certain length, and may be a facial image video of the driver while driving. The facial video may be captured by a camera module, and the duration of the facial video may be 5 seconds or 10 seconds, which is not specifically limited in the embodiments of the present disclosure.

[0039] In a possible embodiment, the collecting of the facial video of the vehicle driver includes: The driver's facial video is collected by a camera module installed above the dashboard of the vehicle.

[0040] It should be understood that the camera module can be a video camera, a panoramic camera, or a surveillance camera, and the embodiments of the present disclosure do not specifically limit this. The camera module can be set in front of the driver and can just capture the position of the driver's face video.

[0041] In the embodiment of the present disclosure, the camera module 1 can be set as follows Figure 2 The instrument panel is shown above. The camera module 1 is a USB (Universal Serial Bus) camera that uses the Linux kernel's native video capture API (Application Programming Interface) V4L2 for video capture. A hole and a 12V, 2A power supply are reserved above the instrument panel to power the camera. The instrument panel can be used to provide the driver with various vehicle information and control vehicle driving.

[0042] By setting up a USB cable camera to collect the driver's facial video, the cost of monitoring the driver during driving can be reduced. Compared with the related technology of directly contacting the driver and monitoring the driver's heart rate, direct contact with the driver can be avoided, and related hardware equipment can be avoided from restricting the driver's driving behavior, and the inconvenience and safety hazards caused by the driver during driving can be reduced, thereby improving the driver's safety during driving.

[0043] For example, facial regions of interest can be used to analyze specific facial areas on the driver's face for driving status and driving behavior. When determining the driver's heart rate value based on facial video, it is mainly based on non-contact pulse signal detection, and can then be determined based on the area where the capillaries in the facial video are located. The driver's face includes abundant capillaries, and the number of capillaries corresponding to different facial areas is different. For example, the cheek area and forehead area of the face contain the most dense capillaries, and the eyebrow area and eye area of the face contain the least capillaries. Therefore, the area with dense capillaries can be used as the facial region of interest, which can improve the convenience and operability of collecting facial videos, and at the same time reduce the difficulty of determining the facial region of interest. At the same time, when the facial region of interest is used for the subsequent calculation of the driver's heart rate value, the accuracy of the driver's heart rate value can be improved.

[0044] In a possible embodiment, determining a facial region of interest for heart rate calculation based on the facial video includes: Extracting a plurality of key points on the driver's face in the facial video, wherein each key point represents a position point of the driver's facial region, and each key point has a corresponding position number; A facial region of interest for heart rate calculation is determined according to the position numbers corresponding to the multiple key points.

[0045] It should be understood that when determining the facial region of interest from a facial video, the facial region of interest can be determined by extracting multiple key points on the driver's face in the facial video. Extracting multiple key points on the driver's face in the facial video can be done using a convolutional neural network or a MediaPipe face detector in a deep learning approach, and this is not specifically limited in the present embodiment.

[0046] like Figure 3As shown, multiple key points on the driver's face can be extracted based on the facial video, and each key point is correspondingly assigned a position number representing the driver's facial area, wherein different position numbers can correspond to the same facial area, and this is not specifically limited in the embodiments of the present disclosure. For example, when the key point is in the forehead area of the face, the position number corresponding to the key point can be a value such as 109, 10, 338, etc. When the key point is in the eyebrow area of the face, the position number corresponding to the key point can be a value such as 52, 53, etc. Then, the facial area of interest for calculating the heart rate can be determined based on the position number corresponding to each key point.

[0047] In a possible manner, the extracting a plurality of key points on the driver's face in the facial video includes: A plurality of key points on the driver's face in the facial video are extracted using a face detector MediaPipe.

[0048] It should be understood that multiple key points on the driver's face in the facial video are extracted through the face detector MediaPipe. When extracting multiple key points through the face detector MediaPipe, 468 key points on the face can be detected. Compared with the face detector Dlib through OpenCV in the related art, the key points extracted by this method are more numerous and more stable. And when determining the facial region of interest, the multiple key points extracted by this method can ensure that the facial region of interest extracted from adjacent frame images in the facial video will not be offset and will not contain background information. At the same time, background interference in the facial region of interest can be eliminated, thereby improving the accuracy of heart rate monitoring.

[0049] In a possible manner, determining a facial region of interest for heart rate calculation according to the position numbers corresponding to the multiple key points includes: Determine the key point whose position number is within a first preset range as the target key point, wherein the first preset range is a preset range of position numbers corresponding to key points located in the cheek area and forehead area of the face; or Determining the key point whose position number is not within a second preset range as the target key point, wherein the second preset range is a preset range of position numbers corresponding to key points located in the eyebrow area, the eye area, and the mouth area of the face; The facial area corresponding to the target key point is determined as the facial region of interest for heart rate calculation.

[0050] It should be understood that, in the driver's face, different facial areas contain different capillaries, and the position numbers of key points in different facial areas are different. Therefore, the target key point can be determined based on the position number of the key point and the first preset range, or the target key point can be determined based on the position number of the key point and the second preset range, and then the facial area corresponding to the target key point is determined as the facial area of interest.

[0051] When determining the target key point based on the position number of the key point and the first preset range, it can be determined whether the position number of each key point falls within the first preset range, and the key point whose position number falls within the first preset range is determined as the target key point. The first preset range is a pre-set position number range corresponding to the key points located in the cheek area and forehead area of the human face. The first preset range can be a multi-segment range threshold or a single range threshold. The embodiment of the present disclosure does not make specific limitations on this. For example, when the first preset range is 100-200, if the position number of the key point is 90, then the position number of the key point does not fall within the first preset range, and the key point is not a target key point; if the position number of the key point is 130, then the position number of the key point falls within the first preset range, and the key point can be determined as a target key point.

[0052] When determining the target key point based on the position number of the key point and the second preset range, it is possible to determine whether the position number of each key point is within the second preset range, and determine the key point whose position number is not within the second preset range as the target key point. The second preset range is a pre-set position number range corresponding to the key points located in the eyebrow area, eye area and mouth area of the face. The second preset range can be a multi-segment range threshold or a single range threshold. The embodiment of the present disclosure does not make specific limitations on this. For example, when the second preset range is 50-60, if the position number of the key point is 90, then the position number of the key point is not within the second preset range, and the key point is the target key point; if the position number of the key point is 55, then the position number of the key point is within the second preset range, and the key point is not the target key point.

[0053] By comparing the position number corresponding to the key point with the first preset range or the second preset range, the accuracy of the target key point can be improved. When the facial region of interest is subsequently determined based on the target key point, the accuracy and precision of the facial region of interest can be improved.

[0054] For example, after obtaining the facial region of interest, the facial region of interest can be processed using a deep learning network method to obtain the driver's heart rate value. The deep learning network method can be a neural network model method, which is not specifically limited in the embodiments of the present disclosure.

[0055] By processing the facial region of interest through a deep learning network, the driver's heart rate value is obtained, which can improve the accuracy of heart rate calculation.

[0056] In a possible manner, the processing of the facial region of interest by a deep learning network to obtain the driver's heart rate value includes: extracting a remote photoplethysmography signal of the facial region of interest through a deep learning network; performing smoothing and denoising processing on the remote photoplethysmography signal to obtain a remote photoplethysmography sub-signal; The heart rate value is determined based on the remote photoplethysmography sub-signal.

[0057] It should be understood that remote photoplethysmography (rPPG) signals can be used to characterize physiological signals of blood flow changes. Furthermore, rPPG signals can be used to observe changes in heart rate during different activities or states. In this embodiment, rPPG signals in facial regions of interest can be extracted using a deep learning network. The deep learning network can include a generative adversarial network, a U-Net (Convolutional Networks for Biomedical Image Segmentation) network, or a Transformer model (a deep learning model with a self-attention mechanism), which is not specifically limited in this disclosed embodiment.

[0058] The extracted remote photoplethysmography signal may contain some noise, necessitating filtering of the remote photoplethysmography signal. In this embodiment, the remote photoplethysmography signal can be denoised and smoothed using a digital Savitzky-Golay filter to obtain a remote photoplethysmography sub-signal. During the denoising and smoothing of the remote photoplethysmography signal using the Savitzky-Golay filter, a local polynomial in the time domain can be used to perform a least-squares fit on the data, yielding a fitting result that serves as the remote photoplethysmography sub-signal. This can be achieved by fitting a low-order polynomial to a contiguous subset of adjacent data points using a convolution operation. When the distances between the resulting fitted data points are equal, an analytical solution to the least-squares equation can be obtained using a set of "convolution coefficients" applicable to all fitted data. In other words, the method can be a moving window weighted average algorithm, and the weighting coefficient can be a polynomial least squares fit of the data within the moving window, and then the weighted average sum between the center point of the window and the adjacent points is calculated. The weighted average sum can be expressed as the following calculation formula.

[0059]

[0060] in, is the remote photoplethysmography signal before smoothing, is the smoothed remote photoplethysmography sub-signal, It is the weight factor for smoothing a moving window of length 2r+1, and the basic idea of fitting the window moving polynomial is to use the polynomial least squares of N=2r+1 spectral points in the polynomial fitting window, and fit the equally spaced data in the window to a k-order polynomial, which can be specifically expressed by the following calculation formula.

[0061]

[0062] in, is the unknown coefficient or weight, is the equally spaced data in the window.

[0063] The remote photoplethysmography signal is de-noised and smoothed using a Savitzky-Golay digital filter to generate a remote photoplethysmography sub-signal. This improves data accuracy without changing the signal trend. The driver's heart rate can then be determined based on the remote photoplethysmography sub-signal, thereby improving the accuracy and precision of the driver's heart rate.

[0064] In a possible embodiment, the extracting the remote photoplethysmography signal in the facial region of interest by using a deep learning network includes: Extracting physiological feature signals in the facial region of interest through an encoder in a deep learning network, wherein the physiological feature signals are used to represent pulsation signals around capillaries in the driver's face; The physiological characteristic signal is input into a decoder in the deep learning network to obtain the remote photoplethysmography signal.

[0065] It should be understood that in a deep learning network, it includes an encoder and a decoder, wherein the encoder can be used to extract features from the input data and convert the features into lower-dimensional data; the decoder can be used to convert the feature signal generated by the encoder into the form of Yuanshu data.

[0066] In the human face, the pulsation signals around the capillaries are more representative of the driver's heart rate. Therefore, in the disclosed embodiments, the encoder in the deep learning network can extract physiological feature signals from the facial region of interest and input these physiological feature signals into a decoder, which can then generate a remote photoplethysmography signal.

[0067] By processing the facial region of interest (ROI) using an encoder and decoder within a deep learning network, a remote PPG signal is generated. Compared to conventional PPG signal extraction methods using blind source separation (BSS) and color channel linear combination algorithms, this method avoids misdetection of noise as a remote PPG signal and avoids errors in heart rate monitoring due to a lack of prior knowledge. Compared to conventional signal processing methods such as dimensionality reduction and principal component analysis (PCA) used to extract remote PPG signals from infrared facial videos of drivers, this method improves the accuracy and precision of the remote PPG signal, thereby improving the accuracy of heart rate calculations using this signal.

[0068] In a possible manner, each layer structure of the encoder is provided with a multi-level feature fusion module, and different multi-level feature fusion modules are used to extract physiological feature signals at different depths in the facial region of interest; Each layer structure of the decoder is provided with a multi-scale enhancement module corresponding to the multi-level feature fusion module, each multi-scale enhancement module is used to connect to the corresponding multi-level feature fusion module, and each multi-scale enhancement module is used to enhance the physiological feature signal transmitted by the corresponding multi-level feature fusion module.

[0069] It should be understood that in this embodiment, each layer of the encoder is equipped with a multi-level feature fusion module, which can be used to extract physiological characteristic signals at different depths in the facial region of interest. Furthermore, the multi-level feature fusion module primarily extracts regions of the human face rich in physiological characteristic signals, such as the areas surrounding capillaries.

[0070] Each layer of the decoder is equipped with a multi-scale enhancement module, which corresponds to the multi-level feature fusion module. The multi-scale enhancement module can be connected to the corresponding multi-level feature fusion module to receive the physiological feature signals extracted by the corresponding multi-level feature fusion module. The multi-scale enhancement module can also enhance the physiological feature signals transmitted by the corresponding multi-level feature fusion module.

[0071] By adding a multi-level feature fusion module to each layer of the encoder structure and a multi-scale enhancement module to each layer of the decoder structure, and processing the facial region of interest by adding the encoder with the multi-level feature fusion module and the decoder with the multi-scale enhancement module, the accuracy and precision of the remote photoplethysmography signal can be improved.

[0072] Among the possible approaches, this also includes: Obtaining a sampling frequency of the driver's pulse signal; Determining the heart rate value according to the remote photoplethysmography sub-signal comprises: performing a fast Fourier transform on the remote photoplethysmography sub-signal to obtain a spectrum of the remote photoplethysmography sub-signal, and determining a spectrum peak in the spectrum; Calculating the frequency of the remote photoplethysmography sub-signal in the frequency domain according to the spectrum peak, the number of transformation points in the fast Fourier transform, and the pulse signal sampling frequency; The heart rate value is calculated according to the frequency.

[0073] It should be understood that the pulse signal sampling frequency can be the number of times the pulse signal is sampled per unit time. In this embodiment, the remote photoplethysmography sub-signal can be processed using a frequency domain method to obtain a heart rate value. When determining the driver's heart rate value based on the remote photoplethysmography sub-signal, the remote photoplethysmography sub-signal can be subjected to a fast Fourier transform to obtain a spectrum of the remote photoplethysmography sub-signal, and then the frequency can be analyzed to obtain a spectrum peak. The frequency of the remote photoplethysmography sub-signal is calculated based on the spectrum peak, the number of transformation points in the fast Fourier transform, and the pulse signal sampling frequency. The frequency can be expressed by the following calculation formula.

[0074]

[0075] in, is the frequency, is the spectrum peak, is the pulse signal sampling frequency, is the number of transformation points in the fast Fourier transform.

[0076] The driver's heart rate value can then be calculated based on the frequency using the following formula.

[0077]

[0078] Among them, HR is the heart rate value.

[0079] In the specific implementation process, the obtained frequency can be Figure 4 As shown, f in the figure can be The vertical axis of the upper figure can be the remote photoplethysmography signal, and the horizontal axis is the number of frames. The vertical axis of the lower figure can be the power spectrum density, and the horizontal axis can be the frequency. When the frequency is 1.21875 Hz, multiplying it by 60 can get a heart rate of 73 bpm.

[0080] Reference Figure 5 , Figure 5 FIG. 1 is a flow chart showing a method for monitoring a driver's heart rate according to an exemplary embodiment of the present disclosure. Figure 5 As shown, the process of the driver heart rate monitoring method includes the following steps.

[0081] S501: Collecting driver's facial video.

[0082] S502: Determine a facial region of interest based on the facial video.

[0083] S503: Extracting remote photoplethysmography signals from the facial region of interest.

[0084] S504: Calculate the heart rate value according to the remote photoplethysmography signal.

[0085] The specific implementation methods of the above-mentioned process steps have been described in detail above and will not be repeated here. In addition, it should be understood that for the above-mentioned system embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited to the order of actions described above. Secondly, those skilled in the art should also know that the embodiments described above are preferred embodiments, and the steps involved are not necessarily required by the present disclosure.

[0086] The above technical solution captures a driver's facial video and uses it to subsequently determine heart rate. Compared to related techniques that measure heart rate using hardware connected to the driver, this avoids direct contact with the driver and, consequently, prevents restrictions on the driver's driving. Furthermore, by processing the facial region of interest using a deep learning network to determine the driver's heart rate, the accuracy of the heart rate value can be improved.

[0087] Based on the same concept, this embodiment also discloses a driving risk warning method, referring to Figure 6 , Figure 6 is a schematic diagram illustrating a driving risk warning method according to an exemplary embodiment of the present disclosure, such as Figure 6 Shown, including: S601: Determine a heart rate value of a vehicle driver, wherein the heart rate value is obtained by the driver heart rate monitoring method disclosed in this embodiment; S602: When the heart rate value is not within a preset heart rate threshold range, playing a voice warning message to the driver via a sound playing device in the vehicle to remind the driver that there is a driving risk; and / or, When the heart rate value is not within a preset heart rate threshold range, a light warning is issued through the interior lights of the vehicle to remind the driver that there is a driving risk, wherein the preset heart rate threshold is the driver's heart rate value during normal driving.

[0088] For example, after monitoring the driver's heart rate, a real-time driving risk warning can be issued based on the heart rate. For example, if the heart rate is not within the preset heart rate threshold, it can indicate that the driver is experiencing an abnormality and needs to be reminded to take a break. This abnormality includes driving while fatigued or experiencing a sudden illness.

[0089] In this embodiment, when the heart rate value is not within the preset heart rate threshold range, a voice warning message can be played to the driver via the vehicle's audio playback device, reminding the driver that the heart rate is too high and there is a driving risk. The voice warning message can be a heart rate monitoring abnormality message, which is not specifically limited in this embodiment. In addition, the vehicle's interior lights can flash to remind the driver that the heart rate is too high and there is a driving risk.

[0090] By comparing the driver's heart rate value with a preset heart rate threshold, and warning the driver through voice alarm or light alarm when the heart rate value is not within the preset heart rate threshold, the driver's alertness can be improved, and the risk of traffic accidents caused by abnormal conditions can be reduced.

[0091] Reference Figure 7 , Figure 7 FIG. 1 is a flow chart showing a driving risk warning method according to an exemplary embodiment of the present disclosure. Figure 7 As shown, the process of the driving risk warning method includes the following steps.

[0092] S701: Determine the driver's heart rate value.

[0093] S702: When the heart rate value is not within the preset heart rate threshold range, execute step S703 or S704.

[0094] S703: Play the voice alarm information.

[0095] S704: Provide a light warning in the vehicle.

[0096] The specific implementation methods of the above-mentioned process steps have been described in detail above and will not be repeated here. In addition, it should be understood that for the above-mentioned system embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited to the order of actions described above. Secondly, those skilled in the art should also know that the embodiments described above are preferred embodiments, and the steps involved are not necessarily required by the present disclosure.

[0097] By comparing the driver's heart rate value with a preset heart rate threshold, and warning the driver through voice alarm or light alarm when the heart rate value is not within the preset heart rate threshold, the driver's alertness can be improved, and the risk of traffic accidents caused by abnormal conditions can be reduced.

[0098] This embodiment also provides a driving risk warning device, referring to Figure 8 , Figure 8 FIG. 1 is a schematic diagram illustrating a driving risk warning device according to an exemplary embodiment of the present disclosure. Figure 8 As shown, the device includes a video acquisition module 801 , an algorithm analysis module 802 , a Bluetooth transmission module 803 , a display module 804 and an early warning module 805 .

[0099] Among them, the video acquisition module 801 is used to collect facial videos of the vehicle driver, and the algorithm analysis module 802 is used to determine the facial interest region for heart rate calculation based on the facial video, and process the facial interest region through a deep learning network to obtain the heart rate value of the driver. The heart rate value is transmitted to the display module 804 and the warning module 805 through the Bluetooth module 803. The display module 804 is used to display the heart rate value, wherein the display module 804 can be a central control screen or a vehicle-mounted display screen in the vehicle, and this embodiment of the present disclosure does not make specific limitations on this. The warning module 805 is used to compare the heart rate value with a preset heart rate threshold. When the heart rate value is not within the preset heart rate threshold, an alarm is issued by playing a voice alarm message or a light alarm to remind the driver of risky driving.

[0100] The specific implementation methods of the above modules have been illustrated in detail in the driver heart rate monitoring method and driving risk warning method above, and will not be repeated here. In addition, it should be understood that for the above device embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited to the order of actions described above. Secondly, those skilled in the art should also know that the embodiments described above are preferred embodiments, and the steps involved are not necessarily required by the present disclosure.

[0101] The above technical solution captures a facial video of the driver and uses it to subsequently determine their heart rate. Compared to related techniques that measure heart rate using hardware connected to the driver, this approach avoids direct contact with the driver and, in turn, prevents restrictions on the driver's driving. Furthermore, by processing the driver's heart rate value using a deep learning network based on a facial region of interest, the accuracy of the heart rate value can be improved. Furthermore, by comparing the driver's heart rate value with a preset heart rate threshold and, if the heart rate value is not within the preset threshold, providing a warning to the driver through voice or light alarms, the driver's alertness can be enhanced and the risk of traffic accidents caused by abnormal driver conditions can be reduced.

[0102] Based on the same concept, this embodiment also discloses a driver heart rate monitoring device, referring to Figure 9 , Figure 9 is a schematic diagram showing a driver heart rate monitoring device 900 according to an exemplary embodiment of the present disclosure, as shown in FIG. Figure 9 As shown, it includes a video acquisition module 901, a first determination module 902 and a first processing module 903; The video acquisition module 901 is used to acquire the facial video of the vehicle driver; The first determining module 902 is configured to determine a facial region of interest for heart rate calculation based on the facial video; The first processing module 903 is configured to process the facial region of interest through a deep learning network to obtain the driver's heart rate value.

[0103] Optionally, the first processing module 903 includes: a signal extraction module, configured to extract a remote photoplethysmography signal of the facial region of interest through a deep learning network; a second processing module, configured to perform smoothing and denoising on the remote photoplethysmography signal to obtain a remote photoplethysmography sub-signal; The second determining module is configured to determine the heart rate value according to the remote photoplethysmography sub-signal.

[0104] Optionally, the signal extraction module includes: a first signal extraction module, configured to extract physiological characteristic signals in the facial region of interest through an encoder in a deep learning network, wherein the physiological characteristic signals are used to represent pulsation signals around capillaries in the driver's face; The second signal extraction module is used to input the physiological characteristic signal into a decoder in the deep learning network to obtain the remote photoplethysmography signal.

[0105] Optionally, a multi-level feature fusion module is provided in each layer structure of the encoder, and different multi-level feature fusion modules are used to extract physiological feature signals at different depths in the facial region of interest; Each layer structure of the decoder is provided with a multi-scale enhancement module corresponding to the multi-level feature fusion module, each multi-scale enhancement module is used to connect to the corresponding multi-level feature fusion module, and each multi-scale enhancement module is used to enhance the physiological feature signal transmitted by the corresponding multi-level feature fusion module.

[0106] Optionally, it also includes: A frequency acquisition module, used to obtain the sampling frequency of the driver's pulse signal; The second determining module includes: a Fourier transform module, configured to perform a fast Fourier transform on the remote photoplethysmography sub-signal to obtain a spectrum of the remote photoplethysmography sub-signal and determine a spectrum peak in the spectrum; a first calculation module, configured to calculate the frequency of the remote photoplethysmography sub-signal in the frequency domain according to the spectrum peak, the number of transformation points in the fast Fourier transform, and the pulse signal sampling frequency; The second calculation module is used to calculate the heart rate value according to the frequency.

[0107] Optionally, the first determining module 902 includes: a key point extraction module, configured to extract a plurality of key points on the driver's face in the facial video, wherein each key point represents a position point in the driver's facial region, and each key point has a corresponding position number; The third determining module is configured to determine a facial region of interest for heart rate calculation according to the position numbers corresponding to the multiple key points.

[0108] Optionally, the key point extraction module is used to: A plurality of key points on the driver's face in the facial video are extracted using a face detector MediaPipe.

[0109] Optionally, the third determining module is configured to: Determine the key point whose position number is within a first preset range as the target key point, wherein the first preset range is a preset range of position numbers corresponding to key points located in the cheek area and forehead area of the face; or Determining the key point whose position number is not within a second preset range as the target key point, wherein the second preset range is a preset range of position numbers corresponding to key points located in the eyebrow area, the eye area, and the mouth area of the face; The facial area corresponding to the target key point is determined as the facial region of interest for heart rate calculation.

[0110] Optionally, the video acquisition module 901 is used to: The driver's facial video is collected by a camera module installed above the dashboard of the vehicle.

[0111] Based on the same concept, this embodiment also discloses a driving risk warning device, referring to Figure 10 , Figure 10 is a schematic diagram showing a driving risk warning device 1000 according to an exemplary embodiment of the present disclosure, as shown in FIG. Figure 10 Shown, including: A fourth determining module 1001 is configured to determine a heart rate value of a vehicle driver, wherein the heart rate value is obtained by the driver heart rate monitoring device or driver heart rate monitoring method disclosed in this embodiment; The alarm module 1002 is used to play a voice alarm message to the driver through the sound playing device in the vehicle when the heart rate value is not within the preset heart rate threshold range, so as to remind the driver that there is a driving risk; and / or, when the heart rate value is not within the preset heart rate threshold range, to issue a light warning through the interior lights of the vehicle to remind the driver that there is a driving risk, wherein the preset heart rate threshold is the heart rate value of the driver during normal driving.

[0112] Based on the same concept, this embodiment also discloses a controller, including: a memory having a computer program stored thereon; A processor is used to execute the computer program in the memory to implement the steps of the driver heart rate monitoring method disclosed in this embodiment and the steps of the driving risk warning method disclosed in this embodiment.

[0113] Based on the same concept, this embodiment also discloses a vehicle, including the controller disclosed in this embodiment.

[0114] Figure 11 FIG1 is a block diagram illustrating a vehicle 1100 according to an exemplary embodiment. For example, vehicle 1100 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or another type of vehicle. Vehicle 1100 may be an autonomous vehicle or a semi-autonomous vehicle.

[0115] Reference Figure 11 Vehicle 1100 may include various subsystems, such as an infotainment system 1110, a perception system 1120, a decision-making and control system 1130, a drive system 1140, and a computing platform 1150. Vehicle 1100 may also include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and each component of vehicle 1100 may be interconnected via wired or wireless means.

[0116] In some embodiments, the infotainment system 1110 may include a communication system, an entertainment system, a navigation system, and the like.

[0117] The perception system 1120 may include several sensors for sensing information about the environment surrounding the vehicle 1100. For example, the perception system 1120 may include a global positioning system (which may be a GPS system, a BeiDou system, or another positioning system), an inertial measurement unit (IMU), a laser radar, a millimeter-wave radar, an ultrasonic radar, and a camera.

[0118] The decision control system 1130 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0119] Drive system 1140 may include components that provide power to vehicle 1100. In one embodiment, drive system 1140 may include an engine, a power source, a transmission system, and wheels. The engine may be an internal combustion engine, an electric motor, an air compression engine, or a combination thereof. The engine is capable of converting energy provided by the power source into mechanical energy.

[0120] Some or all functions of the vehicle 1100 are controlled by a computing platform 1150. The computing platform 1150 may include at least one processor 1151 and a memory 1152. The processor 1151 may execute instructions 1153 stored in the memory 1152.

[0121] The processor 1151 can be any conventional processor, such as a commercially available CPU. The processor can also include a graphics processor (GPU), a field programmable gate array (FPGA), a system on chip (SOC), an application specific integrated circuit (ASIC), or a combination thereof.

[0122] The memory 1152 may be implemented by any type of volatile or nonvolatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0123] In addition to instructions 1153 , memory 1152 may also store data, such as road maps, route information, and vehicle location, direction, speed, etc. The data stored in memory 1152 may be used by computing platform 1150 .

[0124] In an embodiment of the present disclosure, the processor 1151 may execute instruction 1153 to complete all or part of the steps of the above-mentioned driver heart rate monitoring method and driving risk warning method.

[0125] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided. When executed by a processor, the program instructions implement the steps of the above-mentioned driver heart rate monitoring method and driving risk warning method. For example, the computer-readable storage medium may be the aforementioned memory 1152 including program instructions. The above-mentioned program instructions can be executed by the processor 1151 of the vehicle 1100 to complete the above-mentioned driver heart rate monitoring method and driving risk warning method. In another exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program executable by a programmable device, and the computer program has a code portion for executing the above-mentioned driver heart rate monitoring method and driving risk warning method when executed by the programmable device.

[0126] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings. However, the present disclosure is not limited to the specific details of the above embodiments. Within the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all fall within the scope of protection of the present disclosure.

[0127] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any appropriate manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0128] In addition, the various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A driver heart rate monitoring method, characterized in that: include: Collect facial video of the vehicle driver; determining a facial region of interest for heart rate calculation based on the facial video; The facial region of interest is processed through a deep learning network to obtain the driver's heart rate value.

2. The driver heart rate monitoring method according to claim 1, characterized in that: The processing of the facial region of interest by a deep learning network to obtain the driver's heart rate value includes: extracting a remote photoplethysmography signal of the facial region of interest through a deep learning network; performing smoothing and denoising processing on the remote photoplethysmography signal to obtain a remote photoplethysmography sub-signal; The heart rate value is determined based on the remote photoplethysmography sub-signal.

3. The driver heart rate monitoring method according to claim 2, characterized in that: The extracting of the remote photoplethysmography signal in the facial region of interest by a deep learning network comprises: Extracting physiological feature signals in the facial region of interest through an encoder in a deep learning network, wherein the physiological feature signals are used to represent pulsation signals around capillaries in the driver's face; The physiological characteristic signal is input into a decoder in the deep learning network to obtain the remote photoplethysmography signal.

4. The driver heart rate monitoring method according to claim 3, characterized in that: Each layer structure of the encoder is provided with a multi-level feature fusion module, and different multi-level feature fusion modules are used to extract physiological feature signals at different depths in the facial region of interest; Each layer structure of the decoder is provided with a multi-scale enhancement module corresponding to the multi-level feature fusion module, each multi-scale enhancement module is used to connect to the corresponding multi-level feature fusion module, and each multi-scale enhancement module is used to enhance the physiological feature signal transmitted by the corresponding multi-level feature fusion module.

5. The driver heart rate monitoring method according to claim 2, characterized in that: Also includes: Obtaining a sampling frequency of the driver's pulse signal; Determining the heart rate value according to the remote photoplethysmography sub-signal comprises: performing a fast Fourier transform on the remote photoplethysmography sub-signal to obtain a spectrum of the remote photoplethysmography sub-signal, and determining a spectrum peak in the spectrum; Calculating the frequency of the remote photoplethysmography sub-signal in the frequency domain according to the spectrum peak, the number of transformation points in the fast Fourier transform, and the pulse signal sampling frequency; The heart rate value is calculated according to the frequency.

6. The driver heart rate monitoring method according to claim 1, characterized in that: The determining, based on the facial video, a facial region of interest for heart rate calculation includes: Extracting a plurality of key points on the driver's face in the facial video, wherein each key point represents a position point of the driver's facial region, and each key point has a corresponding position number; A facial region of interest for heart rate calculation is determined according to the position numbers corresponding to the multiple key points.

7. The driver heart rate monitoring method according to claim 6, characterized in that: The extracting a plurality of key points on the driver's face in the facial video includes: A plurality of key points on the driver's face in the facial video are extracted using a face detector MediaPipe.

8. The driver heart rate monitoring method according to claim 6, characterized in that: Determining a facial region of interest for heart rate calculation according to the position numbers corresponding to the multiple key points includes: Determine the key points whose position numbers are within a first preset range as target key points, wherein the first preset range is a pre-set range of position numbers corresponding to key points located in the cheek and forehead regions of the face; or Determining the key point whose position number is not within a second preset range as the target key point, wherein the second preset range is a preset range of position numbers corresponding to key points located in the eyebrow area, the eye area, and the mouth area of the face; The facial area corresponding to the target key point is determined as the facial region of interest for heart rate calculation.

9. The driver heart rate monitoring method according to any one of claims 1 to 8, characterized in that: The collecting of the vehicle driver's facial video includes: The driver's facial video is collected by a camera module installed above the dashboard of the vehicle.

10. A driving risk warning method, characterized in that: include: Determining a heart rate value of a vehicle driver, wherein the heart rate value is obtained by the driver heart rate monitoring method according to any one of claims 1 to 8; When the heart rate value is not within a preset heart rate threshold range, a voice warning message is played to the driver via a sound playing device in the vehicle to remind the driver that there is a driving risk; and / or, When the heart rate value is not within a preset heart rate threshold range, a light warning is issued through the interior lights of the vehicle to remind the driver that there is a driving risk, wherein the preset heart rate threshold is the driver's heart rate value during normal driving.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

12. A controller, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 10.

13. A vehicle, characterized in that: Including the controller according to claim 12.

Images