Driver monitoring method based on high-frequency sound wave modeling

Abstract

The invention relates to driver monitoring, in particular to a driver monitoring method based on high-frequency sound wave modeling, and the method comprises the steps: transmitting high-frequency sound waves into a vehicle; when the driver is in the normal driving state, echo signals reflected in the vehicle are received, and a reference sound wave feature library about the limbs of the driver is established; disturbed echo signals in the vehicle are collected in real time, and to-be-recognized sound wave features about the limbs of the driver are extracted; inputting the to-be-recognized acoustic wave features into the trained action recognition model to obtain an action recognition result; the Doppler effect of the echo signals is analyzed, the motion direction and speed of the motion are obtained, and the motion frequency is determined; performing graded early warning by combining the action recognition result, the motion direction and speed of the action and the action frequency; according to the technical scheme provided by the invention, the defect that the action of the driver is difficult to accurately recognize in the prior art can be effectively overcome.

Description

Technical Field

[0001] The present invention relates to driver monitoring, and more particularly to a driver monitoring method based on high-frequency acoustic wave modeling. Background Art

[0002] With the rapid development of automotive intelligence and autonomous driving technologies, driver state monitoring has become a key link in enhancing driving safety. Traditional driver monitoring methods mainly rely on devices such as visual cameras, steering wheel sensors, or seat pressure sensors, but these technologies have obvious limitations:

[0003] 1) The recognition accuracy of visual cameras is significantly reduced in night, strong light, or backlight environments, and is easily interfered by obstacles such as sunglasses and hats worn by drivers. At the same time, privacy issues are also involved;

[0004] 2) Steering wheel sensors can only detect some limb movements of the driver, indirectly reflecting the driver's manipulation behavior, and cannot detect head postures and hand movements;

[0005] 3) Seat pressure sensors are not sensitive enough to small changes in the driver's movements and are difficult to reflect states such as fatigue or distraction in real time.

[0006] At the same time, traditional acoustic wave detection means are only used for distance measurement and cannot accurately identify the specific actions of the driver (such as yawning and normal speech). As a non-contact detection means, high-frequency acoustic wave technology is particularly suitable for real-time monitoring in complex driving environments due to its strong penetration ability, outstanding anti-interference ability, and insensitivity to light conditions. Summary of the Invention

[0007] (I) Technical Problems to be Solved

[0008] In view of the above-mentioned drawbacks of the prior art, the present invention provides a driver monitoring method based on high-frequency acoustic wave modeling, which can effectively overcome the defect of the prior art that it is difficult to accurately identify the actions of the driver.

[0009] (II) Technical Solutions

[0010] To achieve the above object, the present invention is realized through the following technical solutions:

[0011] A driver monitoring method based on high-frequency acoustic wave modeling includes the following steps:

[0012] S1. Transmit high-frequency acoustic waves into the vehicle;

[0013] S2. When the driver is in a normal driving state, receive the echo signal reflected in the vehicle and establish a reference acoustic wave feature library for the driver's limbs;

[0014] S3. Collect the disturbed echo signals inside the vehicle in real time, and extract the acoustic wave features to be recognized regarding the driver's limbs;

[0015] S4. Input the acoustic wave features to be recognized into the trained action recognition model to obtain the action recognition result;

[0016] S5. Analyze the Doppler effect of the echo signal, obtain the movement direction and speed of the action, and determine the action frequency;

[0017] S6. Combine the action recognition result, the movement direction and speed of the action, and the action frequency for hierarchical early warning.

[0018] Preferably, in S1, emitting high-frequency acoustic waves into the vehicle includes:

[0019] Deploy a high-frequency acoustic wave array inside the vehicle, emit high-frequency acoustic waves into the vehicle, and use the receiver array to receive the echo signals reflected inside the vehicle;

[0020] Among them, the frequency of the high-frequency acoustic waves emitted by the high-frequency acoustic wave array is 40 kHz to 120 kHz, and the transmission power of the high-frequency acoustic wave array does not exceed 1 mW / cm 2 .

[0021] Preferably, the deploying a high-frequency acoustic wave array inside the vehicle and emitting high-frequency acoustic waves into the vehicle includes:

[0022] Deploy a high-frequency acoustic wave array at the steering wheel, A-pillar, and roof positions inside the vehicle, and emit high-frequency acoustic waves into the vehicle.

[0023] Preferably, in S2, when the driver is in a normal driving state, receiving the echo signals reflected inside the vehicle and establishing a reference acoustic wave feature library regarding the driver's limbs includes:

[0024] When the driver is in a normal driving state, use the receiver array to receive the echo signals reflected inside the vehicle, and extract the reference acoustic wave features regarding the driver's limbs from the echo signals;

[0025] Establish a reference acoustic wave feature library according to the reference acoustic wave features regarding the driver's limbs extracted in different time periods;

[0026] Among them, the driver's limbs include the head and hands.

[0027] Preferably, the extracting the reference acoustic wave features regarding the driver's limbs from the echo signals includes:

[0028] Perform short-time Fourier transform on the echo signal to obtain the corresponding time-frequency diagram;

[0029] Extract time features, frequency features, energy features, time-frequency distribution features, and modulation features from the time-frequency diagram to obtain the reference acoustic wave features regarding the driver's limbs.

[0030] Preferably, performing short-time Fourier transform on the echo signal to obtain a corresponding time-frequency diagram includes:

[0031] Dividing the echo signal by using multiple time windows, and performing Fourier transform on the sub-echo signals within each time window to obtain corresponding frequency spectrum information;

[0032] Stitching together the frequency spectrum information of the sub-echo signals within each time window to obtain the time-frequency diagram of the echo signal.

[0033] Preferably, in S4, inputting the acoustic wave features to be recognized into the trained action recognition model to obtain an action recognition result, including:

[0034] Training the action recognition model to obtain a trained action recognition model, inputting the acoustic wave features to be recognized into the trained action recognition model to obtain an action recognition result;

[0035] Training the action recognition model to obtain a trained action recognition model includes:

[0036] S41. Dividing the training data set into a training set, a validation set, and a test set according to a preset ratio;

[0037] S42. Setting a loss function and an optimizer for the action recognition model;

[0038] S43. Inputting the training set into the action recognition model for model training;

[0039] S44. Calculating a loss value based on the loss function, and the optimizer updating the model parameters according to the loss value and the network gradient information;

[0040] S45. If the loss value is less than a preset threshold, the model training ends, and the current action recognition model is the trained action recognition model; otherwise, return to S43 and continue to use the training set for model training;

[0041] S46. Inputting the validation set into the trained action recognition model, evaluating the generalization ability of the model by observing the performance on the validation set, and tuning the hyperparameters and structure of the model;

[0042] S47. Inputting the test set into the tuned action recognition model for model performance evaluation;

[0043] Among them, the action recognition model is constructed based on the Resnet+LSTM model.

[0044] Preferably, in S41, dividing the training data set into a training set, a validation set, and a test set according to a preset ratio includes:

[0045] S411. Collect the echo signals disturbed inside the vehicle when the driver shows fatigue driving behavior or distracted driving behavior in multiple historical periods, and extract the corresponding historical acoustic wave features of the driver's limbs.

[0046] S412. Label the historical acoustic wave features of the driver's limbs when the driver shows fatigue driving behavior or distracted driving behavior to construct a training data set.

[0047] S413. Divide the training data set into a training set, a validation set, and a test set according to a preset ratio.

[0048] Preferably, in S5, analyze the Doppler effect of the echo signal, obtain the movement direction and speed of the action, and determine the action frequency, including:

[0049] S51. Measure the phase difference of the echo signals received by different receivers in the receiver array, and calculate the movement direction of the action by using the phase difference method according to the phase difference.

[0050] S52. Calculate the movement speed of the action based on the Doppler shift according to the movement direction of the action.

[0051] S53. Perform clustering analysis on the movement direction - movement speed data pairs of the action within a preset period, and determine the action frequency according to the clustering result.

[0052] Preferably, in S51, measure the phase difference of the echo signals received by different receivers in the receiver array, and calculate the movement direction of the action by using the phase difference method according to the phase difference, including:

[0053] Calculate the movement direction θ of the action by using the following formula:

[0054]

[0055] where is the phase difference of the echo signals received by different receivers in the receiver array, d is the spacing between the receivers receiving the echo signals in the receiver array, and λ is the wavelength of the echo signal.

[0056] In S52, calculate the movement speed of the action based on the Doppler shift according to the movement direction of the action, including:

[0057] Calculate the movement speed v of the action by using the following formula:

[0058]

[0059] where v0 is the propagation speed of the high-frequency acoustic wave, and Δf is the difference between the frequency f' of the echo signal and the frequency f of the high-frequency acoustic wave.

[0060] (III) Beneficial effects

[0061] Compared with the prior art, a driver monitoring method based on high-frequency acoustic wave modeling provided by the present invention has the following beneficial effects:

[0062] 1) Use an action recognition model to recognize the acoustic wave features to be recognized, obtain the action recognition result, analyze the Doppler effect of the echo signal, obtain the movement direction, speed and action frequency of the action, and combine the action recognition result, the movement direction, speed and action frequency of the action to accurately recognize the driver's actions and accurately distinguish similar actions;

[0063] 2) Use high-frequency acoustic waves that are inaudible to the human ear to achieve effective monitoring with zero interference. The whole process does not rely on optical acquisition devices, avoiding capturing biometric features such as faces and body appearances, and meeting the strictest privacy protection requirements;

[0064] 3) It is not affected by environmental factors such as light and can work reliably all day long. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0066] Figure 1 is a flow diagram of the present invention;

[0067] Figure 2 is a flow diagram of establishing a reference acoustic wave feature library for the driver's limbs in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0068] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0069] A driver monitoring method based on high-frequency acoustic wave modeling, as Figure 1 shown, S1. Transmit high-frequency acoustic waves into the vehicle, specifically including:

[0070] Deploy a high-frequency acoustic wave array inside the vehicle, transmit high-frequency acoustic waves into the vehicle, and use a receiver array to receive the echo signals reflected inside the vehicle;

[0071] Among them, the frequency of the high-frequency sound wave array for emitting high-frequency sound waves is 40 kHz to 120 kHz, and the transmission power of the high-frequency sound wave array does not exceed 1 mW / cm 2 .

[0072] Specifically, deploying a high-frequency sound wave array inside the vehicle and emitting high-frequency sound waves into the vehicle includes:

[0073] Deploying a high-frequency sound wave array at the steering wheel, A-pillar, and roof positions inside the vehicle and emitting high-frequency sound waves into the vehicle.

[0074] S2. When the driver is in a normal driving state, receiving the echo signal reflected inside the vehicle and establishing a reference sound wave feature library for the driver's limbs, as Figure 2 shown, specifically including:

[0075] When the driver is in a normal driving state, using the receiver array to receive the echo signal reflected inside the vehicle and extracting the reference sound wave features for the driver's limbs from the echo signal;

[0076] Establishing a reference sound wave feature library according to the reference sound wave features for the driver's limbs extracted in different time periods;

[0077] Among them, the driver's limbs include the head and hands.

[0078] Specifically, extracting the reference sound wave features for the driver's limbs from the echo signal includes:

[0079] Performing short-time Fourier transform on the echo signal to obtain the corresponding time-frequency diagram;

[0080] Extracting time features, frequency features, energy features, time-frequency distribution features, and modulation features from the time-frequency diagram to obtain the reference sound wave features for the driver's limbs.

[0081] Specifically, performing short-time Fourier transform on the echo signal to obtain the corresponding time-frequency diagram includes:

[0082] Dividing the echo signal using multiple time windows and performing Fourier transform on the sub-echo signals within each time window to obtain the corresponding frequency spectrum information;

[0083] Stitching together the frequency spectrum information of the sub-echo signals within each time window to obtain the time-frequency diagram of the echo signal.

[0084] S3. Real-time collecting the disturbed echo signal inside the vehicle and extracting the sound wave features to be recognized for the driver's limbs.

[0085] S4. Inputting the sound wave features to be recognized into the trained action recognition model to obtain the action recognition result, specifically including:

[0086] The action recognition model is trained to obtain a trained action recognition model. The acoustic wave features to be recognized are input into the trained action recognition model to obtain the action recognition result.

[0087] Specifically, training the action recognition model to obtain a trained action recognition model includes:

[0088] S41. Divide the training data set into a training set, a validation set, and a test set according to a preset ratio;

[0089] S42. Set the loss function and optimizer of the action recognition model;

[0090] S43. Input the training set into the action recognition model for model training;

[0091] S44. Calculate the loss value based on the loss function, and the optimizer updates the model parameters according to the loss value and network gradient information;

[0092] S45. If the loss value is less than the preset threshold, the model training ends, and the current action recognition model is the trained action recognition model; otherwise, return to S43 and continue to use the training set for model training;

[0093] S46. Input the validation set into the trained action recognition model, evaluate the generalization ability of the model by observing the performance on the validation set, and tune the hyperparameters and structure of the model;

[0094] S47. Input the test set into the tuned action recognition model for model performance evaluation;

[0095] Among them, the action recognition model is constructed based on the Resnet+LSTM model.

[0096] Specifically, in S41, dividing the training data set into a training set, a validation set, and a test set according to a preset ratio includes:

[0097] S411. Collect the echo signals disturbed inside the vehicle when the driver shows fatigue driving behavior or distracted driving behavior in multiple historical periods, and extract the corresponding historical acoustic wave features of the driver's limbs;

[0098] S412. Label the historical acoustic wave features of the driver's limbs when the driver shows fatigue driving behavior or distracted driving behavior to construct a training data set;

[0099] S413. Divide the training data set into a training set, a validation set, and a test set according to a preset ratio.

[0100] S5. Analyze the Doppler effect of the echo signal, obtain the movement direction and speed of the action (such as suddenly lowering the head and natural head turning), and determine the action frequency. Specifically, it includes:

[0101] S51. Measure the phase difference of the echo signals received by different receivers in the receiver array, and calculate the movement direction of the action using the phase difference method based on the phase difference;

[0102] S52. Calculate the movement speed of the action based on the Doppler frequency shift according to the movement direction of the action;

[0103] S53. Perform cluster analysis on the movement direction - movement speed data pairs of the action within a preset time period, and determine the action frequency according to the clustering result.

[0104] Specifically, in S51, measuring the phase difference of the echo signals received by different receivers in the receiver array and calculating the movement direction of the action using the phase difference method based on the phase difference includes:

[0105] Calculate the movement direction θ of the action using the following formula:

[0106]

[0107] where, is the phase difference of the echo signals received by different receivers in the receiver array, d is the distance between the receivers in the receiver array that receive the echo signals, and λ is the wavelength of the echo signal;

[0108] In S52, calculating the movement speed of the action based on the Doppler frequency shift according to the movement direction of the action includes:

[0109] Calculate the movement speed v of the action using the following formula:

[0110]

[0111] where, v0 is the propagation speed of the high - frequency sound wave, and Δf is the difference between the frequency f’ of the echo signal and the frequency f of the high - frequency sound wave.

[0112] S6. Combine the action recognition result, the movement direction and speed of the action, and the action frequency for hierarchical early warning.

[0113] For fatigue driving behaviors such as yawning, when there are time delays in the acoustic wave reflection time and spectral broadening in the acoustic wave characteristics to be recognized of the driver's limb, the action recognition model recognizes that the driver has made a yawning action. When more than 3 yawning actions are detected within 10 minutes, a first - level early warning is triggered (the dashboard warning light lights up); if accompanied by continuous lowering of the head (the movement speed v of the action < 0.1 m / s), it is upgraded to a second - level early warning (the seat vibrates).

[0114] For distracted driving behaviors such as looking down at a mobile phone or smoking, when the action recognition model recognizes that the driver's hand blocks the steering wheel reflection area and the head moves downward, it is determined that the driver is looking down at the mobile phone; when the action recognition model recognizes that the driver's hand is raised (attenuation of sound waves at a specific frequency) and shows a periodic action, it is determined that the driver is smoking. For distracted driving behaviors, the first occurrence triggers a first-level warning (voice reminder "Please focus on driving"); if it persists, it is upgraded to a second-level warning (automatically reducing the speed and turning on the hazard lights).

[0115] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A driver monitoring method based on high-frequency acoustic wave modeling, characterized in that: It includes the following steps: S1. Transmit high-frequency sound waves into the vehicle interior; S2. When the driver is in a normal driving state, receive the echo signals reflected inside the vehicle and establish a reference acoustic wave feature library for the driver's limbs; S3. Collect the disturbed echo signals inside the vehicle in real time and extract the acoustic wave features to be recognized for the driver's limbs; S4. Input the acoustic wave features to be recognized into a trained action recognition model to obtain an action recognition result; S5. Analyze the Doppler effect of the echo signals, obtain the movement direction and speed of the action, and determine the action frequency; S6. Combine the action recognition result, the movement direction and speed of the action, and the action frequency for hierarchical early warning.

2. The driver monitoring method based on high-frequency acoustic wave modeling according to claim 1, wherein: The transmitting of high-frequency sound waves into the vehicle interior in S1 includes: Deploy a high-frequency sound wave array inside the vehicle and transmit high-frequency sound waves into the vehicle interior, and use a receiver array to receive the echo signals reflected inside the vehicle; Among them, the frequency of the high-frequency acoustic wave array for emitting high-frequency acoustic waves is 40 kHz to 120 kHz, and the transmission power of the high-frequency acoustic wave array does not exceed 1 mW / cm 2 .

3. The driver monitoring method based on high-frequency acoustic wave modeling according to claim 2, wherein: The deploying of a high-frequency sound wave array inside the vehicle and transmitting high-frequency sound waves into the vehicle interior includes: Deploy a high-frequency sound wave array at the positions of the steering wheel, A-pillar, and roof inside the vehicle and transmit high-frequency sound waves into the vehicle interior.

4. The driver monitoring method based on high-frequency acoustic wave modeling according to claim 1, characterized in that: The receiving of the echo signals reflected inside the vehicle and establishing a reference acoustic wave feature library for the driver's limbs in S2 includes: When the driver is in a normal driving state, use the receiver array to receive the echo signals reflected inside the vehicle and extract the reference acoustic wave features for the driver's limbs from the echo signals; Establish a reference acoustic wave feature library according to the reference acoustic wave features for the driver's limbs extracted in different time periods; Among them, the driver's limbs include the head and hands.

5. The driver monitoring method based on high-frequency acoustic wave modeling according to claim 4, characterized in that: The extracting of the reference acoustic wave features for the driver's limbs from the echo signals includes: Perform short-time Fourier transform on the echo signals to obtain the corresponding time-frequency diagram; Extract time features, frequency features, energy features, time-frequency distribution features, and modulation features from the time-frequency diagram to obtain the reference acoustic wave features for the driver's limbs.

6. The driver monitoring method based on high-frequency acoustic wave modeling according to claim 5, characterized in that: The performing of short-time Fourier transform on the echo signals to obtain the corresponding time-frequency diagram includes: Divide the echo signals by using multiple time windows and perform Fourier transform on the sub-echo signals within each time window to obtain the corresponding spectral information; Stitch together the spectral information of the sub-echo signals within each time window to obtain the time-frequency diagram of the echo signals.

7. The driver monitoring method based on high-frequency acoustic wave modeling according to claim 1, characterized in that: The inputting of the acoustic wave features to be recognized into a trained action recognition model in S4 to obtain an action recognition result includes: Perform model training on the action recognition model to obtain a trained action recognition model, input the acoustic wave features to be recognized into the trained action recognition model to obtain an action recognition result; The performing of model training on the action recognition model to obtain a trained action recognition model includes: S41. Divide the training data set into a training set, a validation set, and a test set according to a preset ratio; S42. Set the loss function and optimizer of the action recognition model; S43. Input the training set into the action recognition model for model training; S44. Calculate the loss value based on the loss function, and the optimizer updates the model parameters according to the loss value and the network gradient information; S45. If the loss value is less than the preset threshold, the model training ends, and the current action recognition model is the trained action recognition model; otherwise, return to S43 and continue to train the model using the training set. S46. Input the validation set into the trained action recognition model, evaluate the generalization ability of the model by observing the performance on the validation set, and tune the hyperparameters and structure of the model. S47. Input the test set into the tuned action recognition model to evaluate the model performance. Among them, the action recognition model is constructed based on the Resnet+LSTM model.

8. The driver monitoring method based on high-frequency acoustic wave modeling according to claim 7, characterized in that: In S41, the training data set is divided into a training set, a validation set, and a test set according to a preset ratio, including: S411. Collect the echo signals disturbed inside the vehicle when the driver shows fatigue driving behavior or distracted driving behavior in multiple historical periods, and extract the corresponding historical acoustic wave features of the driver's limbs. S412. Label the historical acoustic wave features of the driver's limbs when the driver shows fatigue driving behavior or distracted driving behavior to construct a training data set. S413. Divide the training data set into a training set, a validation set, and a test set according to a preset ratio.

9. The driver monitoring method based on high-frequency acoustic wave modeling according to claim 1, wherein: In S5, analyze the Doppler effect of the echo signal, obtain the motion direction and speed of the action, and determine the action frequency, including: S51. Measure the phase difference of the echo signals received by different receivers in the receiver array, and calculate the motion direction of the action using the phase difference method based on the phase difference. S52. Calculate the motion speed of the action based on the Doppler shift according to the motion direction of the action. S53. Conduct cluster analysis on the motion direction-motion speed data pairs of the action within a preset period, and determine the action frequency according to the clustering result.

10. The driver monitoring method based on high-frequency acoustic wave modeling according to claim 9, characterized in that: In S51, measure the phase difference of the echo signals received by different receivers in the receiver array, and calculate the motion direction of the action using the phase difference method based on the phase difference, including: Calculate the motion direction θ of the action using the following formula: wherein, is the phase difference of the echo signals received by different receivers in the receiver array, d is the spacing between the receivers in the receiver array that receive the echo signals, and λ is the wavelength of the echo signal; In S52, calculate the motion speed of the action based on the Doppler shift according to the motion direction of the action, including: Calculate the motion speed v of the action using the following formula: Among them, v0 is the propagation speed of the high-frequency acoustic wave, and Δf is the difference between the frequency f’ of the echo signal and the frequency f of the high-frequency acoustic wave.

Images