Visual-WiFi Signal Joint Detection-Based Driving Takeover Risk Perception Method

Abstract

A driving takeover risk state perception method based on joint detection of vision-WiFi signals extracts CSI through in-vehicle WiFi signals, extracts the composite perspective behavior characteristics of the driver through multiple cameras, fuses these two characteristics to construct a driver takeover risk state perception system, and realizes the complementary advantages of the two by fusing image and WiFi signal characteristics, thereby improving the discrimination accuracy of the driver's takeover of dangerous states.

Description

Technical Field

[0001] The present invention relates to the technical field of driving recognition, and particularly to a driving takeover risk state perception method based on joint detection of vision-WiFi signals. Background Art

[0002] With the acceleration of the urbanization process in various countries around the world and the application of science and technology in the field of transportation, motor vehicles have been rapidly popularized globally and have become an indispensable tool in the social and economic life of mankind. The progress and development of transportation have added wealth to human creation. However, with the continuous increase in the number of motor vehicles and the growing expansion of the motor vehicle driver team, traffic safety has become a severe problem faced by modern society. A variety of intelligent electronic devices have greatly increased the probability of drivers' distracted attention, easily causing potential safety hazards and leading to traffic accidents. During the use of transportation tools, the frequent occurrence of traffic accidents caused by drivers' human errors has also brought disasters to mankind. In particular, the number of accidents on highways and urban expressways has shown an increasing trend year by year, greatly threatening the lives and property of the public, causing increasing economic losses and casualties, attracting the widespread attention of government departments, automobile enterprises, and research institutions, and also becoming an important research content in the field of intelligent transportation system research.

[0003] For a driver monitoring system, two problems need to be solved: driver fatigue measurement and distraction detection. Generally, driving fatigue or drowsiness may be related to symptoms such as eye movement, facial expression, heart rate and respiratory rate, and brain activity. To detect whether a driver is drowsy, visual features such as eye movement and facial expression are very important. Yawning is also a good indicator for judging whether a driver is drowsy. Non-visual features such as heart rate variability (HRV), galvanic skin response (GSR), conductivity, steering wheel grip force, body temperature, etc. can indirectly judge the degree of driver fatigue. Electroencephalogram (EEG) and electrooculogram (EOG) provide additional psychophysiological information about drowsiness or emotional responses. Driving behavior information such as steering wheel movement, lane keeping, accelerator pedal movement, braking, etc. should also be considered in the detection of driver drowsiness.

[0004] The driver state recognition method based on driving operations mainly utilizes the parameter information of the driver's operations on the steering wheel, accelerator, and brake pedal, analyzes the driving behavior in different states, and infers whether the driver is in a dangerous driving state. However, the acquisition of driving operation signals requires the driver to operate the vehicle, and when the vehicle is driving automatically, the driver's hands and feet are often separated from the steering wheel and pedals. At this time, the driving operation behavior cannot reflect the driver's behavior state. The method based on visual information mainly discriminates the driver's behavior state by extracting features in facial and body images, and can identify various states such as driver fatigue, visual distraction, and non-driving behavior. Compared with other means, visual perception can obtain rich behavioral information of the driver's face and body in a non-invasive manner at a low cost. However, the detection technology based on image information is easily interfered by external factors such as vibration, ambient light, temperature, and dust.

[0005] With the development of wireless communication, the behavior perception technology based on WiFi signals has received increasing attention. In recent years, some researchers have conducted preliminary research on the driver behavior perception method using WiFi channel state information (CSI). However, the behavior perception technology based on WiFi CSI is vulnerable to electromagnetic signals and the behavior of others, and most of the research and designs are based on ideal situations, without considering the interference of in-vehicle electromagnetic signals and passenger behavior on signal feature extraction, which affects the further application of WiFi in the driver behavior perception scenario. Summary of the Invention

[0006] In order to overcome the above deficiencies in technology, the present invention provides a driving takeover risk state perception method based on joint detection of vision-WiFi signals, which improves the discrimination accuracy of the driver's takeover of the dangerous state.

[0007] The technical solution adopted by the present invention to overcome its technical problems is:

[0008] A driving takeover risk state perception method based on joint detection of vision-WiFi signals, comprising the following steps:

[0009] a) Install cameras at the position directly in front of the driver and on the roof of the vehicle in the upper right front. Obtain the real-time image of the driver's face through the camera directly in front, and obtain the real-time image of the driver's side through the camera on the roof of the vehicle in the upper right front;

[0010] b) Establish a multi-modal learning gated recurrent unit network model MM-GRU;

[0011] c) Install a WiFi signal transmitting device and a WiFi signal receiving device in the cab, extract WiFi-CSI information in the WiFi signal receiving device, and obtain the power delay profile SPDP of the wireless broadband signal;

[0012] d) Construct a WiFi-based driver behavior feature extraction model W-DCNN using DCNN;

[0013] e) Integrate the multi-modal learning gated recurrent unit network model MM-GRU with the WiFi-based driver behavior feature extraction model W-DCNN to obtain a multi-modal fusion network layer structure;

[0014] f) Construct a joint cost function F. By means of the cost function F, determine the optimal parameters of the driver behavior feature extraction model W-DCNN, the multi-modal learning gated recurrent unit network model MM-GRU, and the multi-modal fusion network layer structure respectively, and obtain the optimized driver behavior feature extraction model W-DCNN, the multi-modal learning gated recurrent unit network model MM-GRU, and the multi-modal fusion network layer structure. Input the real-time image of the driver's face and the real-time image of the driver's side into the multi-modal learning gated recurrent unit network model MM-GRU in the optimized multi-modal fusion network layer structure to construct a spectrogram of the SPDP curve. Input the spectrogram of the SPDP curve into the WiFi-based driver behavior feature extraction model W-DCNN in the optimized multi-modal fusion network layer structure, and construct a classifier at the top layer of the multi-modal fusion network layer structure to output the driver behavior category.

[0015] Extract CSI through in-vehicle WiFi signals, extract the driver's composite perspective behavior features through multiple cameras, fuse these two features to construct a driver takeover risk state perception system, and realize the complementary advantages of the two by fusing image and WiFi signal features, so as to improve the discrimination accuracy of the driver's takeover of dangerous states.

[0016] Further, in step b), the multi-modal learning gated recurrent unit network model MM-GRU is successively composed of a multi-modal learning network model MDNN and a gated recurrent unit GRU. Among them, the multi-modal learning network model MDNN is successively composed of a multi-modal feature input layer, two hidden layers, and a feature output layer, and the gated recurrent unit GRU is successively composed of a reset gate and an update gate.

[0017] Further, in step c), use the fast Fourier transform through the formula to calculate the power delay profile SPDP of the broadband signal. In the formula, a is the angular frequency, δ(·) is the Dirac δ function, t is the current time, t 1 is the propagation delay time of the signal, F is a coefficient, and c 0 is the full-channel CSI data.

[0018] Further, the WiFi-based driver behavior feature extraction model W-DCNN in step d) is successively composed of a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a third pooling layer, a first fully connected layer, a second fully connected layer, and a third fully connected layer. Further, in step e), the regularization term J(θ) of the fusion layer of the multi-modal fusion network layer structure is calculated according to the formula J(θ) = L(θ) + λφ(θ), where L(θ) is the loss function, λ is the regularization coefficient, and φ(θ) is the regularization function.

[0019] The first convolutional layer uses two convolutional kernels with a size of 11×11×3×48, a stride S = 4, and zero padding P = 3. The first pooling layer uses a maximum pooling operation with a size of 3×3 and a stride S = 2. The second convolutional layer uses two convolutional kernels with a size of 5×5×48×128, a stride S = 1, and zero padding P = 2. The second pooling layer uses a maximum pooling operation with a size of 3×3 and a stride S = 2. The third convolutional layer uses a convolutional kernel with a size of 3×3×256×384, a stride S = 1, and zero padding P = 2. The fourth convolutional layer uses two convolutional kernels with a size of 3×3×192×192, a stride S = 1, and zero padding P = 1. The fifth convolutional layer uses two convolutional kernels with a size of 3×3×192×128, a stride S = 1, and zero padding P = 1. The third pooling layer uses a maximum pooling operation with a size of 3×3 and a stride S = 2. The number of neurons in the first fully connected layer is 4096, the number of neurons in the second fully connected layer is 4096, and the number of neurons in the third fully connected layer is 1000.

[0020] Further, in step f), through the formula Construct the spectrogram of the SPDP curve, where n and m are both time, ω is the window function with a time length of m, R is the step size, e is the natural constant, and j is the imaginary unit.

[0021] The steps for constructing the joint cost function F in step f) are as follows:

[0022] f-1) Calculate the behavior feature cost function Fb of the multi-modal learning gated recurrent unit network model MM-GRU through the formula where x 1 is the behavior feature of the multi-modal learning gated recurrent unit network model MM-GRU, n is the total number of behavior features, y 1 is the actual value of the multi-modal learning gated recurrent unit network model MM-GRU, and a 1 is the output value of the multi-modal learning gated recurrent unit network model MM-GRU; 1

[0023] f-2) Through the formula The behavior feature cost function Fb of the WiFi-based driver behavior feature extraction model W-DCNN is calculated 2 , where x 2 is the behavior feature of the WiFi-based driver behavior feature extraction model W-DCNN, n is the total number of behavior features, and y 2 is the actual value of the WiFi-based driver behavior feature extraction model W-DCNN, and a 2 is the output value of the WiFi-based driver behavior feature extraction model W-DCNN;

[0024] f-3) The behavior feature cost function Fb of the multi-modal fusion network layer structure is calculated through the formula 0 , where x 0 is the behavior feature of the multi-modal fusion network layer structure, n is the total number of behavior features, and y 0 is the actual value of the multi-modal fusion network layer structure, and a 0 is the output value of the multi-modal fusion network layer structure;

[0025] f-4) The joint cost function F is calculated through the formula F = αFb 0 + βFb 1 + γFb 2 , where α, β, and γ are all weight coefficients.

[0026] The beneficial effects of the present invention are as follows: CSI is extracted through in-vehicle WiFi signals, and the driver's composite perspective behavior features are extracted through multiple cameras. These two features are fused to construct a driver takeover risk perception system, and the advantages of both are complemented by fusing image and WiFi signal features, improving the discrimination accuracy of the driver's takeover of dangerous states. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is the flowchart of the method of the present invention;

[0028] Figure 2 is the installation structure diagram of the WiFi signal transmitting device and receiving device of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0029] The following further describes the present invention with reference to the attached Figure 1 , attached Figure 2 .

[0030] ​A driving takeover risk state perception method based on joint detection of vision-WiFi signals, comprising the following steps: a) Install cameras at the positions in front of the driver and on the roof of the upper right side. Obtain the real-time image of the driver's face through the camera in front, and obtain the real-time image of the driver's side through the camera on the roof of the upper right side. b) Establish a multi-modal learning gated recurrent unit network model MM-GRU.

[0031] c) Install a WiFi signal transmitting device and a WiFi signal receiving device in the cab. Extract WiFi-CSI information in the WiFi signal receiving device to obtain the power delay profile SPDP of the wireless broadband signal.

[0032] d) Use DCNN to construct a WiFi-based driver behavior feature extraction model W-DCNN.

[0033] e) Integrate the multi-modal learning gated recurrent unit network model MM-GRU and the WiFi-based driver behavior feature extraction model W-DCNN to obtain a multi-modal fusion network layer structure.

[0034] f) Construct a joint cost function F. Determine the optimal parameters of the driver behavior feature extraction model W-DCNN, the multi-modal learning gated recurrent unit network model MM-GRU, and the multi-modal fusion network layer structure respectively through the cost function F, and obtain the optimized driver behavior feature extraction model W-DCNN, the multi-modal learning gated recurrent unit network model MM-GRU, and the multi-modal fusion network layer structure. Input the obtained real-time image of the driver's face and the real-time image of the driver's side into the multi-modal learning gated recurrent unit network model MM-GRU in the optimized multi-modal fusion network layer structure, construct the spectrogram of the SPDP curve, input the spectrogram of the SPDP curve into the WiFi-based driver behavior feature extraction model W-DCNN in the optimized multi-modal fusion network layer structure, and construct a classifier at the top layer of the multi-modal fusion network layer structure to output the driver behavior category.

[0035] Extract CSI through in-vehicle WiFi signals, extract the driver's composite perspective behavior features through multiple cameras, fuse these two features to construct a driver takeover risk state perception system, and realize the complementary advantages of the two by fusing image and WiFi signal features, so as to improve the discrimination accuracy of the driver's takeover of dangerous states.

[0036] Example 1:

[0037] In step b), the multi-modal learning gated recurrent unit network model MM-GRU is successively composed of a multi-modal learning network model MDNN and a gated recurrent unit GRU. Among them, the multi-modal learning network model MDNN is successively composed of a multi-modal feature input layer, two hidden layers, and a feature output layer. The gated recurrent unit GRU is successively composed of a reset gate and an update gate.

[0038] Embodiment 2:

[0039] In step c), the power delay profile SPDP of the broadband signal is calculated by using the fast Fourier transform through the formula where a is the angular frequency, δ(·) is the Dirac δ function, t is the current time, t 1 is the propagation delay time of the signal, F is a coefficient, and c 0 is the full channel CSI data.

[0040] Embodiment 3:

[0041] The WiFi-based driver behavior feature extraction model W-DCNN in step d) is successively composed of a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a third pooling layer, a first fully connected layer, a second fully connected layer, and a third fully connected layer.

[0042] Embodiment 4:

[0043] In step e), the regularization term J(θ) of the fusion layer of the multi-modal fusion network layer structure is calculated according to the formula J(θ) = L(θ) + λφ(θ), where L(θ) is the loss function, λ is the regularization coefficient, and φ(θ) is the regularization function.

[0044] Embodiment 5:

[0045] The first convolutional layer uses two convolutional kernels of size 11×11×3×48, with a stride S = 4 and zero-padding P = 3, to obtain two groups of feature maps of size 55×55×48. The first pooling layer uses a max-pooling operation of size 3×3, with a stride S = 2, to obtain two groups of feature maps of size 27×27×48. The second convolutional layer uses two convolutional kernels of size 5×5×48×128, with a stride S = 1 and zero-padding P = 2, to obtain two groups of feature maps of size 27×27×128. The second pooling layer uses a max-pooling operation of size 3×3, with a stride S = 2, to obtain two groups of feature maps of size 13×13×128. The third convolutional layer uses a convolutional kernel of size 3×3×256×384, with a stride S = 1 and zero-padding P = 2, to obtain two groups of feature maps of size 13×13×192. The fourth convolutional layer uses two convolutional kernels of size 3×3×192×192, with a stride S = 1 and zero-padding P = 1, to obtain two groups of feature maps of size 13×13×192. The fifth convolutional layer uses two convolutional kernels of size 3×3×192×128, with a stride S = 1 and zero-padding P = 1, to obtain two groups of feature maps of size 13×13×128. The third pooling layer uses a max-pooling operation of size 3×3, with a stride S = 2, to obtain two groups of feature maps of size 6×6×128. The number of neurons in the first fully connected layer is 4096, the number of neurons in the second fully connected layer is 4096, and the number of neurons in the third fully connected layer is 1000.

[0046] Example 6:

[0047] In step f), through the formula Construct the spectrogram of the SPDP curve, where n and m are both time, ω is the window function with a time length of m, R is the step size, e is the natural constant, and j is the imaginary unit.

[0048] Example 7:

[0049] The steps to construct the joint cost function F in step f) are as follows:

[0050] f-1) Calculate the behavior feature cost function Fb of the multi-modal learning gated recurrent unit network model MM-GRU through the formula where x 1 is the behavior feature of the multi-modal learning gated recurrent unit network model MM-GRU, n is the total number of behavior features, y 1 is the actual value of the multi-modal learning gated recurrent unit network model MM-GRU, and a 1 is the output value of the multi-modal learning gated recurrent unit network model MM-GRU; 1

[0051] f-2) Through the formula The behavior feature cost function Fb of the WiFi-based driver behavior feature extraction model W-DCNN is calculated 2 , where x 2 is the behavior feature of the WiFi-based driver behavior feature extraction model W-DCNN, n is the total number of behavior features, and y 2 is the actual value of the WiFi-based driver behavior feature extraction model W-DCNN, and a 2 is the output value of the WiFi-based driver behavior feature extraction model W-DCNN;

[0052] f-3) The behavior feature cost function Fb of the multi-modal fusion network layer structure is calculated through the formula 0 , where x 0 is the behavior feature of the multi-modal fusion network layer structure, n is the total number of behavior features, and y 0 is the actual value of the multi-modal fusion network layer structure, and a 0 is the output value of the multi-modal fusion network layer structure;

[0053] f-4) The combined cost function F is calculated through the formula F = αFb 0 +βFb 1 +γFb 2 , where α, β, and γ are all weight coefficients.

[0054] Finally, it should be noted that the above are only the preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.​

Claims

1. A driving takeover risk state perception method based on joint detection of visual-WiFi signals, characterized in that, it includes the following steps: a) Install cameras at the position directly in front of the driver and on the roof in the upper right. Obtain real-time images of the driver's face through the camera directly in front, and obtain real-time images of the driver's side through the camera on the roof in the upper right; b) Establish a multi-modal learning gated recurrent unit network model MM-GRU; c) Install a WiFi signal transmitting device and a WiFi signal receiving device in the cab. Extract WiFi-CSI information in the WiFi signal receiving device to obtain the power delay profile SPDP of the wireless broadband signal; d) Use DCNN to construct a WiFi-based driver behavior feature extraction model W-DCNN; e) Integrate the multi-modal learning gated recurrent unit network model MM-GRU and the WiFi-based driver behavior feature extraction model W-DCNN to obtain a multi-modal fusion network layer structure; f) Construct a joint cost function F. Respectively determine the optimal parameters of the driver behavior feature extraction model W-DCNN, the multi-modal learning gated recurrent unit network model MM-GRU, and the multi-modal fusion network layer structure through the cost function F, and obtain an optimized driver behavior feature extraction model W-DCNN, a multi-modal learning gated recurrent unit network model MM-GRU, and a multi-modal fusion network layer structure. Input the obtained real-time images of the driver's face and the real-time images of the driver's side into the multi-modal learning gated recurrent unit network model MM-GRU in the optimized multi-modal fusion network layer structure, construct a spectrogram of the SPDP curve, input the spectrogram of the SPDP curve into the WiFi-based driver behavior feature extraction model W-DCNN in the optimized multi-modal fusion network layer structure, and construct a classifier at the top layer of the multi-modal fusion network layer structure to output the driver behavior category; The steps of constructing the joint cost function F in step f) are: f-1) Calculate the behavior feature cost function Fb of the multi-modal learning gated recurrent unit network model MM-GRU through the formula 1 , where x 1 is the behavior feature of the multi-modal learning gated recurrent unit network model MM-GRU, n is the total number of behavior features, y 1 is the actual value of the multi-modal learning gated recurrent unit network model MM-GRU, and a 1 is the output value of the multi-modal learning gated recurrent unit network model MM-GRU;​ f-2) Calculate the behavior feature cost function Fb of the WiFi-based driver behavior feature extraction model W-DCNN through the formula 2 , where x 2 is the behavior feature of the WiFi-based driver behavior feature extraction model W-DCNN, n is the total number of behavior features, y 2 is the actual value of the WiFi-based driver behavior feature extraction model W-DCNN, and a 2 is the output value of the WiFi-based driver behavior feature extraction model W-DCNN;​ f-3) Calculate the cost function Fb of the behavior characteristics of the multi-modal fusion network layer structure through the formula where x 0 is the behavior characteristic of the multi-modal fusion network layer structure, n is the total number of behavior characteristics, y 0 is the actual value of the multi-modal fusion network layer structure, and a 0 is the output value of the multi-modal fusion network layer structure; 0 ​ f-4) Calculate the combined cost function F through the formula F = αFb 0 + βFb 1 + γFb 2 where α, β, and γ are all weight coefficients.

2. The driving takeover risk state perception method based on joint detection of visual-WiFi signals according to claim 1, characterized in that: In step b), the multi-modal learning gated recurrent unit network model MM-GRU is successively composed of a multi-modal learning network model MDNN and a gated recurrent unit GRU. Among them, the multi-modal learning network model MDNN is successively composed of a multi-modal feature input layer, 2 hidden layers, and a feature output layer. The gated recurrent unit GRU is successively composed of a reset gate and an update gate.

3. The driving takeover risk state perception method based on joint detection of visual-WiFi signals according to claim 1, characterized in that: In step c), the power delay profile SPDP of the wideband signal is calculated by using the fast Fourier transform through the formula where a is the angular frequency, δ(·) is the Dirac delta function, t is the current time, t 1 is the propagation delay time of the signal, F is a coefficient, c 0 is the full channel CSI data.

4. The driving takeover risk state perception method based on joint detection of visual-WiFi signals according to claim 1, characterized in that: The WiFi-based driver behavior feature extraction model W-DCNN in step d) is successively composed of a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a third pooling layer, a first fully connected layer, a second fully connected layer, and a third fully connected layer.

5. The method for driving takeover risk state perception based on visual-WiFi signal joint detection according to claim 1, characterized in that: in step e), the regularization term J(θ) of the fusion layer of the multi-modal fusion network layer structure is calculated according to the formula J(θ) = L(θ) + λφ(θ), where L(θ) is the loss function, λ is the regularization coefficient, and φ(θ) is the regularization function.

6. The method for driving takeover risk state perception based on visual-WiFi signal joint detection according to claim 4, characterized in that: The first convolutional layer uses two convolutional kernels with a size of 11×11×3×48, a stride S = 4, and zero padding P = 3. The first pooling layer uses a maximum pooling operation with a size of 3×3 and a stride S = 2. The second convolutional layer uses two convolutional kernels with a size of 5×5×48×128, a stride S = 1, and zero padding P = 2. The second pooling layer uses a maximum pooling operation with a size of 3×3 and a stride S = 2. The third convolutional layer uses a convolutional kernel with a size of 3×3×256×384, a stride S = 1, and zero padding P = 2. The fourth convolutional layer uses two convolutional kernels with a size of 3×3×192×192, a stride S = 1, and zero padding P = 1. The fifth convolutional layer uses two convolutional kernels with a size of 3×3×192×128, a stride S = 1, and zero padding P = 1. The third pooling layer uses a maximum pooling operation with a size of 3×3 and a stride S = 2. The number of neurons in the first fully connected layer is 4096, the number of neurons in the second fully connected layer is 4096, and the number of neurons in the third fully connected layer is 1000.

7. The method for driving takeover risk state perception based on visual-WiFi signal joint detection according to claim 1, characterized in that: In step f), the spectrogram of the SPDP curve is constructed through the formula where n and m are both time, ω is the window function with a time length of m, R is the step size, e is the natural constant, and j is the imaginary unit.

Images