Vehicle driving risk early warning method oriented to integrated perception of driver and vehicle state

Abstract

The application discloses a vehicle driving risk early warning method for driver and vehicle state integrated perception, and comprises the following steps: training a driving behavior recognition model by using a driving state data set to obtain a trained driving behavior recognition model; training a vehicle trajectory recognition model by using a lane deviation data set to obtain a trained vehicle trajectory recognition model; inputting collected driver images into the trained driving behavior recognition model to output a driving behavior recognition result of the driver; inputting collected vehicle driving state images into the trained vehicle trajectory recognition model to output a lane deviation recognition result; judging whether the driving behavior recognition result and / or the lane deviation recognition result meet early warning requirements, if yes, reminding the driver to correct the driving behavior and drive on the correct lane, and if not, not processing. The application can realize real-time and synchronous perception of the driving state of the driver and the driving trajectory of the vehicle, and enhances the reliability of driving early warning.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of vehicle driving early warning, in particular to a vehicle driving risk early warning method for integrated perception of driver and vehicle state. BACKGROUND

[0002] In recent years, collision accidents are still the main form of road transportation safety accidents, and the number of accidents and deaths accounts for 70.5% and 68.4% of the total number of accidents, which exposes the deficiency of road transportation vehicle collision accident prevention. Before the occurrence of vehicle collision accidents, the driver appears fatigue, distraction, etc., and the vehicle appears lane deviation and too close distance, etc. Phenomenon accounts for 40-50% of the accidents. Vehicle driving risk early warning is of great significance in improving road safety and traffic management.

[0003] Intelligent assistance is increasingly widely used in the field of vehicle driving early warning, but at present, the vehicle driving risk early warning technology has the potential danger of false alarm or missed alarm, in addition, the existing vehicle driving risk early warning technology also has the problem of being unable to effectively identify complex traffic conditions and risk factors, thereby greatly reducing the reliability of risk early warning, therefore, a vehicle driving risk early warning method for integrated perception of driver and vehicle state is needed, which can solve the above problems. SUMMARY

[0004] Therefore, the purpose of the present application is to overcome the defects in the prior art, provide a vehicle driving risk early warning method for integrated perception of driver and vehicle state, which can realize real-time synchronous perception of the driving state of the driver and the driving trajectory of the vehicle, enhance the reliability of driving early warning, and improve the level of safe driving of the vehicle.

[0005] The vehicle driving risk early warning method for integrated perception of driver and vehicle state of the present application comprises the following steps:

[0006] A driving state data set of the driver is made; the driving behavior recognition model is trained using the driving state data set to obtain a trained driving behavior recognition model;

[0007] A lane deviation data set is made; the vehicle trajectory recognition model is trained using the lane deviation data set to obtain a trained vehicle trajectory recognition model;

[0008] The collected driver image is input into the trained driving behavior recognition model, and the driving behavior recognition result of the driver is output;

[0009] The collected vehicle driving state image is input into the trained vehicle trajectory recognition model, and the lane deviation recognition result is output;

[0010] Judge whether the driving behavior recognition result and / or the lane deviation recognition result meets the warning requirement, if yes, remind the driver to correct the driving behavior and drive in the correct lane; if not, do not process.

[0011] Further, the driving behavior recognition model comprises a policy network and a two-dimensional convolutional neural network; the policy network comprises a feature extractor and a long short-term memory module; a mixed attention mechanism module is embedded in the backbone network of the two-dimensional convolutional neural network; the mixed attention mechanism module comprises a space-time excitation submodule, a channel excitation submodule and a motion excitation submodule;

[0012] The space-time excitation submodule uses single-channel three-dimensional convolution to represent space-time features;

[0013] The channel excitation submodule adaptively calibrates the characteristic response of the channel based on the mutual dependence between channels;

[0014] The motion excitation submodule calculates the time difference at the feature level, thereby stimulating the motion-sensitive channel.

[0015] Further, the policy network adaptively selects different frame scales to realize driving behavior recognition efficiency, comprising:

[0016] At time step t t Adjust to the lowest resolution and send it to the feature extractor; wherein T0 is a set time period; I t is the driving personnel state image frame at time t;

[0017] The long short-term memory module uses the extracted features and the previous state to update the hidden state and output;

[0018] Given the hidden state, estimate the policy distribution, and sample the action a t at time t; t Perform Gumbel Softmax operation;

[0019] When the action a t is less than L, adjust the frame size to a spatial resolution of 3×H at ×W at , and forward it to the corresponding backbone network to obtain frame-level prediction; wherein L is the number of state image resolution categories; H at is the height of the image at time t when the action a t is W at is the width of the image at time t when the action a t ;

[0020] When the action a t is greater than or equal to L, the backbone network will skip the current frame for prediction, and the policy network will skip the subsequent F at-L-1frames; wherein, F at-L-1 is a t video frame when L

[0021] Further, the spatio-temporal excitation sub-module uses a single-channel three-dimensional convolution to represent the spatio-temporal features, specifically including:

[0022] For a given input image X∈R N×T×C×H×W , the input tensor of each channel is averaged to obtain a global spatio-temporal tensor F∈R N×T×1×H×W relative to the channel axis; then, F is reshaped to F * ∈R N×T×1×H×W , and fed to a three-dimensional convolution layer K with a core size of 3×3×3 to obtain Finally, F is reshaped to F o ∈R N×T×1×H×W , and fed to a Sigmoid activation to obtain a spatio-temporal mask M∈R N×T×1×H×W , and finally output Y: Y=X+X⊙M.

[0023] Wherein, ⊙ represents that the spatio-temporal mask M is multiplied element by element with all channel inputs X, T represents the number of segments into which the video corresponding to the image is divided; N represents the batch number of segment T; C represents the number of image channels; H represents the height of the image; and W represents the width of the image.

[0024] Further, the channel excitation sub-module adaptively calibrates the characteristic response of the channel based on the mutual dependence between the channels, specifically including:

[0025] For a given input image X∈R N×T×C×H×W , first, the global spatial information of the input elements is obtained by averaging the input F∈R N×T×C×1×1 ; the number of channels of F is compressed by a proportion r to obtain F r =K1*F; wherein, K1 is a two-dimensional convolution layer of 1×1,

[0026] Then, F r is reshaped to until the time reasoning can be enabled, a one-dimensional convolution layer K2 with a kernel size of 3 is used to process to obtain wherein,

[0027] F is reshaped to and then decompressed by using a 1×1 two-dimensional convolution layer K3 to obtain F o =K3*F temp , fed to a Sigmoid activation to obtain a channel mask M; wherein, Fo ∈R N×T×C×1×1 and M∈R N×T×C×1×1 ;

[0028] The final output Y is: Y = X + X⊙M.

[0029] Furthermore, the motion excitation submodule calculates the time difference at the feature level to stimulate the motion-sensitive channel, specifically including:

[0030] For a given input image X∈R N×T×C×H×W Using a 1×1 two-dimensional convolutional layer, the number of channels is compressed proportionally to r, resulting in... Using a 1×1 two-dimensional convolutional layer, for F r Decompress;

[0031] Modeling motion features yields F m =K*F r [:,t+1,:,:,:]-F r [:,t,:,:,:];

[0032] Where K is a 3×3 two-dimensional convolutional layer, Among them, F r [:,t+1,:,:,:] represents the compressed feature map at time t+1, F r [:,t,:,:,:] represents the compressed feature map at time t;

[0033] Connecting motion features along the time dimension and padding the last element with zeros results in:

[0034] F m -[F m (1),...,F m [(t-1),0]; where F m (t-1) represents the (t-1)th motion;

[0035] Then for F m We perform averaging to obtain global spatial information of the input elements.

[0036] Furthermore, the vehicle trajectory recognition model includes an improved Deeplabv3+ network;

[0037] The improved Deeplabv3+ network is based on the Deeplabv3+ framework. The backbone network Xception of Deeplabv3+ is replaced with the lightweight network MobileNetv2, a channel attention mechanism module is added, and the ASPP structure in the Deeplabv3+ network is replaced with Dense-ASPP.

[0038] The channel attention mechanism module is used to focus attention on the channels of the feature map.

[0039] Furthermore, it also includes:

[0040] The vehicle distance recognition model is trained using a front vehicle distance dataset to obtain a trained vehicle distance recognition model; the collected front vehicle distance image is input into the trained vehicle distance recognition model, and the front vehicle distance recognition result is output; it is determined whether the front vehicle distance recognition result is less than the distance threshold. If so, the driver is reminded to correct the driving behavior; otherwise, no action is taken.

[0041] Furthermore, the vehicle distance recognition model includes an improved YOLOv5 network;

[0042] The improved YOLOv5 network is based on the YOLOv5 framework. It replaces the convolution operation of YOLOv5 with the Ghost Module from GhostNet and introduces the Coordinate Attention mechanism to embed positional information into the channels.

[0043] Furthermore, if the vehicle in front is directly in front of this vehicle, the distance d between them is determined using the following formula:

[0044]

[0045] Where h is the vertical distance between the camera on this vehicle and the vehicle in front; θ is the camera's pitch angle; the intersection of the camera's lens optical axis and the image plane is O(x,y), the focal length is f, the imaging point of the center point of the bottom of the vehicle in front on the image plane is D(u,v), and the angle between the straight line from the center point of the bottom of the vehicle in front to the camera and the lens optical axis is α.

[0046] If the vehicle in front is to the side and slightly in front of this vehicle, the distance D between them is determined using the following formula:

[0047]

[0048] Where γ is the yaw angle of the vehicle in front.

[0049] The beneficial effects of this invention are as follows: This invention discloses a vehicle driving risk warning method based on integrated perception of driver and vehicle status. It integrates driver distraction and fatigue detection based on in-vehicle video, as well as lane departure and distance to the vehicle in front. This integrated detection allows for real-time synchronous perception of the driver's driving status and vehicle trajectory, real-time risk assessment of the vehicle's driving status, and risk warnings. Voice reminders are also provided to the driver to correct their driving behavior and quickly return to a safe driving state, thereby improving the level of vehicle safety. Attached Figure Description

[0050] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0051] Figure 1 This is a schematic diagram of the vehicle driving risk warning process of the present invention;

[0052] Figure 2 This is a schematic diagram of video keyframe extraction according to the present invention;

[0053] Figure 3 This is a flowchart illustrating the driver distraction and fatigue driving behavior recognition process of the present invention.

[0054] Figure 4 This is a schematic diagram of the SCM module architecture of ResNet-50 according to the present invention;

[0055] Figure 5 This is a schematic diagram illustrating the working principle of the spatiotemporal excitation submodule of the present invention;

[0056] Figure 6 This is a schematic diagram illustrating the working principle of the channel excitation submodule of the present invention;

[0057] Figure 7 This is a schematic diagram illustrating the working principle of the motion excitation submodule of the present invention;

[0058] Figure 8 This is a schematic diagram of the improved Deeplabv3+ network structure of the present invention;

[0059] Figure 9 This is a schematic diagram illustrating the lane departure frame image of the present invention;

[0060] Figure 10 This is a schematic diagram of the vehicle distance measurement process of the present invention;

[0061] Figure 11 This is a schematic diagram of the improved YOLOv5 network structure of the present invention;

[0062] Figure 12 This is a schematic diagram illustrating the vehicle distance measurement principle based on pitch angle of the present invention;

[0063] Figure 13 This is a schematic diagram illustrating the vehicle distance measurement principle based on pitch angle and yaw angle of the present invention. Detailed Implementation

[0064] The present invention will be further described below with reference to the accompanying drawings, as shown in the figures:

[0065] The vehicle driving risk warning method of the present invention, which integrates driver and vehicle status perception, includes the following steps:

[0066] Create a driver's driving status dataset; use the driving status dataset to train the driving behavior recognition model to obtain a trained driving behavior recognition model;

[0067] Create a lane departure dataset; use the lane departure dataset to train the vehicle trajectory recognition model to obtain a trained vehicle trajectory recognition model;

[0068] The collected driver images are input into the trained driving behavior recognition model, and the driver's driving behavior recognition results are output.

[0069] The collected images of the vehicle's driving status are input into the trained vehicle trajectory recognition model, and the lane departure recognition result is output.

[0070] If the driving behavior recognition result and / or lane departure recognition result meet the warning requirements, the driver is reminded to correct the driving behavior and drive in the correct lane; otherwise, no action is taken.

[0071] like Figure 1 As shown, this invention acquires the real-time driving behavior sequence of the driver and the real-time driving status sequence of the vehicle through a vehicle-mounted two-way camera. Then, in order to obtain a video frame size that meets the model input requirements, the data is preprocessed by cropping, scaling and other operations and then input into a trained driving behavior recognition model for recognition. When the driving behavior recognition model detects driver fatigue or distracted driving behavior, the vehicle trajectory recognition model detects lane departure, or the distance recognition model detects that the distance to the vehicle in front is too close, it is necessary to comprehensively consider the driver's behavior and vehicle trajectory to assess the driving safety risk.

[0072] Furthermore, the identified driving behaviors and vehicle status can be continuously quantified to achieve more accurate and timely warnings. For example, if distracted driving is detected in two consecutive seconds of driver image data, or if distracted driving is detected in one consecutive second of driver image data while lane departure is detected in the vehicle's driving status image, then a driving risk is identified, and a warning is issued, initiating a voice reminder to the driver to correct their driving behavior and quickly return to a safe driving state, thereby improving driving safety.

[0073] In this embodiment, driving behavior is a continuous action. Compared to methods that rely solely on a single image for recognition, using a sequence of video frames as input allows for the identification of the driver's driving state based on the temporal, spatial, and motion dimensions of the input data, resulting in better recognition accuracy. Therefore, this invention implements driver fatigue and distracted driving behavior detection based on adaptive frame resolution.

[0074] Due to the significant redundancy from static scenes or very low frame quality (blur, low light conditions, etc.), processing every frame in a video is often unnecessary and inefficient. Therefore, while using a policy network to adaptively select the frame resolution within a unified framework, a frame-skipping mechanism is also designed to skip frames (i.e., set the resolution to zero) when needed, further improving the efficiency of action recognition. Simultaneously, since 2D convolutional neural networks (CNNs) cannot obtain long-term temporal relationships, but using 3D CNNs would face the problem of excessive computation. Therefore, after the input video is processed by the policy network, a hybrid attention mechanism module embedded in the 2D CNN backbone network is employed.

[0075] The driving behavior recognition model includes a policy network and a two-dimensional convolutional neural network. The policy network includes a feature extractor and a long short-term memory module. The backbone of the two-dimensional convolutional neural network embeds a hybrid attention mechanism module. The hybrid attention mechanism module includes a spatiotemporal activation submodule (STE), a channel activation submodule (CE), and a motion activation submodule (ME). The spatiotemporal activation submodule uses single-channel three-dimensional convolution to represent spatiotemporal features. The channel activation submodule adaptively calibrates the characteristic response of the channels based on the interdependence between channels. The motion activation submodule calculates the time difference at the feature level to stimulate the motion-sensitive channel.

[0076] This invention constructs driving state datasets for normal driving, distracted driving, and fatigued driving using publicly available datasets such as YAWDD. These datasets are divided into training, validation, and test sets in a 6:2:2 ratio to train and validate the driving behavior recognition model, which is then encapsulated and integrated into the system.

[0077] In this embodiment, the policy network adaptively selects different frame scales to improve driving behavior recognition efficiency. A series of resolutions are represented in descending order as follows: Where S0 = (H0, W0) represents the original (and highest) frame resolution, S L-1 =(H L-1 W L-1) () is the lowest resolution. (The last part, "l", appears to be a typo and can be omitted.) th The frame at time t in the scale is represented as Frame skipping is "selecting resolution S" ∞ A special case of "". Define the skip sequence (ascending order) as: i-th th This operation means skipping the current frame and subsequent frames in the prediction (F). i -1) frames. The choice of resolution and jump creates the motion space Ω.

[0078] The policy network includes a lightweight feature extractor φ(·θ). φIt also includes a Long Short-Term Memory (LSTM) module.

[0079] At time step t < T0, frame I t Adjust to the lowest resolution And send it to the feature extractor:

[0080]

[0081] Where T0 is the set time period; I t f is the driver's state image frame at time t; t It is an eigenvector, θ φ This represents the learnable parameters.

[0082] LSTM updates the hidden state h using the extracted features and the previous state. t And output ot:

[0083] [h t ,o t ] = LSTM(f t ,h t-1 ,o t-1 ,θ LSTM (2)

[0084] Given the hidden state, the policy network estimates the policy distribution and assigns actions.

[0085] a t Sampling is performed on ∈Ω={0,1,...,L+M-1} using the Gumbel Softmax operation:

[0086] a t ~GUMBEL(h t ,θ G (3)

[0087] If a t <L, adjust the frame size to a spatial resolution of 3×H at ×W at and forward it to the corresponding backbone network. To obtain frame-level prediction:

[0088]

[0089] in, It's a resizable frame. These are predicted values. L represents the number of states in the image resolution; H represents the predicted values. at For action a t The height of the image at time t; W at For action a t The width of the image at time t.

[0090] When action a t When L ≥ L, the backbone network will skip the current frame for prediction, and the policy network will skip subsequent frames. frame; For a t Video frames when L ≥ L.

[0091] Furthermore, to save computation, a shared policy network can be used to generate the lowest-resolution policy and predictions, i.e. (φ′ is an eigenvector).

[0092] In this embodiment, an SCM module is added to the backbone network to obtain more accurate prediction results. The SCM module consists of three sub-modules: the space-time excitation sub-module (STE), the channel excitation sub-module (CE), and the motion excitation sub-module (ME).

[0093] STE utilizes 3D convolution to elicit spatiotemporal information. Unlike traditional 3D convolution, this module averages all channels to obtain global spatiotemporal features, significantly reducing the computation of 3D convolution. The output of STE contains global spatiotemporal information. CE is used to activate channel correlations related to temporal information, and the output includes channel correlations based on a temporal perspective. ME demonstrates the effectiveness of inferring motion in video. ME models the differences between adjacent frames at the feature level and then combines it with the modules described above to infer the rich information preserved in the video.

[0094] In this approach, all tensors outside our SCM module are 4D, i.e., (N (batch size) × T (number of segments), C (channels), H (height), W (width)). We reshape the input 4D tensors into 5D tensors (N, T, C, H, W) before feeding them into the SCM module to enable operations on specific dimensions within the SCM module. The 5D output tensor is then reshaped back to 4D before being fed into the next 2D convolutional block. By doing so, the output of the SCM module can perceive information from spatiotemporal perspectives, channel correlations, and motion.

[0095] Figure 4 The ResNet-50-SCM module architecture is shown, where the SCM module is inserted at the beginning of each residual block. ResNet-50 provides the size of the output feature map for each layer (CLS represents the number of classes, and T represents the number of segments). First, the input video is divided into T segments on average, and then a frame is randomly sampled from the video after processing by the policy network.

[0096] STE effectively simulates spatiotemporal information using three-dimensional convolution. In this stage, STE generates a spatiotemporal mask M∈R. N×T×1×H×W, used to input X∈R to all channels N×T×C×H×W Multiply each element individually.

[0097] like Figure 5 As shown, given an image input X∈R N×T×C×H×W The input tensors of each channel are averaged to obtain the global spatiotemporal tensor F∈R relative to the channel axis. N×T×1×H×W Then, reshape F into F. * ∈R N×T×1×H×W This is then fed into a 3D convolutional layer K with a core size of 3×3×3. The formula is:

[0098]

[0099] Finally, Remodeled into F o ∈R N×T×1×H×W It is then fed into the Sigmoid activation to obtain the spatiotemporal mask M∈R. N ×T×1×H×W It can be represented as:

[0100] M=δ(F o (6)

[0101] The final output is:

[0102] Y = X + X⊙M (7)

[0103] Where ⊙ represents the element-wise multiplication of the spatiotemporal mask M with all channel inputs X.

[0104] T represents the number of video segments corresponding to the image; N represents the number of segments T in a batch; C represents the number of image channels; H represents the height of the image; and W represents the width of the image.

[0105] CE's design is similar to STE blocks, such as Figure 6 As shown.

[0106] Given an input X∈R N×T×C×H×W First, the global spatial information of the input elements is obtained by averaging the input, which can be represented as:

[0107]

[0108] Where F∈R N×T×C×1×1 Compressing the number of channels in F proportionally by r (r - channel compression ratio) can be interpreted as:

[0109] F r =K1*F (9)

[0110] Where K1 is a 1×1 two-dimensional convolutional layer,

[0111] Then reshape F r arrive Until temporal inference can be enabled. The one-dimensional convolutional layer K2 kernel size is 3 for processing. As:

[0112]

[0113] in Then Reconstruction Then, it is decompressed using a 1×1 2D convolutional layer K3 and fed into a Sigmoid activation. These are the final two steps to obtain the channel mask M, which can be formulated separately:

[0114] F o =K3*F temp (11)

[0115] M=δ(F o (12)

[0116] Where F o ∈R N×T×C×1×1 and M∈R N×T×C×1×1 Finally, the output of CE is formulated using the newly generated mask to be the same as that in equation (7).

[0117] ME can be used in parallel with the two modules STE and CE mentioned above, such as Figure 7 As shown, motion information is modeled from adjacent frames.

[0118] Using the same compression and decompression strategy as the CE submodule, two 1×1 two-dimensional convolutional layers are employed, as shown in equations (9) and (11) respectively. Given the features after compression... Modeling motion features based on similar operations yields the following representation:

[0119] F m =K*F r [:,t+1,:,:,:]-F r [:,t,:,:,:] (13)

[0120] Where K is a 3×3 two-dimensional convolutional layer, F r [:,t+1,:,:,:] represents the compressed feature map at time t+1, F r [:,t,:,:,:] represents the feature map at time t;

[0121] Connecting motion features along the time dimension and padding with zeros to the last element yields F. m -[Fm (1),...,F m [(t-1),0], Among them, F m (t-1) represents the (t-1)th motion.

[0122] Then it is processed by the same spatial averaging and merging as in formula (8), that is, for F m We perform averaging to obtain global spatial information of the input elements.

[0123] In this embodiment, the vehicle trajectory recognition model includes an improved Deeplabv3+ network;

[0124] The improved Deeplabv3+ network is based on the Deeplabv3+ framework. The backbone network Xception of Deeplabv3+ is replaced with the lightweight network MobileNetv2, a channel attention mechanism module is added, and the ASPP structure in the Deeplabv3+ network is replaced with Dense-ASPP.

[0125] The channel attention mechanism module is used to focus attention on the channels of the feature map.

[0126] First, the video captured by the dashcam is segmented and cropped. The cropped images are then combined with public datasets such as Tusimple and CULane to form the lane offset dataset used for lane detection. Second, these datasets are preprocessed to enhance the illumination of darker images, and then manually labeled to inform the model of deeper lane line features. Finally, the labeled dataset is input into the vehicle trajectory recognition model for training. The training progress is observed using a loss function. If the training effect is unsatisfactory, the obtained parameters are backpropagated to the network for continued training until an ideal model for lane detection is trained.

[0127] This invention, based on the Deeplabv3+ framework, establishes a lightweight network model for more efficient feature extraction. Addressing the issue of incomplete feature extraction in the original network, a denser structure is added to improve this. Furthermore, an attention mechanism is incorporated to enhance focus on important features, thereby improving model training and real-time detection accuracy.

[0128] like Figure 8As shown, the Deeplabv3+ backbone network Xception is replaced with the lightweight network MobileNetv2, which has fewer parameters and is faster, making it very suitable for real-time detection scenarios. To improve the performance of the improved model, this invention adds an SE (Squeeze-and-Excitation) module (channel attention mechanism module). The SE module mainly focuses attention on the channels of the feature map, automatically learning the importance of different channels.

[0129] To address the problem that increased inflation rate makes it more difficult to obtain local features, Dense Atrous Spatial Pyramid Pooling (Dense-ASPP) was introduced, replacing the original ASPP structure of the network.

[0130] Based on the lane line detection described above, such as Figure 9 As shown, assuming the pixel coordinates of the front of the car in the image are (180,0), D L D R Represents the distance from the center of the vehicle to the left and right lane lines, (L) x ,L y ), (R x ,R y The numbers () represent the pixel coordinates of the left and right lane lines in the image. The distance D from the vehicle's center to the left lane line is then calculated. L =180-L x Similarly, D R =R x -180.

[0131] Normally, acquiring 20 frames of images takes 0.5 seconds. At this time, it can be done via D... L D R The system determines whether the vehicle has deviated by analyzing continuous changes within 20 frames. An array L can store 20 values ​​of D. L The array R can store 20 values ​​of change. R The change value.

[0132] Furthermore, the lateral velocity V at this time... L V R See equations (14) and (15):

[0133]

[0134]

[0135] This embodiment also includes:

[0136] The vehicle distance recognition model is trained using a front vehicle distance dataset to obtain a trained vehicle distance recognition model; the collected front vehicle distance image is input into the trained vehicle distance recognition model, and the front vehicle distance recognition result is output; it is determined whether the front vehicle distance recognition result is less than the distance threshold. If so, the driver is reminded to correct the driving behavior; otherwise, no action is taken.

[0137] This invention selects monocular vision for vehicle distance recognition and analysis, mainly based on a monocular vision ranging model combined with the detected target bounding boxes to complete the distance measurement. Monocular vision ranging only requires one camera and some coordinate system transformations to complete the corresponding distance measurement, which has the advantages of low computational load and low power consumption.

[0138] This invention uses a dashcam in a vehicle as the main device for data collection, and collects real-time data on the distance to the vehicle in front in various scenarios such as busy urban areas, highways, and roads.

[0139] like Figure 11 As shown, the vehicle distance recognition model is based on YOLOv5, with corresponding modifications to the backbone network and the integration of an attention mechanism to meet the requirements of lightweight, high precision, multi-scale vehicle detection, vehicle detection under complex environmental conditions, and real-time performance.

[0140] In the YOLOv5 backbone network, its CSP mechanism uses a large number of convolutional (Conv) operations. The computational cost of convolution increases with the number of layers. To address this issue, this invention replaces the convolutional operations in YOLOv5 with the Ghost Module from GhostNet, improving model speed and reducing the number of related parameters without affecting accuracy. The Coordinate Attention (CA) mechanism is introduced to embed positional information into the channels, overcoming the shortcomings of some mechanisms that focus on channel attention and ignore positional information.

[0141] After detecting vehicles in front of the vehicle, the pixel values ​​of the corresponding targets are extracted. Distance modeling is then performed based on the camera pitch angle and the yaw angle of the vehicle in front. Figure 12 As shown.

[0142] In the diagram, the dashed line at the bottom left corner represents the lens optical axis, intersecting the image plane at point O(x,y) (image plane coordinate system), with a focal length of f. The dashed line at the top left corner represents the straight-line distance from the detected center point of the vehicle's bottom to the camera, with its image point on the image plane at point D(u,v) (pixel coordinate system), and an angle α with the optical axis. If the camera is positioned on the vehicle itself, then the horizontal distance from the camera to the vehicle, which is also the distance between the vehicle and the vehicle in front, is:

[0143]

[0144] in, h is the vertical distance between the camera and the vehicle in front, and θ is the camera pitch angle (the angle between the camera lens optical axis and the horizontal plane).

[0145] In real-world scenarios, the vehicle in front may not be directly in front of the vehicle; it could be to the left or right. Therefore, the vehicle in front is positioned to the side and slightly in front of the vehicle. See the specific implementation details below. Figure 13 As shown.

[0146] β is the horizontal angle between the external optical axes of the camera and γ is the yaw angle of the vehicle ahead. B′(B x B y Let B be the position of the landing point B of the bottom center yaw trajectory of the vehicle in front in the pixel coordinate system, with the pixel center point being O′(u,v). The ranging based on the camera pitch angle and the yaw angle of the vehicle in front is as follows:

[0147]

[0148] In equation (17), D is the distance between the vehicle and the vehicle in front, and θ and γ represent the camera pitch angle and the yaw angle of the vehicle in front, respectively.

[0149] Since pitch and yaw angles are difficult to obtain, a method for real-time acquisition of changing angles can be designed: Lane lines are generally parallel; if the lane lines are photographed, they will eventually intersect at a single point, called the vanishing point. The vanishing point is then used to calculate the camera's yaw and pitch angles.

[0150] First, a Gabor filter is used to calculate the texture direction of each pixel in the captured image. Then, based on the confidence level of these pixels, it is determined whether voting is needed. Finally, a fast local voting method is used to determine the location of the vanishing point.

[0151] The functional expression for the Gabor filter is as follows:

[0152]

[0153] In the formula, ω, Indicates scale and direction,

[0154] x and y represent pixel coordinates.

[0155] Since Gabor filtering yields 36 directional textures for each pixel, it cannot guarantee that the texture in each direction is desirable. Therefore, a confidence level is introduced. When this parameter exceeds a set threshold, the texture in that direction is considered a "voting point." The confidence level of a pixel value d(x,y) at a given point in the image can be defined by the following formula:

[0156]

[0157] Where, r i (d) represents the response value in the i-th direction at this pixel, r5(d) to r 15 (d) represents the local maximum response values ​​that occur between these values, while the other parameters rarely appear. The threshold is set as follows:

[0158] t=0.4(maxC(d)-minC(d)) (20)

[0159] When the confidence level is greater than t, the pixel is considered a voting point. The final road vanishing point is selected based on the voting points and the confidence level.

[0160] The voting algorithm works by selecting a candidate vanishing point H, and then using H as the center of a circle with a radius of one-third of the image size as the candidate region. The specific voting formula is as follows:

[0161]

[0162] In the formula, α is the angle between the texture direction of a voting point P and the line segment PH, and d(P,H) represents the distance from point P to the center of the circle. This formula takes each pixel in the image as a candidate point, and finally, the point with the highest vote score is the vanishing point.

[0163] After obtaining the vanishing point, the pitch and yaw angles are respectively:

[0164]

[0165] Where R is the camera image rotation matrix; R xz Represents the matrix of rotation about the x-axis and z-axis; R yz This represents a matrix that rotates about the y-axis and z-axis.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for early warning of vehicle driving risks based on integrated perception of driver and vehicle status, characterized in that: include: Create a dataset of drivers' driving status; The driving behavior recognition model is trained using a driving status dataset to obtain a trained driving behavior recognition model. The driving behavior recognition model includes a policy network and a two-dimensional convolutional neural network; the policy network includes a feature extractor and a long short-term memory module; the backbone of the two-dimensional convolutional neural network is embedded with a hybrid attention mechanism module; the hybrid attention mechanism module includes a spatiotemporal activation submodule, a channel activation submodule, and a motion activation submodule. The spatiotemporal excitation submodule uses single-channel three-dimensional convolution to represent spatiotemporal features; The channel excitation submodule adaptively calibrates the characteristic response of the channels based on the interdependence between channels; The motion excitation submodule calculates the time difference at the feature level, thereby stimulating the motion-sensitive channel; Create a lane departure dataset; The vehicle trajectory recognition model is trained using the lane departure dataset to obtain a trained vehicle trajectory recognition model. The collected driver images are input into the trained driving behavior recognition model, and the driver's driving behavior recognition results are output. The collected images of the vehicle's driving status are input into the trained vehicle trajectory recognition model, and the lane departure recognition result is output. Determine whether the driving behavior recognition results and / or lane departure recognition results meet the warning requirements. If so, remind the driver to correct the driving behavior and drive in the correct lane. If not, no action will be taken.

2. The vehicle driving risk early warning method based on integrated perception of driver and vehicle status according to claim 1, characterized in that: The policy network adaptively selects different frame scales to improve driving behavior recognition efficiency, including: At time step t < T0, frame I t Adjust to the lowest resolution and send it to the feature extractor; where T0 is the set time period; I t The image frame showing the driver's state at time t; The Long Short-Term Memory (LSTM) module updates the hidden state using the extracted features and the previous state and then outputs the result. Given the hidden state, estimate the policy distribution for action a at time t. t Sampling is performed, followed by a GumbelSoftmax operation; When action a t <L, adjust the frame size to a spatial resolution of 3×H at ×W at It is then forwarded to the corresponding backbone network to obtain frame-level predictions; where L is the number of state image resolution categories; H at For action a t The height of the image at time t; W at For action a t Width of the image at time t; When action a t When L ≥ L, the backbone network will skip the current frame for prediction, and the policy network will skip subsequent frames. Frame; where, For a t Video frames when L ≥ L.

3. The vehicle driving risk early warning method based on integrated perception of driver and vehicle status according to claim 1, characterized in that: The spatiotemporal excitation submodule uses single-channel 3D convolution to represent spatiotemporal features, specifically including: For a given input image X∈R N×T×C×H×W The input tensors of each channel are averaged to obtain the global spatiotemporal tensor F∈R relative to the channel axis. N×T×1×H×W Then, reshape F into F. * ∈R N×T×1×H×W It is then fed into a 3D convolutional layer K with a core size of 3×3×3 to obtain... Finally, Remodeled into F o ∈R N×T×1×H×W It is then fed into the Sigmoid activation to obtain the spatiotemporal mask M∈R. N×T×1×H×W The final output is Y: Y = X + X⊙M; Where ⊙ represents the element-wise multiplication of the spatiotemporal mask M with all channel inputs X; T represents the number of segments into which the video corresponding to the image is divided; N represents the number of batches of segments T; C represents the number of image channels; H represents the height of the image; and W represents the width of the image.

4. The vehicle driving risk warning method based on integrated perception of driver and vehicle status according to claim 1, characterized in that: The channel excitation submodule adaptively calibrates the characteristic response of the channels based on the interdependence between channels, specifically including: For a given input image X∈R N×T×C×H×W First, the global spatial information F∈R of the input elements is obtained by averaging the input. N×T×C×1×1 Compress the number of channels in F by a ratio r to obtain F. r =K1*F; where K1 is a 1×1 two-dimensional convolutional layer. Then, reshape F r arrive Until temporal inference can be enabled, the one-dimensional convolutional layer K2 kernel size is 3 for processing. get in, Will Reconstruction Then, it is decompressed using a 1×1 two-dimensional convolutional layer K3 to obtain F. o =K3*F temp The signal is fed into the Sigmoid activation layer to obtain the channel mask M; where F o ∈R N×T×C×1×1 and M∈R N ×T×C×1×1 ; The final output Y is: Y = X + X⊙M.

5. The vehicle driving risk warning method based on integrated perception of driver and vehicle status according to claim 1, characterized in that: The motion excitation submodule calculates the time difference at the feature level to stimulate the motion-sensitive channel, specifically including: For a given input image X∈R N×T×C×H×W Using a 1×1 two-dimensional convolutional layer, the number of channels is compressed proportionally to r, resulting in... Using a 1×1 two-dimensional convolutional layer, for F r Decompress; Modeling motion features yields F m =K*F r [:,t+1,:,:,:]-F r [:,t,:,:,:]; Where K is a 3×3 two-dimensional convolutional layer, Among them, F r [:,t+1,:,:,:] represents the compressed feature map at time t+1, F r [:,t,:,:,:] represents the compressed feature map at time t; Connecting motion features along the time dimension and padding the last element with zeros results in: F m -[F m (1),...,F m [(t-1),0]; where F m (t-1) represents the (t-1)th motion; Then for F m We perform averaging to obtain global spatial information of the input elements.

6. The vehicle driving risk early warning method based on integrated perception of driver and vehicle status according to claim 1, characterized in that: The vehicle trajectory recognition model includes an improved Deeplabv3+ network; The improved Deeplabv3+ network is based on the Deeplabv3+ framework. The backbone network Xception of Deeplabv3+ is replaced with the lightweight network MobileNetv2, a channel attention mechanism module is added, and the ASPP structure in the Deeplabv3+ network is replaced with Dense-ASPP. The channel attention mechanism module is used to focus attention on the channels of the feature map.

7. The vehicle driving risk warning method based on integrated perception of driver and vehicle status according to claim 1, characterized in that: Also includes: The vehicle distance recognition model is trained using the front vehicle distance dataset to obtain a trained vehicle distance recognition model; The collected image of the distance to the vehicle in front is input into the trained vehicle distance recognition model, and the vehicle distance recognition result is output. It is determined whether the vehicle distance recognition result is less than the distance threshold. If so, the driver is reminded to correct the driving behavior; otherwise, no action is taken.

8. The vehicle driving risk warning method based on integrated perception of driver and vehicle status according to claim 7, characterized in that: The vehicle distance recognition model includes an improved YOLOv5 network; The improved YOLOv5 network is based on the YOLOv5 framework. It replaces the convolution operation of YOLOv5 with the Ghost Module from GhostNet and introduces the Coordinate Attention mechanism to embed positional information into the channels.

9. The vehicle driving risk warning method based on integrated perception of driver and vehicle status according to claim 7, characterized in that: If the vehicle in front is directly in front of your vehicle, then the distance d between the two vehicles is determined using the following formula: Where h is the vertical distance between the camera on this vehicle and the vehicle in front; θ is the camera's pitch angle; the intersection of the camera's lens optical axis and the image plane is O(x,y), the focal length is f, the imaging point of the center point of the bottom of the vehicle in front on the image plane is D(u,v), and the angle between the straight line from the center point of the bottom of the vehicle in front to the camera and the lens optical axis is α. If the vehicle in front is to the side and slightly in front of this vehicle, the distance D between them is determined using the following formula: Where γ is the yaw angle of the vehicle in front.

Images