Driving Ignorance Risk Detection Method Based on Matching of In-vehicle and Out-of-vehicle Environments

Abstract

The present invention discloses a method for detecting driving neglect risk based on in-vehicle and out-of-vehicle matching, which relates to the field of driving safety. Since it is difficult for the prior art to accurately integrate the driver's gaze information and the out-of-vehicle environmental data, the method of the present invention converts the out-of-vehicle environmental target data into the in-vehicle coordinate system, constructs the driver's gaze cone, so as to generate a more accurate in-vehicle gaze heat map. In order to improve the accuracy of the matching between the in-vehicle and out-of-vehicle environmental data, the method of the present invention integrates the out-of-vehicle targets and the internal gaze information, realizing the accurate matching of the two and the generation of the heat map. By adding the above data conversion and matching modules to the existing driving risk assessment model, the method of the present invention significantly improves the accuracy of driving risk detection, can be applied to the driving assistance system, and improves the overall driving safety.

Description

Technical Field

[0001] The present invention relates to the field of driving safety, and particularly to a method for detecting driving neglect risks based on the matching of in-vehicle and out-of-vehicle environments. Background Art

[0002] With the development of autonomous driving technology, driving safety has become a crucial research direction. Existing technologies mainly focus on the detection and analysis of the external environment, such as obstacle recognition, lane departure detection, etc. However, these technologies usually ignore in-vehicle factors such as the driver's attention and gaze direction, resulting in the inability to accurately evaluate driving risks at critical moments. In addition, although some systems attempt to combine internal and external data for risk assessment, there are still significant challenges in the accuracy and real-time performance of data fusion and matching. The present invention proposes a new method that can more comprehensively evaluate driving risks by integrating in-vehicle gaze data and external environment detection data, improving the detection accuracy and reaction speed of the system, thereby enhancing driving safety.

[0003] Chinese Patent Application Publication No. CN 114026611 A, "Detecting Driver Attention Using Heat Maps", proposes a method for detecting driver attention based on heat map analysis. This method generates a reference heat map by collecting vehicle data and scene information, and captures the driver's gaze data through a driver distraction system to generate a gaze heat map. However, the in-vehicle driver data and out-of-vehicle environment data mentioned in this patent are processed independently, without considering the mapping from out-of-vehicle targets to in-vehicle images and the mapping from in-vehicle heat maps to out-of-vehicle images. The generated in-vehicle heat map cannot accurately reflect the actual driving scenario. This method lacks a direct connection between the driver's attention distribution and the actual external environment, likely resulting in deviations in the gaze heat map.

[0004] Chinese Patent Application Publication No. CN 115082895 A, "An Intelligent Vehicle Vision Target Detection Method Based on Attention Neural Network", provides an intelligent vehicle vision target detection method based on attention neural network. This patent includes driver images and out-of-vehicle images, but does not consider the mapping from out-of-vehicle targets to in-vehicle images, which will lead to errors in the in-vehicle gaze heat map.

[0005] However, in the detection of driving neglect risks based on the matching of in-vehicle and out-of-vehicle environments, we need to fully consider the problem of the combination of in-vehicle driver data and out-of-vehicle environment data. We map out-of-vehicle targets into the in-vehicle coordinate system to construct a gaze relationship graph model, thereby constructing a more accurate gaze heat map. Moreover, we transform the in-vehicle gaze heat map into the out-of-vehicle coordinate system and match it with out-of-vehicle targets. In this way, we can more accurately detect the situation where the driver ignores out-of-vehicle risks and has stronger representational ability when detecting the driver's driving risks. Summary of the Invention

[0006] The object of the present invention is to make up for the defects of the existing technologies and provide a driving neglect risk detection method based on the matching of the in-vehicle and out-of-vehicle environments. The present invention converts the out-of-vehicle targets into the in-vehicle coordinate system, constructs an accurate gaze cone, and generates a more accurate gaze heat map. First, the target data in the out-of-vehicle environment is acquired and converted into the in-vehicle coordinate system. Subsequently, this data is combined with the driver's gaze information to generate a gaze heat map, and a risk perception model that matches the in-vehicle and out-of-vehicle environments is constructed and trained. Finally, the risk perception model is used to obtain a risk perception score and evaluate the driving risk based on the matching of the in-vehicle and out-of-vehicle environments. If the threshold is exceeded, a warning is issued to remind the driver of potential safety hazards.

[0007] The present invention is achieved through the following technical solutions: A driving neglect risk detection method based on the matching of the in-vehicle and out-of-vehicle environments, specifically including the following steps:

[0008] S1: Define the coordinate system and the data set;

[0009] S2: Detect the out-of-vehicle environment;

[0010] S3: Construct a gaze relationship graph model;

[0011] S4: Calculate the in-vehicle gaze heat map;

[0012] S5: Match the in-vehicle and out-of-vehicle environments;

[0013] S6: Judge the risk alarm;

[0014] The specific steps of defining the coordinate system and the data set in step S1 are as follows:

[0015] S1-1: Define the in-vehicle coordinate system S in ={O in , X in , Y in , Z in};

[0016] S1-1-1: Define the origin O in as the center of the in-vehicle camera;

[0017] S1-1-2: Define the X-axis (X in ) of the coordinate system as the horizontal axis;

[0018] S1-1-3: Define the Y-axis (Y in ) of the coordinate system as the vertical axis;

[0019] S1-1-4: Define the Z-axis (Z in ) of the coordinate system as the depth axis;

[0020] S1-2: Define the external vehicle coordinate system S out ={O out , X out , Y out , Z out}:

[0021] S1-2-1: Define the origin O out as the center of the external vehicle camera;

[0022] S1-2-2: Define the X-axis (X out ) of the coordinate system as the horizontal axis;

[0023] S1-2-3: Define the Y-axis (Y out ) of the coordinate system as the vertical axis;

[0024] S1-2-4: Define the Z-axis (Z out ) of the coordinate system as the depth axis, pointing to the end of the road;

[0025] S1-3: Define the in-vehicle / external vehicle image coordinate system S image :

[0026] S1-3-1: Define the origin O image as the upper left corner vertex of the in-vehicle / external vehicle image;

[0027] S1-3-2: Define the X-axis (X image ) of the coordinate system as the horizontal axis;

[0028] S1-3-3: Define the Y-axis (Y image ) of the coordinate system as the vertical axis;

[0029] S1-4: Define the dataset, and the specific steps are as follows:

[0030] S1-4-1: Define the driver's head dataset {Driver t}: Use an RGB-D camera to obtain the driver's head video frames, denoted as Driver t :

[0031]

[0032] Among them,

[0033] Driver t is the RGBD video frame at time t, where t = 1, 2,..., T

[0034] are the coordinates of the pixel points;

[0035] RGB_in n is the RGB color information of the pixel points;

[0036] depth_in n is the depth value of the pixel point;

[0037] N is the total number of pixels in the video frame;

[0038] S1-4-2: Define the out-of-vehicle dataset {F t}: Use an RGB-D camera to obtain an out-of-vehicle RGB-D video frame, denoted as F t :

[0039]

[0040] where F t is the RGBD video frame at time t, t = 1, 2,..., T;

[0041] are the coordinates of the pixel point;

[0042] RGB_out n is the RGB color information of the pixel point;

[0043] depth_out n is the depth value of the pixel point;

[0044] N is the total number of pixels in the video frame;

[0045] S1-4-3: Define the true gaze heatmap dataset as

[0046] Detect the out-of-vehicle environment as described in step S2, and the specific steps are as follows:

[0047] S2-1: Extract out-of-vehicle targets:

[0048] S2-1-1: Input the video frame F t , and use the DETR (Detection Transformer) model to extract the set O of out-of-vehicle targets t = {object b |b = 1, 2,..., B}, where t is the time, b is the index of the target, and B is the total number of targets;

[0049] S2-1-2: Each target object b is a rectangular area, and the upper left vertex of this rectangular area is The lower right vertex is is the X-axis out-of-vehicle image coordinate of the upper left corner of the target, is the Y-axis out-of-vehicle image coordinate of the upper left corner of the target; is the X-axis out-of-vehicle image coordinate of the lower right corner of the target, is the Y-axis out-of-vehicle image coordinate at the lower right corner of the target;

[0050] S2-2: Calculate the 3D in-vehicle coordinates of the out-of-vehicle target, and the specific steps are as follows:

[0051] S2-2-1: Calculate the average depth of the target,

[0052]

[0053] where, |object b | is the total number of pixels in the target object b depth_out n is the depth value of the pixel point, and n is the index of the pixel point;

[0054] S2-2-2: Calculate the 2D out-of-vehicle image coordinates of the target center:

[0055] S2-2-2-1: Input the upper left vertex of the target Input the lower right vertex of the target

[0056] S2-2-2-2: Calculate the 2D out-of-vehicle image coordinates of the target center

[0057] S2-2-3: Use the camera internal parameters to convert the 2D out-of-vehicle image coordinates to the out-of-vehicle coordinate system S out , and obtain the 3D out-of-vehicle coordinates of the target center

[0058]

[0059] where, d b is the average depth of the target, and R image_to_out is the rotation matrix, and the two constitute the internal parameters for converting from the out-of-vehicle image coordinate system to the out-of-vehicle coordinate system;

[0060] S2-2-4: Use the external parameters to convert the out-of-vehicle target center from the out-of-vehicle coordinate system S out to the in-vehicle coordinate system S in , and obtain the 3D in-vehicle coordinates of the target center

[0061]

[0062] where, R out_to_in is the rotation matrix, and T out_to_in is the translation vector, and the two constitute the external parameters for converting from the out-of-vehicle coordinate system to the in-vehicle coordinate system;

[0063] S2-3: Define the risk type:

[0064] S2-3-1: Define risk type 1 as the risk of too close a distance to the vehicle in front;

[0065] S2-3-2: Define risk type 2 as the risk of the vehicle in front changing lanes sideways;

[0066] S2-3-3: Define risk type 3 as the risk of the own vehicle deviating from the lane;

[0067] S2-4: Calculate the risk value of the dangerous target, and the specific steps are as follows

[0068] S2-4-1: Calculate the risk value of too close a distance to the vehicle in front:

[0069] S2-4-1-1: Input a single-frame external vehicle image F t , extract the depth value of the own vehicle, extract the depth value of the vehicle in front, and calculate the depth distance Dz b，t ;

[0070] S2-4-1-2: Calculate the relative speed of the own vehicle and the vehicle in front, and the specific steps are as follows:

[0071] S2-4-1-2-1: Input two consecutive frames of external vehicle images F t 、F t-1 ;

[0072] S2-4-1-2-2: Calculate the Y-axis coordinate of the center of the own vehicle in each frame, and calculate the Y-axis coordinate of the center of the vehicle in front in each frame;

[0073] S2-4-1-2-3: Calculate the difference in Y-axis coordinates ΔY between the two in each frame t 、ΔY t-1 , and calculate the time difference Δt;

[0074] S2-4-1-2-4: Calculate the relative speed of the own vehicle and the vehicle in front,

[0075]

[0076] S2-4-1-3: Calculate the risk value of too close a distance to the vehicle in front,

[0077]

[0078] Among them, the depth distance Dz b，t The larger it is, the smaller the risk; the relative speed V_rel b，t The larger it is, the greater the risk;

[0079] S2-4-2: Calculate the risk value of the vehicle in front changing lanes sideways:

[0080] S2-4-2-1: Input a single-frame external vehicle image F t, Extract the X-axis coordinate of the vehicle, extract the X-axis coordinate of the vehicle in front, and calculate the horizontal distance Dx between the vehicle itself and the vehicle in front b，t ;

[0081] S2-4-2-2: Input a single-frame external vehicle image F t , Extract the depth value of the vehicle, extract the depth value of the vehicle in front, and calculate the depth distance Dz between the vehicle itself and the vehicle in front b，t ;

[0082] S2-4-2-3: Calculate the lane-changing speed of the vehicle in front. The specific steps are as follows:

[0083] S2-4-2-3-1: Input two consecutive frames of external vehicle images F t 、F t-1 ;

[0084] S2-4-2-3-2: Calculate the central X-axis coordinate of the vehicle in front in each frame

[0085] S2-4-2-3-3: Calculate the lane-changing speed of the vehicle in front

[0086]

[0087] S2-4-2-4: Calculate the risk value of the vehicle in front's side lane change

[0088]

[0089] Among them, the lane-changing speed of the vehicle in front is V_change b , the larger t is, the greater the risk. The horizontal distance Dx b , the larger t is; the smaller the risk; the depth distance Dz b,t the larger it is, the smaller the risk, λ change , λ z is the weight,;

[0090] S2-4-3: Calculate the risk value of the vehicle itself deviating from the lane:

[0091] S2-4-3-1: Calculate the speed of deviating from the lane. The specific steps are as follows:

[0092] S2-4-3-1-1: Input two consecutive frames of external vehicle images F t 、F t-1 ;

[0093] S2-4-3-1-2: Calculate the lane deviation distance D_lane of each frame b,t and D_lane b,t-1 ;

[0094] S2-4-3-1-3: Calculate the speed of deviating from the lane

[0095]

[0096] S2-4-3-2: Calculate the risk value of the vehicle deviating from the lane,

[0097] Risk_deviation b = V_deviation b , t

[0098] where the lane deviation speed V_deviation b , the larger t is, the greater the risk;

[0099] S2-4-4: Calculate the total risk value of each target:

[0100] Risk_max b = max(Risk_change b , Risk_deviation b , Risk_s b )

[0101] S2-4-5: Input the risk value of each target object b If its Risk_max b is greater than the threshold, then the target b is judged as a risk target, and its risk flag Risk_pred b is set to 1, otherwise it is 0;

[0102] The steps of constructing the gaze relationship graph model described in step S3 are as follows:

[0103] S3-1: Input the driver's head video frame Driver t ;

[0104] S3-2: Calculate the node features of the graph model, and the specific steps are as follows:

[0105] S3-2-1: Input the driver's head image block and calculate the driver's head features;

[0106] S3-2-1-1: Calculate the 3D in-vehicle coordinates of the center point of the driver's head

[0107] S3-2-1-1-1: Use the YOLO model to calculate the 2D in-vehicle image coordinates of the center point of the driver's head, and obtain

[0108] S3-2-1-1-2: Extract the depth depth_in t of the center point of the driver's head from Driver head ;

[0109] S3-2-1-1-3: Using the camera intrinsic parameters, convert the 2D in-vehicle image coordinates to the in-vehicle coordinate system S in , and obtain the 3D in-vehicle coordinates of the center point of the driver's head

[0110]

[0111] where, R image_to_in is the rotation matrix, and the two constitute the internal parameters for converting from the in-vehicle image coordinate system to the in-vehicle coordinate system;

[0112] S3-2-1-2: Use the ResNet50 network to calculate the head appearance feature A head ;

[0113] S3-2-1-3: Calculate the head feature of the driver

[0114]

[0115] S3-2-2: Input the out-of-vehicle target image patch and calculate the feature of the out-of-vehicle target b;

[0116] S3-2-2-1: Calculate the 3D out-of-vehicle coordinates of the out-of-vehicle target b

[0117] S3-2-2-2: Convert the out-of-vehicle target b to the in-vehicle coordinate system to obtain

[0118] S3-2-2-3: Use the ResNet50 network to calculate the appearance feature A_object of the out-of-vehicle target b b ;

[0119] S3-2-2-4: Calculate the feature of the out-of-vehicle target b

[0120] S3-2-3: Construct the node set V = {F head} ∪ {F_object b}, b = 1, 2,..., B;

[0121] S3-3: Construct the relationship edge feature of the graph model:

[0122] S3-3-1: Concatenate the driver's head feature and the out-of-vehicle target feature to obtain the preliminary feature:

[0123] F head，b = concat(F head , F_object b )

[0124] S3-3-2: Calculate the edge features of the graph model,

[0125] E head，b = sigmoid(W edge ·F head，b + bias edge )

[0126] where sigmoid is the activation function, W edge is the weight matrix of the fully connected layer, and bias edge is the bias vector of the fully connected layer;

[0127] S3-3-3: Construct the edge set

[0128] S3-4: Construct the gaze relationship graph model

[0129] S3-5: Calculate the edge scores of the graph model, and the specific steps are as follows;

[0130] S3-5-1: Input the edge feature E _ head b ;

[0131] S3-5-2: Calculate the edge scores,

[0132] Score_e head，b = sigmoid(W score ·E head，b + bias score )

[0133] where sigmoid is the activation function, W score is the weight matrix for score calculation, and bias score is the bias vector;

[0134] S3-5-3: Output the edge score sequence {Score_e head，b};

[0135] The construction of the gaze heat map described in step S4 is specifically as follows:

[0136] S4-1: Calculate the gaze direction:

[0137] S4-1-1: Sort the edge score sequence and select the edge with the highest score,

[0138] [b * , Socre_max] = argmax b ({Score_e head，b})

[0139] Then define the out-of-vehicle target point with the highest score as gaze = b * , and obtain the driver's gaze point as

[0140] S4-1-2: Calculate the gaze direction

[0141] S4-2: Construct a gaze cone, and the specific steps are as follows:

[0142] S4-2-1: Set the vertex of the gaze cone as the center point of the driver's head

[0143] S4-2-2: Set the direction of the gaze cone as

[0144] S4-2-3: Set the apex angle of the gaze cone as θ (0 < θ < π). A diameter of the cone bottom and the vertex form a triangle, and the apex angle of this triangle is the apex angle of the cone;

[0145] S4-2-4: Set the maximum gaze distance of the gaze cone as h, representing the effective range of the driver's line of sight;

[0146] S4-2-5: Generate the gaze cone

[0147]

[0148] where (x, y, z) is a point in three-dimensional space in the in-vehicle coordinate system;

[0149] is the vector from the point (x, y, z) to the cone vertex ;

[0150] is the dot product of the vector from the point (x, y, z) to the cone vertex and the gaze direction , and this dot product is used to calculate the angle between the two vectors;

[0151] Calculate the cosine value of the angle between the vector from the point (x, yz) to the vertex and the gaze direction , and this cosine value is used to determine whether the point is inside the cone;

[0152] is the distance from the point (x, y, z) to the cone vertex , is used to verify the distance from the point (x, y, z) to the cone vertex The distance is between 0 and the maximum gazing distance h;

[0153] S4-3: Generate a gazing heatmap, and the specific steps are as follows:

[0154] S4-3-1: Input the set of targets in the out-of-vehicle image

[0155] S4-3-2: Calculate the set of gazing targets within the gazing cone Region

[0156]

[0157] S4-3-3: Initialize the gazing heatmap, Heatmap in (x, y, z) = 0, where (x, y, z) are the in-vehicle coordinates;

[0158] S4-3-4: Calculate the gazing value of each gazing target

[0159]

[0160] where b ∈ FocusObjects is the gazing point of the driver, and σ is a Gaussian standard deviation parameter;

[0161] S4-3-5: Repeat step S3-3-4 to update the heatmap matrix for b ∈ FocusObjects to obtain the gazing heatmap Heatmap in (x, y, z);

[0162] S4-3-6: Output the gazing heatmap Heatmap in (x, y, z);

[0163] S4-4: Calculate the gazing heatmap loss, and the specific steps are as follows:

[0164] S4-4-1: Input the true gazing heatmap

[0165] S4-4-2: Calculate the gazing heatmap loss:

[0166]

[0167] where N is the total number of pixels in the heatmap;

[0168] The in-vehicle and out-of-vehicle environment matching described in step S5 is as follows:

[0169] S5-1: Input the in-vehicle gazing heatmap Heatmap in (x, y, z);

[0170] S5-2: Convert Heatmap in (x, y, z) to the external coordinate system of the vehicle to obtain the Heatmap out (x, y, z), and the specific steps are as follows:

[0171] Use external parameters to convert the Heatmap in Each point of (x, y, z) from the internal coordinate system S of the vehicle in to the external coordinate system S of the vehicle out :

[0172]

[0173] Among them, R in_to_out is the rotation matrix, and T in_to_out is the translation vector, which together form the external parameters for converting from the internal coordinate system to the external coordinate system of the vehicle;

[0174] S5-3: Calculate the target detection score, and the specific steps are as follows:

[0175] S5-3-1: Input each external target point

[0176] S5-3-2: Calculate the target detection score for each external target:

[0177]

[0178] S5-4: Calculate the risk detection loss:

[0179]

[0180] Among them, B is the total number of external targets; Risk_pred b is the risk label of the external target b, and the Risk_pred b value of 1 indicates a risk target, and the Risk_pred b value of 0 indicates a non-risk target;

[0181] S5-5: Calculate the total loss:

[0182] L total = L gaze + L risk

[0183] Among them, L gaze is obtained from S4-4-2;

[0184] S5-6: Use the loss L total , perform model training to obtain the optimal parameters, and the parameters include those in S3-4-3 and S4-3-4;

[0185] The judgment of risk warning described in step S6 is as follows:

[0186] S6-1: Use S2-4 to calculate the 3D in-vehicle coordinates of the out-of-vehicle risk target

[0187] S6-2: Use the model parameters of S3-4-3, execute S3, and calculate the edge scores of the gaze relationship graph module

[0188] Score_e head，b :

[0189] S6-3: Use the model parameters of S4-3-4, execute S4, and generate the gaze heatmap Heatmap in (x, y, z);

[0190] S6-4: Use S5 to calculate the target detection score:

[0191] S6-4-1: Use S5-2 to convert the gaze heatmap to the out-of-vehicle coordinate system to obtain Heatmap out (x, y, z);

[0192] S6-4-2: Use S5-3 to calculate the target detection score score_risk of the out-of-vehicle target and the driver's gaze point b ;

[0193] S6-5: Output an alarm signal, and for each out-of-vehicle target b, make the following judgments in sequence:

[0194] S6-5-1: If Risk_pred b = 0, it means that the target b is a non-risk target and no alarm is triggered;

[0195] S6-5-2: If Risk_pred b = 1, and score_risk b is greater than the threshold δ, it means that the target b is a risk target, and the matching degree between the out-of-vehicle target b and the driver's gaze point is high, that is, the driver notices the risk target and no alarm is triggered;

[0196] S6-5-3: If Risk_pred b = 1, and score_risk b is less than the threshold δ, it means that the target b is a risk target, but the matching degree between the out-of-vehicle target b and the driver's gaze point is low, that is, the driver ignores the risk target and an alarm is triggered.

[0197] The advantages of the present invention are as follows: By converting the external environment target data into the vehicle interior coordinate system, the present invention constructs the driver's gaze cone to generate a more accurate in-vehicle gaze heat map; in order to improve the accuracy of matching the in-vehicle and external environment data, the present invention integrates the external target and internal gaze information to achieve accurate matching and heat map generation of the two; by adding the above data conversion and matching module to the existing driving risk assessment model, the present invention significantly improves the accuracy of driving risk detection, can be applied to the driving assistance system, and improves the overall driving safety. Description of the Drawings

[0198] Figure 1 It is a flowchart for defining the coordinate system and data set mentioned in the present invention;

[0199] Figure 2 It is a flowchart for detecting the external environment mentioned in the present invention;

[0200] Figure 3 It is a flowchart for constructing the in-vehicle gaze relationship graph model mentioned in the present invention;

[0201] Figure 4 It is a flowchart for calculating the in-vehicle gaze heat map mentioned in the present invention;

[0202] Figure 5 It is a flowchart for matching the in-vehicle and external environments mentioned in the present invention;

[0203] Figure 6 It is a flowchart for judging risk warnings mentioned in the present invention;

[0204] Figure 7 It is a flowchart for implementing the embodiment of the present invention. Detailed Embodiment

[0205] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be introduced in detail below with reference to the accompanying drawings and specific embodiments;

[0206] As Figure 7 shown, a driving inattention risk detection method based on in-vehicle and external environment matching specifically includes the following steps:

[0207] S1: Define the coordinate system required in the implementation steps, and the specific steps are as follows:

[0208] S1-1: Define the in-vehicle coordinate system S in ={O in , X in , Y in , Z in}, as Figure 1 shown;

[0209] S1-1-1: Define the origin Oin is the center of the in-vehicle camera;

[0210] S1-1-2: Define the X-axis (X in ) of the coordinate system as the horizontal axis;

[0211] S1-1-3: Define the Y-axis (Y in ) of the coordinate system as the vertical axis;

[0212] S1-1-4: Define the Z-axis (Z in ) of the coordinate system as the depth axis;

[0213] S1-2: Define the out-of-vehicle coordinate system S out ={O out , X out , Y out , Z out}, as Figure 1 shown:

[0214] S1-2-1: Define the origin O out as the center of the out-of-vehicle camera, as Figure 1 shown;

[0215] S1-2-2: Define the X-axis (X out ) of the coordinate system as the horizontal axis;

[0216] S1-2-3: Define the Y-axis (Y out ) of the coordinate system as the vertical axis;

[0217] S1-2-4: Define the Z-axis (Z out ) of the coordinate system as the depth axis, pointing to the end of the road;

[0218] S1-3: Define the in-vehicle / out-of-vehicle image coordinate system S image , as Figure 1 shown:

[0219] S1-3-1: Define the origin O image as the upper left corner vertex of the in-vehicle / out-of-vehicle image;

[0220] S1-3-2: Define the X-axis (X image ) of the coordinate system as the horizontal axis;

[0221] S1-3-3: Define the Y-axis (Y image ) of the coordinate system as the vertical axis;

[0222] S1-4: Define the data set, the specific steps are as follows:

[0223] S1-4-1: Define the driver's head data set {Driver t}: Use an RGB-D camera to obtain the driver's head video frame, denoted as Driver t :

[0224]

[0225] Among them,

[0226] Driver t is the RGB-D video frame at time t, where t = 1, 2,..., T

[0227] are the coordinates of the pixel point;

[0228] RGB_in n is the RGB color information of the pixel point;

[0229] depth_in n is the depth value of the pixel point;

[0230] N is the total number of pixels in the video frame;

[0231] S1-4-2: Define the out-of-vehicle dataset {F t}: Use an RGB-D camera to obtain the out-of-vehicle RGB-D video frame, denoted as F t :

[0232]

[0233] Among them, F t is the RGB-D video frame at time t, where t = 1, 2,..., T;

[0234] are the coordinates of the pixel point;

[0235] RGB_out n is the RGB color information of the pixel point;

[0236] depth_out n is the depth value of the pixel point;

[0237] N is the total number of pixels in the video frame;

[0238] S1-4-3: Define the true gaze heatmap dataset as

[0239] S2: Detect the out-of-vehicle environment, as Figure 2 shown, the specific steps are as follows;

[0240] S2-1: Extract the out-of-vehicle target, the specific steps are as follows:

[0241] S2-1-1: Input the video frame F t, use the DETR (Detection Transformer) model to extract the set O of external vehicle targets t ={object b |b = 1, 2, ..., B}, where t is the time, b is the index of the target, and B is the total number of targets;

[0242] S2-1-2: Each target object b is a rectangular area, and the upper left vertex of this rectangular area is The lower right vertex is is the X-axis external vehicle image coordinate of the upper left corner of the target, is the Y-axis external vehicle image coordinate of the upper left corner of the target; is the X-axis external vehicle image coordinate of the lower right corner of the target, is the Y-axis external vehicle image coordinate of the lower right corner of the target;

[0243] S2-2: Calculate the 3D in-vehicle coordinates of the external vehicle targets. The specific steps are as follows:

[0244] S2-2-1: Calculate the average depth of the target,

[0245]

[0246] where, |object b | is the total number of pixels in the target object b depth_out n is the depth value of the pixel point, and n is the index of the pixel point;

[0247] S2-2-2: Calculate the 2D external vehicle image coordinates of the target center:

[0248] S2-2-2-1: Input the upper left vertex of the target Input the lower right vertex of the target

[0249] S2-2-2-2: Calculate the 2D external vehicle image coordinates of the target center

[0250] S2-2-3: Use the camera internal parameters to convert the 2D external vehicle image coordinates to the external vehicle coordinate system S out , and obtain the 3D external vehicle coordinates of the target center

[0251]

[0252] where, d b is the average depth of the target, R image_to_outis the rotation matrix, and the two constitute the internal parameters for converting from the external vehicle image coordinate system to the external vehicle coordinate system;

[0253] S2-2-4: Use the external parameters to convert the center of the external vehicle target from the external vehicle coordinate system S out to the internal vehicle coordinate system S in to obtain the 3D internal vehicle coordinates of the target center

[0254]

[0255] where R out_to_in is the rotation matrix, and T out_to_in is the translation vector. The two constitute the external parameters for converting from the external vehicle coordinate system to the internal vehicle coordinate system;

[0256] S2-3: Define the risk types:

[0257] S2-3-1: Define risk type 1 as the risk of too close distance to the vehicle in front;

[0258] S2-3-2: Define risk type 2 as the risk of the vehicle in front changing lanes sideways;

[0259] S2-3-3: Define risk type 3 as the risk of the own vehicle deviating from the lane;

[0260] S2-4: Calculate the risk value of the dangerous target. The specific steps are as follows

[0261] S2-4-1: Calculate the risk value of too close distance to the vehicle in front:

[0262] S2-4-1-1: Input a single-frame external vehicle image F t , extract the depth value of the own vehicle, extract the depth value of the vehicle in front, and calculate the depth distance Dz b，t ;

[0263] S2-4-1-2: Calculate the relative speed of the own vehicle and the vehicle in front. The specific steps are as follows:

[0264] S2-4-1-2-1: Input two consecutive frames of external vehicle images F t and F t-1 ;

[0265] S2-4-1-2-2: Calculate the Y-axis coordinate of the center of the own vehicle in each frame, and calculate the Y-axis coordinate of the center of the vehicle in front in each frame;

[0266] S2-4-1-2-3: Calculate the difference in Y-axis coordinates ΔY t and ΔY t-1 in each frame, and calculate the time difference Δt;

[0267] S2-4-1-2-4: Calculate the relative speed of the own vehicle and the vehicle in front

[0268]

[0269] S2-4-1-3: Calculate the risk value of too close distance to the vehicle ahead,

[0270]

[0271] Among them, the depth distance Dz b，t The larger it is, the smaller the risk; the relative speed V_rel b，t The larger it is, the greater the risk;

[0272] S2-4-2: Calculate the risk value of the vehicle ahead changing lanes sideways:

[0273] S2-4-2-1: Input a single-frame external vehicle image F t , extract the X-axis coordinate of the vehicle itself, extract the X-axis coordinate of the vehicle ahead, and calculate the horizontal distance Dx between the vehicle itself and the vehicle ahead b，t ;

[0274] S2-4-2-2: Input a single-frame external vehicle image F t , extract the depth value of the vehicle itself, extract the depth value of the vehicle ahead, and calculate the depth distance Dz between the vehicle itself and the vehicle ahead b，t ;

[0275] S2-4-2-3: Calculate the lane-changing speed of the vehicle ahead. The specific steps are as follows:

[0276] S2-4-2-3-1: Input two consecutive frames of external vehicle images F t 、F t-1 ;

[0277] S2-4-2-3-2: Calculate the central X-axis coordinate of the vehicle ahead in each frame

[0278] S2-4-2-3-3: Calculate the lane-changing speed of the vehicle ahead,

[0279]

[0280] S2-4-2-4: Calculate the risk value of the vehicle ahead changing lanes sideways,

[0281]

[0282] Among them, the lane-changing speed V_change of the vehicle ahead b，t The larger it is, the greater the risk, and the horizontal distance Dx b，t The larger it is; the smaller the risk; the depth distance Dz b，t The larger it is, the smaller the risk, and λ change , λ z is the weight,;

[0283] S2-4-3: Calculate the risk value of the vehicle deviating from the lane:

[0284] S2-4-3-1: Calculate the speed of deviating from the lane. The specific steps are as follows:

[0285] S2-4-3-1-1: Input two consecutive frames of out-of-vehicle images F t 、F t-1 ;

[0286] S2-4-3-1-2: Calculate the lane deviation distance D_lane for each frame b，t and D_lane b，t-1 ;

[0287] S2-4-3-1-3: Calculate the speed of deviating from the lane

[0288]

[0289] S2-4-3-2: Calculate the risk value of the vehicle deviating from the lane

[0290] Risk_deviation b =V_deviation b，t

[0291] where the speed of deviating from the lane V_deviation b , the larger t is, the greater the risk;

[0292] S2-4-4: Calculate the total risk value of each target:

[0293] Risk_max b =max(Risk_change b , Risk_deviation b , Risk_s b )

[0294] S2-4-5: Input the risk value of each target object b . If its Risk_max b is greater than the threshold, then this target b is judged as a risk target, and its risk flag Risk_pred b is set to 1, otherwise it is 0;

[0295] S3: Build a gaze relationship graph model, as Figure 3 shown, the specific steps are as follows:

[0296] S3-1: Input the driver's head video frame Driver t ;

[0297] S3-2: Calculate the node features of the graph model, and the specific steps are as follows:

[0298] S3-2-1: Input the driver's head image patch and calculate the driver's head features;

[0299] S3-2-1-1: Calculate the 3D in-vehicle coordinates of the center point of the driver's head

[0300] S3-2-1-1-1: Use the YOLO model to calculate the 2D in-vehicle image coordinates of the center point of the driver's head, and obtain

[0301] S3-2-1-1-2: Extract the depth depth_in of the center point of the driver's head from Driver t ; head

[0302] S3-2-1-1-3: Use the camera internal parameters to convert the 2D in-vehicle image coordinates to the in-vehicle coordinate system S in , and obtain the 3D in-vehicle coordinates of the center point of the driver's head

[0303]

[0304] where, R image_to_in is the rotation matrix, and the two constitute the internal parameters for converting from the in-vehicle image coordinate system to the in-vehicle coordinate system;

[0305] S3-2-1-2: Use the ResNet50 network to calculate the head appearance feature A head ;

[0306] S3-2-1-3: Calculate the driver's head features,

[0307]

[0308] S3-2-2: Input the out-of-vehicle target image patch and calculate the features of the out-of-vehicle target b;

[0309] S3-2-2-1: Calculate the 3D out-of-vehicle coordinates of the out-of-vehicle target b

[0310] S3-2-2-2: Convert the out-of-vehicle target b to the in-vehicle coordinate system to obtain

[0311] S3-2-2-3: Use the ResNetS0 network to calculate the appearance feature A_object of the out-of-vehicle target b b ;

[0312] ​S3-2-2-4: Calculate the features of the external target b

[0313] S3-2-3: Construct the node set V = {F head} ∪ {F_object b}, b = 1, 2,..., B;

[0314] S3-3: Construct the relationship edge features of the graph model:

[0315] S3-3-1: Concatenate the driver's head features and the external target features to obtain the preliminary features:

[0316] F head，b = concat(F head , F_object b )

[0317] S3-3-2: Calculate the edge features of the graph model,

[0318] E head，b = sigmoid(W edge ·F head，b + bias edge )

[0319] where sigmoid is the activation function, W edge is the weight matrix of the fully connected layer, and bias edge is the bias vector of the fully connected layer;

[0320] S3-3-3: Construct the edge set

[0321] S3-4: Construct the gaze relationship graph model

[0322] S3-5: Calculate the edge scores of the graph model, and the specific steps are as follows;

[0323] S3-5-1: Input the edge feature E_head b ;

[0324] S3-5-2: Calculate the edge scores,

[0325] Score_e head，b = sigmoid(W score ·E head，b + bias score )

[0326] where sigmoid is the activation function, W score is the weight matrix for score calculation, and bias score is the bias vector;

[0327] S3-5-3: Output the edge score sequence {Score_e head，b};

[0328] S4: Construct a gaze heat map, as shown below, and the specific steps are as follows: Figure 4 as shown below:

[0329] S4-1: Calculate the gaze direction:

[0330] S4-1-1: Sort the edge score sequence and select the edge with the highest score,

[0331] [b * , Score_max] = argmax b ({Score_e head，b})

[0332] Then define the out-of-vehicle target point with the highest score as gaze = b * , and obtain the driver's gaze point as

[0333] S4-1-2: Calculate the gaze direction

[0334] S4-2: Construct a gaze cone, and the specific steps are as follows:

[0335] S4-2-1: Set the vertex of the gaze cone as the center point of the driver's head

[0336] S4-2-2: Set the direction of the gaze cone as

[0337] S4-2-3: Set the apex angle of the gaze cone as θ (0 < θ < π), as shown below. A diameter of the bottom surface of the cone is connected to the vertex to form a triangle, and the apex angle of this triangle is the apex angle of the cone; Figure 4 as shown below. A diameter of the bottom surface of the cone is connected to the vertex to form a triangle, and the apex angle of this triangle is the apex angle of the cone;

[0338] S4-2-4: Set the maximum gaze distance of the gaze cone as h, representing the effective range of the driver's line of sight;

[0339] S4-2-5: Generate the gaze cone

[0340]

[0341] where (x, y, z) is a point in three-dimensional space in the vehicle coordinate system;

[0342] is the vector from the point (x, y, z) to the vertex of the cone ;

[0343] The dot product of the vector from the point (x, y, z) to the vertex of the cone and the gazing direction is used to calculate the angle between the two vectors;

[0344] Calculate the cosine value of the angle between the vector from the point (x, y, z) to the vertex and the gazing direction This cosine value is used to determine whether the point is inside the cone;

[0345] is the distance from the point (x, y, z) to the vertex of the cone and is used to verify that the distance from the point (x, y, z) to the vertex of the cone is between 0 and the maximum gazing distance h;

[0346] S4-3: Generate a gazing heatmap, and the specific steps are as follows:

[0347] S4-3-1: Input the set of targets in the out-of-vehicle image

[0348] S4-3-2: Calculate the set of gazing targets within the gazing cone Region

[0349]

[0350] S4-3-3: Initialize the gazing heatmap, Heatmapin(x, y, z) = 0, where (x, y, z) are the in-vehicle coordinates;

[0351] S4-3-4: Calculate the gazing value of each gazing target

[0352]

[0353] where b ∈ FocusObjects is the gazing point of the driver, and σ is a Gaussian standard deviation parameter;

[0354] S4-3-5: Repeat step S3-3-4 to update the heatmap matrix for b ∈ FocusObjects to obtain the gazing heatmap Heatmap in (x, y, z);

[0355] S4-3-6: Output the gazing heatmap Heatmap in (x, y, z);

[0356] S4-4: Calculate the gazing heatmap loss, and the specific steps are as follows:

[0357] S4-4-1: Input the real gaze heat map

[0358] S4-4-2: Calculate the gaze heat map loss:

[0359]

[0360] Where N is the total number of pixels in the heat map;

[0361] S5: Match the in-vehicle and out-of-vehicle environments, as Figure 5 shown, the specific steps are as follows:

[0362] S5-1: Input the in-vehicle gaze heat map Heatmap in (x, y, z);

[0363] S5-2: Convert Heatmap in (x, y, z) to the out-of-vehicle coordinate system to obtain Heatmap out (x, y, z), and the specific steps are:

[0364] Use the external parameters to convert each point of Heatmap in (x, y, z) from the in-vehicle coordinate system S in to the out-of-vehicle coordinate system S out :

[0365]

[0366] Where R in_to_out is the rotation matrix, and T in_to_out is the translation vector, which together form the external parameters for converting from the in-vehicle coordinate system to the out-of-vehicle coordinate system;

[0367] S5-3: Calculate the target detection score, and the specific steps are as follows:

[0368] S5-3-1: Input each out-of-vehicle target point

[0369] S5-3-2: Calculate the target detection score for each out-of-vehicle target:

[0370]

[0371] S5-4: Calculate the risk detection loss:

[0372]

[0373] Where B is the total number of out-of-vehicle targets; Risk_pred b is the risk label of out-of-vehicle target b, Risk_predb A value of 1 indicates a risk target, Risk_pred b A value of 0 indicates a non-risk target;

[0374] S5-5: Calculate the total loss:

[0375] L total = L gaze + L risk

[0376] where L gaze is obtained from S4-4-2;

[0377] S5-6: Use the loss L total , perform model training to obtain the optimal parameters, where the parameters include those in S3-4-3 and S4-3-4;

[0378] S6: Judge the risk warning, as Figure 6 shown, the specific steps are as follows:

[0379] S6-1: Use S2-4 to calculate the 3D in-vehicle coordinates of the out-of-vehicle risk target

[0380] S6-2: Use the model parameters of S3-4-3, execute S3, and calculate the edge scores of the gaze relationship graph

[0381] Score_e head，b :

[0382] S6-3: Use the model parameters of S4-3-4, execute S4, and generate the gaze heatmap Heatmap in (x, y, z);

[0383] S6-4: Use S5 to calculate the target detection score:

[0384] S6-4-1: Use S5-2 to convert the gaze heatmap to the out-of-vehicle coordinate system to obtain Heatmap out (x, y, z);

[0385] S6-4-2: Use S5-3 to calculate the target detection score score_risk of the out-of-vehicle target and the driver's gaze point b ;

[0386] S6-5: Output an alarm signal, and for each out-of-vehicle target b, make the following judgments in sequence:

[0387] S6-5-1: If Risk_pred b = 0, it means that the target b is a non-risk target and no alarm is triggered;

[0388] S6-5-2: If Risk_pred b = 1 and score_risk b is greater than the threshold δ, it means that the target b is a risk target, and the matching degree between the off-vehicle target b and the driver's gaze point is high, that is, the driver notices the risk target and no alarm is triggered;

[0389] S6-5-3: If Risk_pred b = 1 and score_risk b is less than the threshold δ, it means that the target b is a risk target, but the matching degree between the off-vehicle target b and the driver's gaze point is low, that is, the driver ignores the risk target and an alarm is triggered.

Claims

1. A method for detecting driving neglect risk based on in-vehicle and out-vehicle environment matching, characterized in that: The specific steps include: S1: define the coordinate system and dataset; S2: Detect the environment outside the vehicle; S3: Construct gaze relationship graph model; S4: Calculate the gaze heat map inside the car; S5: Matching the inside and outside environment of the car; S6: Determine risk alert; The step S3 of constructing the gaze relationship graph model specifically includes the following steps: S3-1: Input driver head video frame Driver t ; S3-2: Node characteristics of the computational graph model; S3-3: Construct the relationship edge features of the graph model; S3-4: construct gaze relationship graph model G = (V, E); S3-5: Calculate edge scores of the graph model; The step S4 of calculating the in-vehicle gaze heat map specifically includes the following steps: S4-1: Calculate gaze direction; S4-2: Constructing gaze cones; S4-3: Generate gaze heatmap; S4-4: Calculate gaze heatmap loss; The matching of the vehicle interior and exterior environments in step S5 specifically includes the following steps: S5-1: Input the in-car gaze heatmap in (x,y,z); S5-2: Convert Heatmap in (x,y,z) to the vehicle's external coordinate system to get the Heatmap out (x,y,z); Using external parameters, Heatmap in Each point (x, y, z) is calculated from the in-vehicle coordinate system S in Transform to the vehicle external coordinate system S out : Among them, R in_to_out is the rotation matrix, T in_to_out is the translation vector, which constitutes the external parameter and is used to transform from the internal coordinate system to the external coordinate system; Step S5-3 calculates the target perception score: S5-3-1: Input each external target point S5-3-2: Calculate the target perception score for each target outside the vehicle: S5-4: Calculate risk perceived loss; S5-5: Calculate total loss; S5-6: Loss of use L total , perform model training and obtain the optimal parameters, which include the parameters in S3-4-3 and S4-3-4.

2. The method for detecting driving neglect risk based on in-vehicle and out-vehicle environment matching according to claim 1, characterized in that: The definition of the coordinate system and the data set described in step S1 specifically includes the following steps: S1-1: Define the in-vehicle coordinate system S in = {O in ,X in ,Y in ,Z in }; where the origin O in is the center of the camera inside the car, and the X axis X in is the horizontal axis, Y axis in is the vertical axis, Z axis in is the depth axis; S1-2: Define the vehicle's external coordinate system S out = {O out ,X out ,Y out ,Z out }: The origin O out is the center of the camera outside the vehicle, and the X axis X out is the horizontal axis; Y axis out is the vertical axis; Z axis out is the depth axis, pointing to the end of the road; S1-3: Define the in-vehicle / out-vehicle image coordinate system S image ; Define the origin O image The top left corner of the image inside or outside the car defines the X axis of the coordinate system. image Is the horizontal axis; defines the Y axis of the coordinate system image is the vertical axis; S1-4: Define the dataset: S1-4-1: Define the driver head dataset {Driver t }: Use the RGB-D camera to obtain the driver's head video frame, denoted as Driver t : in, Driver t is the RGBD video frame at time t, t = 1, 2, ..., T is the coordinate of the pixel point; RGB_in n is the RGB color information of the pixel; depth_in n is the depth value of the pixel; N is the total number of pixels in the video frame; S1-4-2: Define the off-vehicle dataset {F t }: Use the RGB-D camera to obtain the RGB-D video frame outside the car, denoted as F t : Among them, F t is the RGBD video frame at time t, t = 1, 2, ..., T; is the coordinate of the pixel point; RGB_out n is the RGB color information of the pixel; depth_out n is the depth value of the pixel; N is the total number of pixels in the video frame; S1-4-3: Define the real gaze heat map dataset as 3. The method for detecting driving neglect risk based on in-vehicle and out-vehicle environment matching according to claim 2, characterized in that: The detection of the vehicle exterior environment described in step S2 specifically includes the following steps: S2-1: Extracting targets outside the vehicle: S2-1-1: Input video frame F t , using the DETR model, extract the set of external targets O t ={object b |b=1,2,…,B}, where t is the time, b is the index of the target, and B is the total number of targets; S2-1-2: Each target object b is a rectangular area whose upper left corner vertex is The lower right vertex is is the X-axis external image coordinate of the upper left corner of the target, is the Y-axis external image coordinate of the upper left corner of the target; is the X-axis external image coordinate of the lower right corner of the target, is the Y-axis external image coordinate of the lower right corner of the target; S2-2: Calculate the 3D in-vehicle coordinates of the external target; S2-3: Define risk types; S2-4: Calculate the risk value of dangerous targets.

4. The method for detecting driving neglect risk based on in-vehicle and out-vehicle environment matching according to claim 3, characterized in that: The step S2-2 of calculating the 3D in-vehicle coordinates of the external target specifically includes the following steps: S2-2-1: Calculate the average depth of the target, Among them, |object b | is the target object b The total number of pixels in depth_out n is the depth value of the pixel, and n is the index of the pixel; S2-2-2: Calculate the 2D external image coordinates of the target center: S2-2-2-1: Enter the top left corner of the target Enter the target lower right corner vertex S2-2-2-2: Calculate the 2D external image coordinates of the target center S2-2-3: Use the camera internal parameters to convert the 2D vehicle external image coordinates Transform to the vehicle external coordinate system S out , get the 3D external coordinates of the target center Among them, d b is the average depth of the target, R image_to_out is the rotation matrix, and the two constitute the internal parameters used to transform from the external image coordinate system to the external coordinate system; S2-2-4: Use external parameters to convert the center of the target outside the vehicle from the external coordinate system S out Transform to the in-vehicle coordinate system S in , get the 3D in-vehicle coordinates of the target center Among them, R out_to_in is the rotation matrix, T out_to_in is the translation vector, and the two constitute the external parameters, which are used to transform from the external coordinate system to the internal coordinate system; The step S2-3 of defining the risk type specifically includes the following steps: S2-3-1: Define risk type 1 as the risk of the vehicle ahead being too close; S2-3-2: Define risk type 2 as the risk of the vehicle ahead changing lanes sideways; S2-3-3: Define risk type 3 as the risk of the vehicle drifting from the lane; The calculation of the risk value of the dangerous target described in step S2-4 specifically includes the following steps: S2-4-1: Calculate the risk value of the vehicle ahead being too close: S2-4-1-1: Input a single frame of vehicle exterior image F t , extract the depth value of the vehicle, extract the depth value of the front vehicle, and calculate the depth distance Dz b,t ; S2-4-1-2: Calculate the relative speed between the vehicle and the preceding vehicle. The specific steps are as follows: S2-4-1-2-1: Input two consecutive frames of vehicle exterior images F t 、F t-1 ; S2-4-1-2-2: Calculate the Y-axis coordinate of the center of the self-vehicle in each frame, and calculate the Y-axis coordinate of the center of the front vehicle in each frame; S2-4-1-2-3: Calculate the Y-axis coordinate difference ΔY between the two in each frame t , ΔY t-1 , calculate the time difference Δt; S2-4-1-2-4: Calculate the relative speed between the vehicle and the preceding vehicle. S2-4-1-3: Calculate the risk value of the vehicle ahead being too close. Among them, the depth distance Dz b,t The larger the relative speed V_rel, the smaller the risk. b,t The bigger it is, the greater the risk; S2-4-2: Calculate the risk value of the vehicle ahead changing lanes sideways: S2-4-2-1: Input a single frame of vehicle exterior image F t , extract the X-axis coordinate of the vehicle, extract the X-axis coordinate of the front vehicle, and calculate the horizontal distance Dx between the vehicle and the front vehicle b,t ; S2-4-2-2: Input a single frame of vehicle exterior image F t , extract the depth value of the vehicle, extract the depth value of the front vehicle, and calculate the depth distance Dz between the vehicle and the front vehicle b,t ; S2-4-2-3: Calculate the lane-changing speed of the vehicle ahead. The specific steps are as follows: S2-4-2-3-1: Input two consecutive frames of vehicle exterior images F t 、F t-1 ; S2-4-2-3-2: Calculate the center X-axis coordinate of the front vehicle in each frame S2-4-2-3-3: Calculate the lane-changing speed of the vehicle ahead. S2-4-2-4: Calculate the risk value of the vehicle ahead changing lanes sideways. Among them, the lane change speed of the front vehicle V_change b,t The larger the value, the greater the risk, and the horizontal distance Dx b,t The larger the risk, the lower the depth distance Dz b,t The larger the value, the smaller the risk. change ,λ z is the weight; S2-4-3: Calculate the risk value of the vehicle deviating from the lane: S2-4-3-1: Calculate the speed of the offset lane. The specific steps are as follows: S2-4-3-1-1: Input two consecutive frames of vehicle exterior images F t 、F t-1 ; S2-4-3-1-2: Calculate the lane offset distance D_lane for each frame b,r and Délane b,t-1 ; S2-4-3-1-3: Calculate the speed of the offset lane, S2-4-3-2: Calculate the risk value of the vehicle's lane deviation. Risk_deviation b =V_deviation b,t Among them, the deviation lane speed V_deviation b,t The bigger it is, the greater the risk; S2-4-4: Calculate the total risk value of each target: Risk_max b =max(Risk_change b ,Risk_deviation b ,Risk_s b ) S2-4-5: Enter each target object b The risk value of b If it is greater than the threshold, the target b is judged as a risk target and its risk flag Risk_pred is set. b is 1, otherwise it is 0.

5. The method for detecting driving neglect risk based on in-vehicle and out-vehicle environment matching according to claim 1, characterized in that: The node features of the computational graph model described in step S3-2 specifically include the following steps: S3-2-1: input the driver's head image block and calculate the driver's head features; S3-2-1-1: Calculate the 3D in-vehicle coordinates of the center point of the driver's head; S3-2-1-1-1: Use the YOLO model to calculate the 2D in-car image coordinates of the center point of the driver’s head. get S3-2-1-1-2: From Driver t Extract the depth depth_in of the center point of the driver's head head ; S3-2-1-1-3: Use the camera internal reference to convert the 2D in-vehicle image coordinates Transform to the in-vehicle coordinate system S in , get the 3D in-car coordinates of the center point of the driver's head Among them, R image_to_in is the rotation matrix, and the two constitute the internal parameters used to transform from the in-vehicle image coordinate system to the in-vehicle coordinate system; S3-2-1-2: Use the ResNet50 network to calculate the head appearance feature A head ; S3-2-1-3: Calculate the driver's head features, S3-2-2: Input the image block of the target outside the vehicle and calculate the features of the target b outside the vehicle; S3-2-2-1: Calculate the 3D external coordinates of the external target b S3-2-2-2: Convert the external target b to the internal coordinate system, and get S3-2-2-3: Use the ResNet50 network to calculate the appearance feature A_object of the target b outside the vehicle b ; S3-2-2-4: Calculate the features of the target b outside the vehicle S3-2-3: Construct node set V = {F head }∪{F_object b },b=1,2,…,B; The step S3-3 of constructing the relationship edge features of the graph model specifically includes the following steps: S3-3-1: Combine the driver's head features and the target features outside the vehicle to obtain preliminary features: F head,b =concat(F head ,F_object b ) S3-3-2: Calculate the edge features of the graph model, E head,b =sigmoid(W edge ·F head,b +bias edge ) Among them, sigmoid is the activation function, W edge is the weight matrix of the fully connected layer, bias edge is the bias vector of the fully connected layer; S3-3-3: Construct edge set E = {E head,b }; The calculation of the edge score of the graph model in step S3-5 specifically includes the following steps: S3-5-1: Input edge feature E_head b ; S3-5-2: Calculate edge score, Score_e head,b =sigmoid(W score ·E head,b +bias score ) Among them, sigmoid is the activation function, W score The weight matrix for score calculation, bias score is the bias vector; S3-5-3: Output edge score sequence {Score_e head,b }.

6. The method for detecting driving neglect risk based on in-vehicle and out-vehicle environment matching according to claim 1, characterized in that: The construction of the gaze cone described in step S4-2 specifically includes the following steps: S4-2-1: Set the vertex of the gaze cone to the center of the driver's head S4-2-2: Set the direction of the gaze cone to S4-2-3: Set the vertex angle of the gaze cone to θ, 0<θ<π; S4-2-4: Set the maximum gaze distance of the gaze cone to h, which represents the effective range of the driver's line of sight; S4-2-5: Generate gaze cone Among them, (x, y, z) is a point in the three-dimensional space in the vehicle's internal coordinate system; From point (x, y, z) to the vertex of the cone A vector of From point (x,y,z) to the vertex of the cone The vector and gaze direction The dot product is used to calculate the angle between two vectors; Calculate the distance from point (x,y,z) to vertex The vector and gaze direction The cosine value of the angle between the two points is used to determine whether the point is within the cone. From point (x,y,z) to the vertex of the cone The distance, Used to verify the point (x, y, z) to the cone vertex The distance is between 0 and the maximum gaze distance h; The step S4-3 of generating the gaze heat map specifically includes the following steps: S4-3-1: Input the set of targets in the image outside the vehicle S4-3-2: Calculate the gaze target set within the gaze cone Region. S4-3-3: Initialize gaze heatmap, Heatmap in (x, y, z) = 0, (x, y, z) is the coordinate inside the car; S4-3-4: Calculate the gaze value of each gaze target, Among them, b∈FocusObjects, is the driver’s gaze point, σ is a Gaussian standard deviation parameter; S4-3-5: Repeat step S3-3-4, update the heatmap matrix for b∈FocusObjects, and obtain the gaze heatmap Heatmap in (x,y,z); S4-3-6: Output gaze heatmap in (x,y,z); S4-4: Calculate gaze heatmap loss. The specific steps are as follows: S4-4-1: Input the real gaze heat map S4-4-2: Calculate gaze heatmap loss: Where N is the total number of pixels in the heat map.

7. The method for detecting driving neglect risk based on in-vehicle and out-vehicle environment matching according to claim 1, characterized in that: The step S6 of determining the risk alert specifically includes the following steps: S6-1: Use S2-4 to calculate the 3D in-vehicle coordinates of the risk target outside the vehicle S6-2: Use the model parameters of S3-4-3 to execute S3 and calculate the edge score of the gaze relationship graph model Score_e head,b : S6-3: Use the model parameters of S4-3-4 to execute S4 and generate a gaze heatmap in (x,y,z); S6-4: Calculate target perception score using S5: S6-4-1: Use S5-2 to convert the gaze heat map to the vehicle's external coordinate system to obtain the Heatmap out (x,y,z); S6-4-2: Use S5-3 to calculate the target perception score score_risk of the target outside the vehicle and the driver's gaze point b ; S6-5: Output alarm signal and make risk assessment for each target b outside the vehicle: S6-5-1: If Risk_pred b =0, indicating that target b is a non-risk target and does not trigger an alarm; S6-5-2: If Risk_pred b =1, and score_risk b If it is greater than the threshold δ, it means that the target b is a risk target, and the matching degree between the external target b and the driver’s gaze point is high, that is, the driver notices the risk target and does not trigger the alarm; S6-5-3: If Risk_pred b =1, and score_risk b If it is less than the threshold δ, it means that the target b is a risky target, but the matching degree between the external target b and the driver’s gaze point is low, that is, the driver ignores the risky target, triggering an alarm.

Images