Driving neglect risk detection method based on in-vehicle and out-vehicle environment matching

Abstract

The invention discloses a driving neglect risk detection method based on in-vehicle and out-vehicle matching, and relates to the field of driving safety. Due to the fact that staring information of a driver and external environment data are difficult to fuse accurately in the prior art, the method of the invention constructs a staring cone of the driver by converting the external environment target data into an in-vehicle coordinate system so as to generate a more accurate in-vehicle staring heat map. In order to improve the matching precision of the in-vehicle and out-vehicle environment data, the method integrates the out-vehicle target and the internal staring information, and realizes the accurate matching of the two and the generation of the heat map. By adding the data conversion and matching module in the existing driving risk assessment model, the method of the invention significantly improves the accuracy of driving risk detection, can be applied to a driving assistance system, and improves the overall driving safety.

Description

Technical Field

[0001] The present invention relates to the field of driving safety, and in particular to a method for detecting driving neglect risks based on matching of in-vehicle and out-vehicle environments. Background Art

[0002] With the development of autonomous driving technology, driving safety has become a crucial research direction. Existing technologies mainly focus on the detection and analysis of the external environment, such as obstacle recognition, lane departure detection, etc. However, these technologies usually ignore factors inside the car, such as the driver's attention and gaze direction, resulting in the inability to accurately assess driving risks at critical moments. In addition, although some systems attempt to combine internal and external data for risk assessment, there are still major challenges in the accuracy and real-time performance of data fusion and matching. The present invention proposes a new method that integrates the gaze data inside the car and the external environmental detection data, which can more comprehensively assess driving risks, improve the detection accuracy and response speed of the system, and thus improve driving safety.

[0003] Chinese patent application publication number CN 114026611 A "Detecting driver attention using heat maps" proposes a driver attention detection method based on heat map analysis. This method generates a reference heat map by collecting vehicle data and scene information, and captures the driver's gaze data through the driver distraction system to generate a gaze heat map. However, the in-vehicle driver data and the out-of-vehicle environment data mentioned in the patent are processed independently, without considering the mapping of external targets to in-vehicle images and the mapping of in-vehicle heat maps to external images. The generated in-vehicle heat map cannot accurately reflect the actual driving scene. This method lacks a direct connection between the driver's attention distribution and the actual external environment, which is likely to lead to deviations in the gaze heat map.

[0004] Chinese patent application publication number CN 115082895 A "A method for intelligent vehicle visual target detection based on attention neural network" This patent provides an intelligent vehicle visual target detection method based on attention neural network. The patent contains driver images and images outside the vehicle, but does not consider the mapping of external targets to images inside the vehicle, which will lead to errors in the gaze heat map inside the vehicle.

[0005] However, in the detection of driving neglect risk based on the matching of the in-car and out-car environment, we need to fully consider the problem of combining the in-car driver data and the out-car environment data. We construct a gaze relationship graph model by mapping the out-car target to the in-car coordinate system, thereby constructing a more accurate gaze heat map. Not only that, we convert the in-car gaze heat map to the out-car coordinate system and match it with the out-car target. In this way, we can more accurately detect the situation where the driver ignores the risk outside the car, and have a stronger representation ability when detecting the driver's driving risk. Summary of the invention

[0006] The purpose of the present invention is to make up for the defects of the existing technology and provide a driving neglect risk detection method based on the matching of the in-vehicle and out-vehicle environments. The present invention constructs an accurate gaze cone and generates a more accurate gaze heat map by converting the target outside the vehicle into the in-vehicle coordinate system. First, the target data in the out-vehicle environment is obtained and converted to the in-vehicle coordinate system. Then, these data are combined with the driver's gaze information to generate a gaze heat map, and a risk perception model that matches the in-vehicle and out-vehicle environments is constructed and trained. Finally, the risk perception model is used to obtain the risk perception score and evaluate the driving risk based on the matching of the in-vehicle and out-vehicle environments. If the threshold is exceeded, a warning is issued to remind the driver of safety hazards.

[0007] The present invention is implemented by the following technical solution: a method for detecting driving neglect risk based on matching of the in-vehicle and out-vehicle environments, specifically comprising the following steps:

[0008] S1: define the coordinate system and dataset;

[0009] S2: Detect the environment outside the vehicle;

[0010] S3: Construct gaze relationship graph model;

[0011] S4: Calculate the gaze heat map inside the car;

[0012] S5: Matching the inside and outside environment of the car;

[0013] S6: Determine risk alert;

[0014] The specific steps of defining the coordinate system and data set in step S1 are as follows:

[0015] S1-1: Define the in-vehicle coordinate system S in = {O in , X in , Y in , Z in};

[0016] S1-1-1: Define the origin O in is the center of the camera inside the car;

[0017] S1-1-2: Define the X-axis of the coordinate system (X in ) is the horizontal axis;

[0018] S1-1-3: Define the Y axis of the coordinate system (Y in ) is the vertical axis;

[0019] S1-1-4: Define the Z axis of the coordinate system (Z in ) is the depth axis;

[0020] S1-2: Define the vehicle's external coordinate system S out = {O out , X out , Y out , Z out}:

[0021] S1-2-1: Define the origin O out is the center of the camera outside the vehicle;

[0022] S1-2-2: Define the X-axis of the coordinate system (X out ) is the horizontal axis;

[0023] S1-2-3: Define the Y axis of the coordinate system (Y out ) is the vertical axis;

[0024] S1-2-4: Define the Z axis of the coordinate system (Z out ) is the depth axis, pointing to the end of the road;

[0025] S1-3: Define the in-vehicle / out-vehicle image coordinate system S image :

[0026] S1-3-1: Define the origin O image is the upper left corner vertex of the inside / outside image;

[0027] S1-3-2: Define the X-axis of the coordinate system (X image ) is the horizontal axis;

[0028] S1-3-3: Define the Y axis of the coordinate system (Y image ) is the vertical axis;

[0029] S1-4: Define the data set. The specific steps are as follows:

[0030] S1-4-1: Define the driver head dataset {Driver t}: Use the RGB-D camera to obtain the driver's head video frame, denoted as Driver t :

[0031]

[0032] in,

[0033] Driver t is the RGBD video frame at time t, t = 1, 2, ..., T

[0034] is the coordinate of the pixel point;

[0035] RGB_in n is the RGB color information of the pixel;

[0036] depth_in n is the depth value of the pixel;

[0037] N is the total number of pixels in the video frame;

[0038] S1-4-2: Define the off-vehicle dataset {F t}: Use the RGB-D camera to obtain the RGB-D video frame outside the car, denoted as F t :

[0039]

[0040] Among them, F t is the RGBD video frame at time t, t=1,2,...,T;

[0041] is the coordinate of the pixel point;

[0042] RGB_out n is the RGB color information of the pixel;

[0043] depth_out n is the depth value of the pixel;

[0044] N is the total number of pixels in the video frame;

[0045] S1-4-3: Define the real gaze heat map dataset as

[0046] The specific steps of detecting the vehicle exterior environment in step S2 are as follows:

[0047] S2-1: Extracting targets outside the vehicle:

[0048] S2-1-1: Input video frame F t , using the DETR (Detection Transformer) model, extract the set of external targets O t ={object b |b=1,2,...,B}, where t is the time, b is the index of the target, and B is the total number of targets;

[0049] S2-1-2: Each target object b is a rectangular area whose upper left corner vertex is The lower right vertex is is the X-axis external image coordinate of the upper left corner of the target, is the Y-axis external image coordinate of the upper left corner of the target; is the X-axis external image coordinate of the lower right corner of the target, is the Y-axis external image coordinate of the lower right corner of the target;

[0050] S2-2: Calculate the 3D in-vehicle coordinates of the external target. The specific steps are as follows:

[0051] S2-2-1: Calculate the average depth of the target,

[0052]

[0053] Among them, |object b | is the target object b The total number of pixels in depth_out n is the depth value of the pixel, and n is the index of the pixel;

[0054] S2-2-2: Calculate the 2D external image coordinates of the target center:

[0055] S2-2-2-1: Enter the top left corner of the target Enter the target lower right corner vertex

[0056] S2-2-2-2: Calculate the 2D external image coordinates of the target center

[0057] S2-2-3: Use the camera internal parameters to convert the 2D vehicle external image coordinates Transform to the vehicle external coordinate system S out , get the 3D external coordinates of the target center

[0058]

[0059] Among them, d b is the average depth of the target, R image_to_out is the rotation matrix. The two constitute the internal parameters used to transform from the external image coordinate system to the external coordinate system;

[0060] S2-2-4: Use external parameters to convert the center of the target outside the vehicle from the external coordinate system S out Transform to the in-vehicle coordinate system S in , get the 3D in-vehicle coordinates of the target center

[0061]

[0062] Among them, R out_to_in is the rotation matrix, T out_to_in is the translation vector, and the two constitute the external parameters, which are used to transform from the external coordinate system to the internal coordinate system;

[0063] S2-3: Define risk types:

[0064] S2-3-1: Define risk type 1 as the risk of the vehicle ahead being too close;

[0065] S2-3-2: Risk type 2 is defined as the risk of the vehicle ahead changing lanes sideways;

[0066] S2-3-3: Define risk type 3 as the risk of the vehicle drifting from the lane;

[0067] S2-4: Calculate the risk value of dangerous targets. The specific steps are as follows

[0068] S2-4-1: Calculate the risk value of the vehicle ahead being too close:

[0069] S2-4-1-1: Input a single frame of vehicle exterior image F t , extract the depth value of the vehicle, extract the depth value of the front vehicle, and calculate the depth distance Dz b，t ;

[0070] S2-4-1-2: Calculate the relative speed between the vehicle and the preceding vehicle. The specific steps are as follows:

[0071] S2-4-1-2-1: Input two consecutive frames of vehicle exterior images F t 、F t-1 ;

[0072] S2-4-1-2-2: Calculate the Y-axis coordinate of the center of the vehicle in each frame, and calculate the Y-axis coordinate of the center of the front vehicle in each frame;

[0073] S2-4-1-2-3: Calculate the Y-axis coordinate difference ΔY between the two in each frame t , ΔY t-1 , calculate the time difference Δt;

[0074] S2-4-1-2-4: Calculate the relative speed between the vehicle and the preceding vehicle.

[0075]

[0076] S2-4-1-3: Calculate the risk value of the vehicle ahead being too close.

[0077]

[0078] Among them, the depth distance Dz b，t The larger the relative speed V_rel, the smaller the risk. b，t The bigger it is, the greater the risk;

[0079] S2-4-2: Calculate the risk value of the vehicle ahead changing lanes sideways:

[0080] S2-4-2-1: Input a single frame of vehicle exterior image F t, extract the X-axis coordinate of the vehicle, extract the X-axis coordinate of the front vehicle, and calculate the horizontal distance Dx between the vehicle and the front vehicle b，t ;

[0081] S2-4-2-2: Input a single frame of vehicle exterior image F t , extract the depth value of the vehicle, extract the depth value of the front vehicle, and calculate the depth distance Dz between the vehicle and the front vehicle b，t ;

[0082] S2-4-2-3: Calculate the lane-changing speed of the vehicle ahead. The specific steps are as follows:

[0083] S2-4-2-3-1: Input two consecutive frames of vehicle exterior images F t 、F t-1 ;

[0084] S2-4-2-3-2: Calculate the center X-axis coordinate of the front vehicle in each frame

[0085] S2-4-2-3-3: Calculate the lane-changing speed of the vehicle ahead.

[0086]

[0087] S2-4-2-4: Calculate the risk value of the vehicle ahead changing lanes sideways.

[0088]

[0089] Among them, the lane change speed of the front vehicle V_change b , the larger the t, the greater the risk, the horizontal distance Dx b , the larger the t, the smaller the risk; the depth distance Dz b,t The larger the value, the smaller the risk. change ,λ z is the weight,

[0090] S2-4-3: Calculate the risk value of the vehicle deviating from the lane:

[0091] S2-4-3-1: Calculate the speed of the offset lane. The specific steps are as follows:

[0092] S2-4-3-1-1: Input two consecutive frames of vehicle exterior images F t 、F t-1 ;

[0093] S2-4-3-1-2: Calculate the lane offset distance D_lane for each frame b,t and Délane b,t-1 ;

[0094] S2-4-3-1-3: Calculate the speed of the offset lane,

[0095]

[0096] S2-4-3-2: Calculate the risk value of the vehicle's lane deviation.

[0097] Risk_deviation b =V_deviation b , t

[0098] Among them, the deviation lane speed V_deviation b ,The larger t is, the greater the risk is;

[0099] S2-4-4: Calculate the total risk value of each target:

[0100] Risk_max b =max(Risk_change b , Risk_deviation b ,Risk_s b )

[0101] S2-4-5: Enter each target object b The risk value of b If it is greater than the threshold, the target b is judged as a risk target and its risk flag Risk_pred is set. b is 1, otherwise it is 0;

[0102] The specific steps of constructing the gaze relationship graph model in step S3 are as follows:

[0103] S3-1: Input driver head video frame Driver t ;

[0104] S3-2: Calculate the node features of the graph model. The specific steps are as follows:

[0105] S3-2-1: input the driver's head image block and calculate the driver's head features;

[0106] S3-2-1-1: Calculate the 3D in-vehicle coordinates of the center point of the driver's head

[0107] S3-2-1-1-1: Use the YOLO model to calculate the 2D in-car image coordinates of the center point of the driver’s head.

[0108] S3-2-1-1-2: From Driver t Extract the depth depth_in of the center point of the driver's head head ;

[0109] S3-2-1-1-3: Use the camera internal reference to convert the 2D in-car image coordinates Transform to the in-vehicle coordinate system S in , get the 3D in-car coordinates of the center point of the driver's head

[0110]

[0111] Among them, R image_to_in is the rotation matrix. The two constitute the internal parameters used to transform from the in-vehicle image coordinate system to the in-vehicle coordinate system;

[0112] S3-2-1-2: Use the ResNet50 network to calculate the head appearance feature A head ;

[0113] S3-2-1-3: Calculate the driver's head features,

[0114]

[0115] S3-2-2: Input the image block of the target outside the vehicle and calculate the features of the target b outside the vehicle;

[0116] S3-2-2-1: Calculate the 3D external coordinates of the external target b

[0117] S3-2-2-2: Convert the external target b to the internal coordinate system, and get

[0118] S3-2-2-3: Use the ResNet50 network to calculate the appearance feature A_object of the target b outside the vehicle b ;

[0119] S3-2-2-4: Calculate the features of the target b outside the vehicle

[0120] S3-2-3: Construct node set V = {F head}∪{F_object b}, b = 1, 2, ..., B;

[0121] S3-3: Constructing the relationship edge features of the graph model:

[0122] S3-3-1: Combine the driver's head features and the features of the target outside the vehicle to obtain preliminary features:

[0123] F head，b =concat(F head , F_object b )

[0124] S3-3-2: Calculate the edge features of the graph model,

[0125] E head，b =sigmoid(W edge ·F head，b +bias edge )

[0126] Among them, sigmoid is the activation function, W edge is the weight matrix of the fully connected layer, bias edge is the bias vector of the fully connected layer;

[0127] S3-3-3: Construct edge set

[0128] S3-4: Building a gaze relationship graph model

[0129] S3-5: Calculate the edge score of the graph model. The specific steps are as follows;

[0130] S3-5-1: Input edge feature E _ head b ;

[0131] S3-5-2: Calculate edge score,

[0132] Score_e head，b =sigmoid(W score ·E head，b +bias score )

[0133] Among them, sigmoid is the activation function, W score The weight matrix for score calculation, bias score is the bias vector;

[0134] S3-5-3: Output edge score sequence {Score_e head，b};

[0135] The specific steps of constructing the gaze heat map in step S4 are as follows:

[0136] S4-1: Calculate gaze direction:

[0137] S4-1-1: Sort the edge score sequence and select the edge with the highest score.

[0138] [b * ,Socre_max]=argmax b ({Score_e head，b})

[0139] Then define the target point outside the vehicle with the highest score as gaze = b * , the driver's gaze point is

[0140] S4-1-2: Calculate gaze direction

[0141] S4-2: Construct the gaze cone. The specific steps are as follows:

[0142] S4-2-1: Set the vertex of the gaze cone to the center of the driver's head

[0143] S4-2-2: Set the direction of the gaze cone to

[0144] S4-2-3: Set the vertex angle of the gaze cone to θ (0<θ<π). A diameter of the bottom surface of the cone and the vertex form a triangle, and the vertex angle of the triangle is the vertex angle of the cone.

[0145] S4-2-4: Set the maximum gaze distance of the gaze cone to h, which represents the effective range of the driver's line of sight;

[0146] S4-2-5: Generate gaze cone

[0147]

[0148] Where (x, y, z) is a point in the three-dimensional space in the vehicle's internal coordinate system;

[0149] From point (x, y, z) to the vertex of the cone A vector of

[0150] From point (x, y, z) to the vertex of the cone The vector and gaze direction The dot product of , which is used to calculate the angle between two vectors;

[0151] Calculate the distance from point (x, yz) to vertex The vector and gaze direction The cosine value of the angle between is used to determine whether the point is within the cone;

[0152] From point (x, y, z) to the vertex of the cone The distance Used to verify the point (x, y, z) to the cone vertex The distance is between 0 and the maximum gaze distance h;

[0153] S4-3: Generate gaze heat map. The specific steps are as follows:

[0154] S4-3-1: Input the set of targets in the image outside the vehicle

[0155] S4-3-2: Calculate the gaze target set within the gaze cone Region.

[0156]

[0157] S4-3-3: Initialize gaze heatmap, Heatmap in (x, y, z) = 0, (x, y, z) is the coordinate inside the vehicle;

[0158] S4-3-4: Calculate the gaze value of each gaze target,

[0159]

[0160] Among them, b∈FocusObjects, is the driver’s gaze point, σ is a Gaussian standard deviation parameter;

[0161] S4-3-5: Repeat step S3-3-4, update the heatmap matrix for b∈FocusObjects, and obtain the gaze heatmap Heatmap in (x, y, z);

[0162] S4-3-6: Output gaze heatmap in (x, y, z);

[0163] S4-4: Calculate gaze heatmap loss. The specific steps are as follows:

[0164] S4-4-1: Input the real gaze heat map

[0165] S4-4-2: Calculate gaze heatmap loss:

[0166]

[0167] Where N is the total number of pixels in the heat map;

[0168] The specific steps of the vehicle interior and exterior environment matching described in step S5 are as follows:

[0169] S5-1: Input the in-car gaze heatmap in (x, y, z);

[0170] S5-2: Convert Heatmap in (x, y, z) to the vehicle's external coordinate system to get the Heatmap out (x, y, z), the specific steps are:

[0171] Using external parameters, Heatmap in Each point (x, y, z) is calculated from the in-vehicle coordinate system S in Transform to the vehicle external coordinate system S out :

[0172]

[0173] Among them, R in_to_out is the rotation matrix, T in_to_out is the translation vector, which constitutes the external parameter and is used to transform from the internal coordinate system to the external coordinate system;

[0174] S5-3: Calculate the target perception score. The specific steps are as follows:

[0175] S5-3-1: Input each external target point

[0176] S5-3-2: Calculate the target perception score for each target outside the vehicle:

[0177]

[0178] S5-4: Calculate risk perceived loss:

[0179]

[0180] Among them, B is the total number of targets outside the vehicle; Risk_pred b is the risk mark of the external object b, Risk_pred b A value of 1 indicates a risk target, Risk_pred b A value of 0 indicates a non-risk target;

[0181] S5-5: Calculate the total loss:

[0182] L total =L gaze +L risk

[0183] Among them, L gaze Obtained from S4-4-2;

[0184] S5-6: Loss of use L total , perform model training to obtain the optimal parameters, which include the parameters in S3-4-3 and S4-3-4;

[0185] The specific steps of determining the risk warning in step S6 are as follows:

[0186] S6-1: Use S2-4 to calculate the 3D in-vehicle coordinates of the risk target outside the vehicle

[0187] S6-2: Use the model parameters of S3-4-3 to execute S3 and calculate the edge score of the gaze relationship graph model

[0188] Score_e head，b :

[0189] S6-3: Use the model parameters of S4-3-4 to execute S4 and generate a gaze heatmap in (x, y, z);

[0190] S6-4: Calculate target perception score using S5:

[0191] S6-4-1: Use S5-2 to convert the gaze heat map to the vehicle's external coordinate system to obtain the Heatmap out (x, y, z);

[0192] S6-4-2: Use S5-3 to calculate the target perception score score_risk of the target outside the vehicle and the driver's gaze point b ;

[0193] S6-5: Output an alarm signal, and make the following judgments for each external target b in turn:

[0194] S6-5-1: If Risk_pred b =0, indicating that target b is a non-risk target and does not trigger an alarm;

[0195] S6-5-2: If Risk_pred b =1, and score_risk b If it is greater than the threshold δ, it means that the target b is a risk target, and the matching degree between the external target b and the driver’s gaze point is high, that is, the driver notices the risk target and does not trigger the alarm;

[0196] S6-5-3: If Risk_pred b =1, and score_risk b If it is less than the threshold δ, it means that the target b is a risky target, but the matching degree between the external target b and the driver’s gaze point is low, that is, the driver ignores the risky target, triggering an alarm.

[0197] The advantages of the present invention are as follows: the present invention converts the target data of the external environment into the internal coordinate system of the vehicle to construct the gaze cone of the driver to generate a more accurate internal gaze heat map; in order to improve the accuracy of matching the internal and external environment data, the present invention integrates the external target and internal gaze information to achieve accurate matching and heat map generation between the two; the present invention significantly improves the accuracy of driving risk detection by adding the above-mentioned data conversion and matching modules to the existing driving risk assessment model, and can be applied to the driving assistance system to improve the overall driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0198] Figure 1 A flow chart for defining a coordinate system and a data set as mentioned in the present invention;

[0199] Figure 2 This is a flow chart of detecting the vehicle exterior environment mentioned in the present invention;

[0200] Figure 3 A flowchart of constructing a gaze relationship graph model in a vehicle mentioned in the present invention;

[0201] Figure 4 A flow chart of calculating the in-vehicle gaze heat map mentioned in the present invention;

[0202] Figure 5 This is a flow chart of matching the in-vehicle and out-vehicle environments mentioned in the present invention;

[0203] Figure 6 This is a flow chart for determining risk warnings mentioned in the present invention;

[0204] Figure 7 The figure is a flowchart of an implementation of an embodiment of the present invention. DETAILED DESCRIPTION

[0205] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments;

[0206] like Figure 7 As shown, a method for detecting driving neglect risk based on matching of the in-vehicle and out-vehicle environments specifically includes the following steps:

[0207] S1: Define the coordinate system required in the implementation steps. The specific steps are as follows:

[0208] S1-1: Define the in-vehicle coordinate system S in = {O in , X in , Y in , Z in},like Figure 1 As shown;

[0209] S1-1-1: Define the origin Oin is the center of the camera inside the car;

[0210] S1-1-2: Define the X-axis of the coordinate system (X in ) is the horizontal axis;

[0211] S1-1-3: Define the Y axis of the coordinate system (Y in ) is the vertical axis;

[0212] S1-1-4: Define the Z axis of the coordinate system (Z in ) is the depth axis;

[0213] S1-2: Define the vehicle's external coordinate system S out = {O out , X out , Y out , Z out},like Figure 1 As shown:

[0214] S1-2-1: Define the origin O out is the center of the camera outside the vehicle, such as Figure 1 As shown;

[0215] S1-2-2: Define the X-axis of the coordinate system (X out ) is the horizontal axis;

[0216] S1-2-3: Define the Y axis of the coordinate system (Y out ) is the vertical axis;

[0217] S1-2-4: Define the Z axis of the coordinate system (Z out ) is the depth axis, pointing to the end of the road;

[0218] S1-3: Define the in-vehicle / out-vehicle image coordinate system S image ,like Figure 1 As shown:

[0219] S1-3-1: Define the origin O image is the upper left corner vertex of the inside / outside image;

[0220] S1-3-2: Define the X-axis of the coordinate system (X image ) is the horizontal axis;

[0221] S1-3-3: Define the Y axis of the coordinate system (Y image ) is the vertical axis;

[0222] S1-4: Define the data set. The specific steps are as follows:

[0223] S1-4-1: Define the driver head dataset {Driver t}: Use the RGB-D camera to obtain the driver's head video frame, denoted as Driver t :

[0224]

[0225] in,

[0226] Driver t is the RGBD video frame at time t, t = 1, 2, ..., T

[0227] is the coordinate of the pixel point;

[0228] RGB_in n is the RGB color information of the pixel;

[0229] depth_in n is the depth value of the pixel;

[0230] N is the total number of pixels in the video frame;

[0231] S1-4-2: Define the off-vehicle dataset {F t}: Use the RGB-D camera to obtain the RGB-D video frame outside the car, denoted as F t :

[0232]

[0233] Among them, F t is the RGBD video frame at time t, t=1,2,...,T;

[0234] is the coordinate of the pixel point;

[0235] RGB_out n is the RGB color information of the pixel;

[0236] depth_out n is the depth value of the pixel;

[0237] N is the total number of pixels in the video frame;

[0238] S1-4-3: Define the real gaze heat map dataset as

[0239] S2: Detect the external environment of the vehicle, such as Figure 2 As shown, the specific steps are as follows;

[0240] S2-1: Extracting targets outside the vehicle. The specific steps are as follows:

[0241] S2-1-1: Input video frame F t, using the DETR (Detection Transformer) model, extract the set of external targets O t ={object b |b=1,2,...,B}, where t is the time, b is the index of the target, and B is the total number of targets;

[0242] S2-1-2: Each target object b is a rectangular area whose upper left corner vertex is The lower right vertex is is the X-axis external image coordinate of the upper left corner of the target, is the Y-axis external image coordinate of the upper left corner of the target; is the X-axis external image coordinate of the lower right corner of the target, is the Y-axis external image coordinate of the lower right corner of the target;

[0243] S2-2: Calculate the 3D in-vehicle coordinates of the external target. The specific steps are as follows:

[0244] S2-2-1: Calculate the average depth of the target,

[0245]

[0246] Among them, |object b | is the target object b The total number of pixels in depth_out n is the depth value of the pixel, and n is the index of the pixel;

[0247] S2-2-2: Calculate the 2D external image coordinates of the target center:

[0248] S2-2-2-1: Enter the top left corner of the target Enter the target lower right corner vertex

[0249] S2-2-2-2: Calculate the 2D external image coordinates of the target center

[0250] S2-2-3: Use the camera internal parameters to convert the 2D vehicle external image coordinates Transform to the vehicle external coordinate system S out , get the 3D external coordinates of the target center

[0251]

[0252] Among them, d b is the average depth of the target, R image_to_outis the rotation matrix. The two constitute the internal parameters used to transform from the external image coordinate system to the external coordinate system;

[0253] S2-2-4: Use external parameters to convert the center of the target outside the vehicle from the external coordinate system S out Transform to the in-vehicle coordinate system S in , get the 3D in-vehicle coordinates of the target center

[0254]

[0255] Among them, R out_to_in is the rotation matrix, T out_to_in is the translation vector, and the two constitute the external parameters, which are used to transform from the external coordinate system to the internal coordinate system;

[0256] S2-3: Define risk types:

[0257] S2-3-1: Define risk type 1 as the risk of the vehicle ahead being too close;

[0258] S2-3-2: Risk type 2 is defined as the risk of the vehicle ahead changing lanes sideways;

[0259] S2-3-3: Define risk type 3 as the risk of the vehicle drifting from the lane;

[0260] S2-4: Calculate the risk value of dangerous targets. The specific steps are as follows

[0261] S2-4-1: Calculate the risk value of the vehicle ahead being too close:

[0262] S2-4-1-1: Input a single frame of vehicle exterior image F t , extract the depth value of the vehicle, extract the depth value of the front vehicle, and calculate the depth distance Dz b，t ;

[0263] S2-4-1-2: Calculate the relative speed between the vehicle and the preceding vehicle. The specific steps are as follows:

[0264] S2-4-1-2-1: Input two consecutive frames of vehicle exterior images F t 、F t-1 ;

[0265] S2-4-1-2-2: Calculate the Y-axis coordinate of the center of the vehicle in each frame, and calculate the Y-axis coordinate of the center of the front vehicle in each frame;

[0266] S2-4-1-2-3: Calculate the Y-axis coordinate difference ΔY between the two in each frame t , ΔY t-1 , calculate the time difference Δt;

[0267] S2-4-1-2-4: Calculate the relative speed between the vehicle and the preceding vehicle.

[0268]

[0269] S2-4-1-3: Calculate the risk value of the vehicle ahead being too close.

[0270]

[0271] Among them, the depth distance Dz b，t The larger the relative speed V_rel, the smaller the risk. b，t The bigger it is, the greater the risk;

[0272] S2-4-2: Calculate the risk value of the vehicle ahead changing lanes sideways:

[0273] S2-4-2-1: Input a single frame of vehicle exterior image F t , extract the X-axis coordinate of the vehicle, extract the X-axis coordinate of the front vehicle, and calculate the horizontal distance Dx between the vehicle and the front vehicle b，t ;

[0274] S2-4-2-2: Input a single frame of vehicle exterior image F t , extract the depth value of the vehicle, extract the depth value of the front vehicle, and calculate the depth distance Dz between the vehicle and the front vehicle b，t ;

[0275] S2-4-2-3: Calculate the lane-changing speed of the vehicle ahead. The specific steps are as follows:

[0276] S2-4-2-3-1: Input two consecutive frames of vehicle exterior images F t 、F t-1 ;

[0277] S2-4-2-3-2: Calculate the center X-axis coordinate of the front vehicle in each frame

[0278] S2-4-2-3-3: Calculate the lane-changing speed of the vehicle ahead.

[0279]

[0280] S2-4-2-4: Calculate the risk value of the vehicle ahead changing lanes sideways.

[0281]

[0282] Among them, the lane change speed of the front vehicle V_change b，t The larger the value, the greater the risk, and the horizontal distance Dx b，t The larger the risk, the lower the depth distance Dz b，t The larger the value, the smaller the risk. change ,λ z is the weight,

[0283] S2-4-3: Calculate the risk value of the vehicle deviating from the lane:

[0284] S2-4-3-1: Calculate the speed of the offset lane. The specific steps are as follows:

[0285] S2-4-3-1-1: Input two consecutive frames of vehicle exterior images F t 、F t-1 ;

[0286] S2-4-3-1-2: Calculate the lane offset distance D_lane for each frame b，t and Délane b，t-1 ;

[0287] S2-4-3-1-3: Calculate the speed of the offset lane,

[0288]

[0289] S2-4-3-2: Calculate the risk value of the vehicle's lane deviation.

[0290] Risk_deviation b =V_deviation b，t

[0291] Among them, the deviation lane speed V_deviation b ,The larger t is, the greater the risk is;

[0292] S2-4-4: Calculate the total risk value of each target:

[0293] Risk_max b =max(Risk_change b , Risk_deviation b ,Risk_s b )

[0294] S2-4-5: Enter each target object b The risk value of b If it is greater than the threshold, the target b is judged as a risk target and its risk flag Risk_pred is set. b is 1, otherwise it is 0;

[0295] S3: Construct gaze relationship graph model, such as Figure 3 As shown, the specific steps are as follows:

[0296] S3-1: Input driver head video frame Driver t ;

[0297] S3-2: Calculate the node features of the graph model. The specific steps are as follows:

[0298] S3-2-1: input the driver's head image block and calculate the driver's head features;

[0299] S3-2-1-1: Calculate the 3D in-vehicle coordinates of the center point of the driver's head

[0300] S3-2-1-1-1: Use the YOLO model to calculate the 2D in-car image coordinates of the center point of the driver’s head.

[0301] S3-2-1-1-2: From Driver t Extract the depth depth_in of the center point of the driver's head head ;

[0302] S3-2-1-1-3: Use the camera internal reference to convert the 2D in-car image coordinates Transform to the in-vehicle coordinate system S in , get the 3D in-car coordinates of the center point of the driver's head

[0303]

[0304] Among them, R image_to_in is the rotation matrix. The two constitute the internal parameters used to transform from the in-vehicle image coordinate system to the in-vehicle coordinate system;

[0305] S3-2-1-2: Use the ResNet50 network to calculate the head appearance feature A head ;

[0306] S3-2-1-3: Calculate the driver's head features,

[0307]

[0308] S3-2-2: Input the image block of the target outside the vehicle and calculate the features of the target b outside the vehicle;

[0309] S3-2-2-1: Calculate the 3D external coordinates of the external target b

[0310] S3-2-2-2: Convert the external target b to the internal coordinate system, and get

[0311] S3-2-2-3: Use the ResNetS0 network to calculate the appearance feature A_object of the target b outside the vehicle b ;

[0312] S3-2-2-4: Calculate the features of the target b outside the vehicle

[0313] S3-2-3: Construct node set V = {F head}∪{F_object b}, b = 1, 2, ..., B;

[0314] S3-3: Constructing the relationship edge features of the graph model:

[0315] S3-3-1: Combine the driver's head features and the features of the target outside the vehicle to obtain preliminary features:

[0316] F head，b =concat(F head , F_object b )

[0317] S3-3-2: Calculate the edge features of the graph model,

[0318] E head，b =sigmoid(W edge ·F head，b +bias edge )

[0319] Among them, sigmoid is the activation function, W edge is the weight matrix of the fully connected layer, bias edge is the bias vector of the fully connected layer;

[0320] S3-3-3: Construct edge set

[0321] S3-4: Building a gaze relationship graph model

[0322] S3-5: Calculate the edge score of the graph model. The specific steps are as follows;

[0323] S3-5-1: Input edge feature E_head b ;

[0324] S3-5-2: Calculate edge score,

[0325] Score_e head，b =sigmoid(W score ·E head，b +bias score )

[0326] Among them, sigmoid is the activation function, W score The weight matrix for score calculation, bias score is the bias vector;

[0327] S3-5-3: Output edge score sequence {Score_e head，b};

[0328] S4: Construct gaze heatmap, such as Figure 4 As shown, the specific steps are as follows:

[0329] S4-1: Calculate gaze direction:

[0330] S4-1-1: Sort the edge score sequence and select the edge with the highest score.

[0331] [b * ,Socre_max]=argmax b ({Score_e head，b})

[0332] Then define the target point outside the vehicle with the highest score as gaze = b * , the driver's gaze point is

[0333] S4-1-2: Calculate gaze direction

[0334] S4-2: Construct the gaze cone. The specific steps are as follows:

[0335] S4-2-1: Set the vertex of the gaze cone to the center of the driver's head

[0336] S4-2-2: Set the direction of the gaze cone to

[0337] S4-2-3: Set the vertex angle of the gaze cone to θ (0<θ<π), such as Figure 4 As shown, a diameter of the base of the cone is connected to the vertex to form a triangle, and the vertex angle of the triangle is the vertex angle of the cone;

[0338] S4-2-4: Set the maximum gaze distance of the gaze cone to h, which represents the effective range of the driver's line of sight;

[0339] S4-2-5: Generate gaze cone

[0340]

[0341] Where (x, y, z) is a point in the three-dimensional space in the vehicle's internal coordinate system;

[0342] From point (x, y, z) to the vertex of the cone A vector of

[0343] From point (x, y, z) to the vertex of the cone The vector and gaze direction The dot product of , which is used to calculate the angle between two vectors;

[0344] Calculate the distance from point (x, y, z) to vertex The vector and gaze direction The cosine value of the angle between is used to determine whether the point is within the cone;

[0345] From point (x, y, z) to the vertex of the cone The distance Used to verify the point (x, y, z) to the cone vertex The distance is between 0 and the maximum gaze distance h;

[0346] S4-3: Generate gaze heat map. The specific steps are as follows:

[0347] S4-3-1: Input the set of targets in the image outside the vehicle

[0348] S4-3-2: Calculate the gaze target set within the gaze cone Region.

[0349]

[0350] S4-3-3: Initialize gaze heat map, Heatmapin(x, y, z) = 0, (x, y, z) is the coordinate inside the vehicle;

[0351] S4-3-4: Calculate the gaze value of each gaze target,

[0352]

[0353] Among them, b∈FocusObjects, is the driver’s gaze point, σ is a Gaussian standard deviation parameter;

[0354] S4-3-5: Repeat step S3-3-4, update the heatmap matrix for b∈FocusObjects, and obtain the gaze heatmap Heatmap in (x, y, z);

[0355] S4-3-6: Output gaze heatmap in (x, y, z);

[0356] S4-4: Calculate gaze heatmap loss. The specific steps are as follows:

[0357] S4-4-1: Input the real gaze heat map

[0358] S4-4-2: Calculate gaze heatmap loss:

[0359]

[0360] Where N is the total number of pixels in the heat map;

[0361] S5: Matching of the in-vehicle and out-vehicle environments, such as Figure 5 As shown, the specific steps are as follows:

[0362] S5-1: Input the in-car gaze heatmap in (x, y, z);

[0363] S5-2: Convert Heatmap in (x, y, z) to the vehicle's external coordinate system to get the Heatmap out (x, y, z), the specific steps are:

[0364] Using external parameters, Heatmap in Each point (x, y, z) is calculated from the in-vehicle coordinate system S in Transform to the vehicle external coordinate system S out :

[0365]

[0366] Among them, R in_to_out is the rotation matrix, T in_to_out is the translation vector, which constitutes the external parameter and is used to transform from the internal coordinate system to the external coordinate system;

[0367] S5-3: Calculate the target perception score. The specific steps are as follows:

[0368] S5-3-1: Input each external target point

[0369] S5-3-2: Calculate the target perception score for each target outside the vehicle:

[0370]

[0371] S5-4: Calculate risk perceived loss:

[0372]

[0373] Among them, B is the total number of targets outside the vehicle; Risk_pred b is the risk mark of the external object b, Risk_predb A value of 1 indicates a risk target, Risk_pred b A value of 0 indicates a non-risk target;

[0374] S5-5: Calculate the total loss:

[0375] L total =L gaze +L risk

[0376] Among them, L gaze Obtained from S4-4-2;

[0377] S5-6: Loss of use L total , perform model training to obtain the optimal parameters, which include the parameters in S3-4-3 and S4-3-4;

[0378] S6: Judge the risk warning, e.g. Figure 6 As shown, the specific steps are as follows:

[0379] S6-1: Use S2-4 to calculate the 3D in-vehicle coordinates of the risk target outside the vehicle

[0380] S6-2: Use the model parameters of S3-4-3 to execute S3 and calculate the edge score of the gaze relationship graph model

[0381] Score_e head，b :

[0382] S6-3: Use the model parameters of S4-3-4 to execute S4 and generate a gaze heatmap in (x, y, z);

[0383] S6-4: Calculate target perception score using S5:

[0384] S6-4-1: Use S5-2 to convert the gaze heat map to the vehicle's external coordinate system to obtain the Heatmap out (x, y, z);

[0385] S6-4-2: Use S5-3 to calculate the target perception score score_risk of the target outside the vehicle and the driver's gaze point b ;

[0386] S6-5: Output an alarm signal, and make the following judgments for each external target b in turn:

[0387] S6-5-1: If Risk_pred b =0, indicating that target b is a non-risk target and does not trigger an alarm;

[0388] S6-5-2: If Risk_pred b =1, and score_risk b If it is greater than the threshold δ, it means that the target b is a risk target, and the matching degree between the external target b and the driver’s gaze point is high, that is, the driver notices the risk target and does not trigger the alarm;

[0389] S6-5-3: If Risk_pred b =1, and score_risk b If it is less than the threshold δ, it means that the target b is a risky target, but the matching degree between the external target b and the driver’s gaze point is low, that is, the driver ignores the risky target, triggering an alarm.

Claims

1. A method for detecting driving neglect risk based on in-vehicle and out-vehicle environment matching, characterized in that: The specific steps include: S1: define the coordinate system and dataset; S2: Detect the environment outside the vehicle; S3: Construct gaze relationship graph model; S4: Calculate the gaze heat map inside the car; S5: Matching the inside and outside environment of the car; S6: Determine risk alert.

2. The method for detecting driving neglect risk based on in-vehicle and out-vehicle environment matching according to claim 1, characterized in that: The definition of the coordinate system and the data set described in step S1 specifically includes the following steps: S1-1: Define the in-vehicle coordinate system S in = {O in , X in , Y in , Z in }; where the origin O in is the center of the camera inside the car, and the X axis X in is the horizontal axis, Y axis in is the vertical axis, Z axis in is the depth axis; S1-2: Define the vehicle's external coordinate system S out = {O out , X out , Y out , Z out }: The origin O out is the center of the camera outside the vehicle, and the X axis X out is the horizontal axis; Y axis out is the vertical axis; Z axis out is the depth axis, pointing to the end of the road; S1-3: Define the in-vehicle / out-vehicle image coordinate system S image ; Define the origin O image The top left corner of the image inside or outside the car defines the X axis of the coordinate system. image Is the horizontal axis; defines the Y axis of the coordinate system image is the vertical axis; S1-4: Define the dataset: S1-4-1: Define the driver head dataset {Driver t }: Use the RGB-D camera to obtain the driver's head video frame, denoted as Driver t : in, Driver t is the RGBD video frame at time t, t = 1, 2, ..., T is the coordinate of the pixel point; RGB_in n is the RGB color information of the pixel; depth_in n is the depth value of the pixel; N is the total number of pixels in the video frame; S1-4-2: Define the off-vehicle dataset {F t }: Use the RGB-D camera to obtain the RGB-D video frame outside the car, denoted as F t : Among them, F t is the RGBD video frame at time t, t=1,2,...,T; is the coordinate of the pixel point; RGB_out n is the RGB color information of the pixel; depth_out n is the depth value of the pixel; N is the total number of pixels in the video frame; S1-4-3: Define the real gaze heat map dataset as 3. The method for detecting driving neglect risk based on in-vehicle and out-vehicle environment matching according to claim 2, characterized in that: The detection of the vehicle exterior environment described in step S2 specifically includes the following steps: S2-1: Extracting targets outside the vehicle: S2-1-1: Input video frame F t , using the DETR model, extract the set of external targets O t ={object b |b=1,2,...,B}, where t is the time, b is the index of the target, and B is the total number of targets; S2-1-2: Each target object b is a rectangular area whose upper left corner vertex is The lower right vertex is is the X-axis external image coordinate of the upper left corner of the target, is the Y-axis external image coordinate of the upper left corner of the target; is the X-axis external image coordinate of the lower right corner of the target, is the Y-axis external image coordinate of the lower right corner of the target; S2-2: Calculate the 3D in-vehicle coordinates of the external target; S2-3: Define risk types; S2-4: Calculate the risk value of dangerous targets.

4. The method for detecting driving neglect risk based on in-vehicle and out-vehicle environment matching according to claim 3, characterized in that: The step S2-2 of calculating the 3D in-vehicle coordinates of the external target specifically includes the following steps: S2-2-1: Calculate the average depth of the target, Among them, |object b | is the target object b The total number of pixels in depth_out n is the depth value of the pixel, and n is the index of the pixel; S2-2-2: Calculate the 2D external image coordinates of the target center: S2-2-2-1: Enter the top left corner of the target Enter the target lower right corner vertex S2-2-2-2: Calculate the 2D external image coordinates of the target center S2-2-3: Use the camera internal parameters to convert the 2D vehicle external image coordinates Transform to the vehicle external coordinate system S out , get the 3D external coordinates of the target center Among them, d b is the average depth of the target, R image_to_out is the rotation matrix. The two constitute the internal parameters used to transform from the external image coordinate system to the external coordinate system; S2-2-4: Use external parameters to convert the center of the target outside the vehicle from the external coordinate system S ou t is converted to the in-vehicle coordinate system S in , get the 3D in-vehicle coordinates of the target center Among them, R out_to_in is the rotation matrix, T out_to_in is the translation vector, and the two constitute the external parameters, which are used to transform from the external coordinate system to the internal coordinate system; The step S2-3 of defining the risk type specifically includes the following steps: S2-3-1: Define risk type 1 as the risk of the vehicle ahead being too close; S2-3-2: Define risk type 2 as the risk of the vehicle ahead changing lanes sideways; S2-3-3: Define risk type 3 as the risk of the vehicle drifting from the lane; The calculation of the risk value of the dangerous target described in step S2-4 specifically includes the following steps: S2-4-1: Calculate the risk value of the vehicle ahead being too close: S2-4-1-1: Input a single frame of vehicle exterior image F t , extract the depth value of the vehicle, extract the depth value of the front vehicle, and calculate the depth distance Dz b ,t; S2-4-1-2: Calculate the relative speed between the vehicle and the preceding vehicle. The specific steps are as follows: S2-4-1-2-1: Input two consecutive frames of vehicle exterior images F t 、F t-1 ; S2-4-1-2-2: Calculate the Y-axis coordinate of the center of the self-vehicle in each frame, and calculate the Y-axis coordinate of the center of the front vehicle in each frame; S2-4-1-2-3: Calculate the Y-axis coordinate difference ΔY between the two in each frame t , ΔV t-1 , calculate the time difference Δt; S2-4-1-2-4: Calculate the relative speed between the vehicle and the preceding vehicle. S2-4-1-3: Calculate the risk value of the vehicle ahead being too close. Among them, the depth distance Dz b，t The larger the relative speed V_rel, the smaller the risk. b，t The bigger it is, the greater the risk; S2-4-2: Calculate the risk value of the vehicle ahead changing lanes sideways: S2-4-2-1: Input a single frame of vehicle exterior image F t , extract the X-axis coordinate of the vehicle, extract the X-axis coordinate of the front vehicle, and calculate the horizontal distance Dx between the vehicle and the front vehicle b，t ; S2-4-2-2: Input a single frame of vehicle exterior image F t , extract the depth value of the vehicle, extract the depth value of the front vehicle, and calculate the depth distance Dz between the vehicle and the front vehicle b .t; S2-4-2-3: Calculate the lane-changing speed of the vehicle ahead. The specific steps are as follows: S2-4-2-3-1: Input two consecutive frames of vehicle exterior images F t 、F t-1 ; S2-4-2-3-2: Calculate the center X-axis coordinate of the front vehicle in each frame S2-4-2-3-3: Calculate the lane-changing speed of the vehicle ahead. S2-4-2-4: Calculate the risk value of the vehicle ahead changing lanes sideways. Among them, the lane change speed of the front vehicle V_change b，t The larger the value, the greater the risk, and the horizontal distance Dx b，t The larger the risk, the lower the depth distance Dz b，t The larger the value, the smaller the risk. change ,λ z is the weight; S2-4-3: Calculate the risk value of the vehicle deviating from the lane: S2-4-3-1: Calculate the speed of the offset lane. The specific steps are as follows: S2-4-3-1-1: Input two consecutive frames of vehicle exterior images F t 、F t-1 ; S2-4-3-1-2: Calculate the lane offset distance D_lane for each frame b,t and Délane b,t-1 ; S2-4-3-1-3: Calculate the speed of the offset lane, S2-4-3-2: Calculate the risk value of the vehicle's lane deviation. Risk_deviation b =V_deviation b,t Among them, the deviation lane speed V_deviation b ,The larger t is, the greater the risk is; S2-4-4: Calculate the total risk value of each target: Risk_max b =max(Risk_change b ,Risk_deviation b ,Risk_s b ) S2-4-5: Enter each target object b The risk value of b If it is greater than the threshold, the target b is judged as a risk target and its risk flag Risk_pred is set. b is 1, otherwise it is 0.

5. The method for detecting driving neglect risk based on in-vehicle and out-vehicle environment matching according to claim 4, characterized in that: The step S3 of constructing the gaze relationship graph model specifically includes the following steps: S3-1: Input driver head video frame Driver t ; S3-2: Node characteristics of the computational graph model; S3-3: Construct the relationship edge features of the graph model; S3-4: construct gaze relationship graph model G = (V, E); S3-5: Calculate edge scores of the graph model.

6. The method for detecting driving neglect risk based on in-vehicle and out-vehicle environment matching according to claim 5, characterized in that: The node features of the computational graph model described in step S3-2 specifically include the following steps: S3-2-1: input the driver's head image block and calculate the driver's head features; S3-2-1-1: Calculate the 3D in-vehicle coordinates of the center point of the driver's head; S3-2-1-1-1: Use the YOLO model to calculate the 2D in-car image coordinates of the center point of the driver’s head. get S3-2-1-1-2: From Driver t Extract the depth depth_in of the center point of the driver's head head ; S3-2-1-1-3: Use the camera internal reference to convert the 2D in-vehicle image coordinates Transform to the in-vehicle coordinate system S in , get the 3D in-car coordinates of the center point of the driver's head Among them, R image_to_in is the rotation matrix. The two constitute the internal parameters used to transform from the in-vehicle image coordinate system to the in-vehicle coordinate system; S3-2-1-2: Use the ResNet50 network to calculate the head appearance feature A head ; S3-2-1-3: Calculate the driver's head features, S3-2-2: Input the image block of the target outside the vehicle and calculate the features of the target b outside the vehicle; S3-2-2-1: Calculate the 3D external coordinates of the external target b S3-2-2-2: Convert the external target b to the internal coordinate system, and get S3-2-2-3: Use the ResNet50 network to calculate the appearance feature A_object of the target b outside the vehicle b ; S3-2-2-4: Calculate the features of the target b outside the vehicle S3-2-3: Construct node set V = {F head }∪{F{object b }, b = 1, 2, ..., B; The step S3-3 of constructing the relationship edge features of the graph model specifically includes the following steps: S3-3-1: Combine the driver's head features and the features of the target outside the vehicle to obtain preliminary features: F head，b =concat(F head ,F_object b ) S3-3-2: Calculate the edge features of the graph model, E head，b =sigmoid(W edge ·F head，b +bias edge ) Among them, sigmoid is the activation function, W edge is the weight matrix of the fully connected layer, bias edge is the bias vector of the fully connected layer; S3-3-3: Construct edge set E = {E head，b ); The calculation of the edge score of the graph model in step S3-5 specifically includes the following steps: S3-5-1: Input edge feature E{head b ; S3-5-2: Calculate edge score, Score_e head，b =sigmoid(W score ·E head，b +bias score ) Among them, sigmoid is the activation function, W score The weight matrix for score calculation, bias score is the bias vector; S3-5-3: Output edge score sequence {Score{e head，b }.

7. The method for detecting driving neglect risk based on in-vehicle and out-vehicle environment matching according to claim 6, characterized in that: The step S4 of calculating the in-vehicle gaze heat map specifically includes the following steps: S4-1: Calculate gaze direction; S4-2: Constructing gaze cones; S4-3: Generate gaze heatmap; S4-4: Compute gaze heatmap loss.

8. The method for detecting driving neglect risk based on in-vehicle and out-vehicle environment matching according to claim 7, characterized in that: The construction of the gaze cone described in step S4-2 specifically includes the following steps: S4-2-1: Set the vertex of the gaze cone to the center of the driver's head S4-2-2: Set the direction of the gaze cone to S4-2-3: Set the vertex angle of the gaze cone to θ, 0<θ<π; S4-2-4: Set the maximum gaze distance of the gaze cone to h, which represents the effective range of the driver's line of sight; S4-2-5: Generate gaze cone Among them, (x, y, z) is a point in the three-dimensional space in the vehicle's internal coordinate system; From point (x, y, z) to the vertex of the cone A vector of From point (x, y, z) to the vertex of the cone The vector and gaze direction The dot product of , which is used to calculate the angle between two vectors; Calculate the distance from point (x, y, z) to vertex The vector and gaze direction The cosine value of the angle between the two points is used to determine whether the point is within the cone. From point (x, y, z) to the vertex of the cone The distance Used to verify the point (x, y, z) to the cone vertex The distance is between 0 and the maximum gaze distance h; The step S4-3 of generating the gaze heat map specifically includes the following steps: S4-3-1: Input the set of targets in the image outside the vehicle S4-3-2: Calculate the gaze target set within the gaze cone Region. S4-3-3: Initialize gaze heatmap, Heatmap in (x, y, z) = 0, (x, y, z) is the coordinate inside the vehicle; S4-3-4: Calculate the gaze value of each gaze target, Among them, b∈FocusObjects, is the driver’s gaze point, σ is a Gaussian standard deviation parameter; S4-3-5: Repeat step S3-3-4, update the heatmap matrix for b∈FocusObjects, and obtain the gaze heatmap Heatmap in (x, y, z); S4-3-6: Output gaze heatmap in (x, y, z); S4-4: Calculate gaze heatmap loss. The specific steps are as follows: S4-4-1: Input the real gaze heat map S4-4-2: Calculate gaze heatmap loss: Where N is the total number of pixels in the heat map.

9. The method for detecting driving neglect risk based on in-vehicle and out-vehicle environment matching according to claim 8, characterized in that: The matching of the vehicle interior and exterior environments in step S5 specifically includes the following steps: S5-1: Input the in-car gaze heatmap in (x, y, z); S5-2: Convert Heatmap in (x, y, z) to the vehicle's external coordinate system to get the Heatmap ou t(x, y, z); Using external parameters, Heatmap in Each point (x, y, z) is calculated from the in-vehicle coordinate system S in Transform to the vehicle external coordinate system S out : Among them, R in_to_out is the rotation matrix, T in_to_out is the translation vector, which constitutes the external parameter and is used to transform from the internal coordinate system to the external coordinate system; Step S5-3 calculates the target perception score: S5-3-1: Input each external target point S5-3-2: Calculate the target perception score for each target outside the vehicle: S5-4: Calculate risk perceived loss; S5-5: Calculate total loss; S5-6: Loss of use L total , perform model training and obtain the optimal parameters, which include the parameters in S3-4-3 and S4-3-4.

10. The method for detecting driving neglect risk based on in-vehicle and out-vehicle environment matching according to claim 9, characterized in that: The step S6 of determining the risk alert specifically includes the following steps: S6-1: Use S2-4 to calculate the 3D in-vehicle coordinates of the risk target outside the vehicle S6-2: Use the model parameters of S3-4-3 to execute S3 and calculate the edge score of the gaze relationship graph model Score_e head，b : S6-3: Use the model parameters of S4-3-4 to execute S4 and generate a gaze heatmap in (x, y, z); S6-4: Calculate target perception score using S5: S6-4-1: Use S5-2 to convert the gaze heat map to the vehicle's external coordinate system to obtain the Heatmap out (x, y, z); S6-4-2: Use S5-3 to calculate the target perception score score_risk of the target outside the vehicle and the driver's gaze point b ; S6-5: Output alarm signal and make risk assessment for each target b outside the vehicle: S6-5-1: If Risk_pred b =0, indicating that target b is a non-risk target and does not trigger an alarm; S6-5-2: If Risk_pred b =1, and score_risk b If it is greater than the threshold δ, it means that the target b is a risk target, and the matching degree between the external target b and the driver’s gaze point is high, that is, the driver notices the risk target and does not trigger the alarm; S6-5-3: If Risk_pred b =1, and score_risk b If it is less than the threshold δ, it means that the target b is a risky target, but the matching degree between the external target b and the driver’s gaze point is low, that is, the driver ignores the risky target, triggering an alarm.

Images