An interpretable prediction method for driving stress based on layered road environment scene graph

Abstract

This invention discloses an interpretable prediction method for driving stress based on a layered road environment scene map, belonging to the field of intelligent driving assistance technology. The method includes the following steps: S1, extracting visual road environment information; S2, constructing a multi-layered scene map of the road environment based on the acquired visual road environment information; S3, establishing an attention-enhanced spatiotemporal graph convolutional LSTM network, and using the constructed multi-layered scene map as input to output an interpretable prediction result for driving stress. This invention, employing the above-mentioned interpretable prediction method for driving stress based on a layered road environment scene map, can provide important insights for road design, traffic management, and optimization of driving assistance systems, helping to reduce driving stress and improve road safety.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving assistance technology, and in particular to an interpretable prediction method for driving stress based on a layered road environment scene map. Background Technology

[0002] Traffic accidents are a leading cause of death across all age groups, with over 90% of accidents attributed to driver-related errors, and driving stress being a key factor. According to the Australian Traffic Accident Report, increased driving stress can increase the probability of a traffic accident by nearly tenfold. Driving stress refers to the physical and psychological tension that arises when a driver's cognitive demands exceed expectations, often accompanied by negative emotions such as anxiety. High driving stress impairs a driver's decision-making abilities and leads to inattention, thus affecting driving performance.

[0003] Factors contributing to driving stress can be categorized into intrinsic factors (such as the driver's age, personality, physical condition, and driving experience) and extrinsic factors (i.e., road environment factors, including road geometry, infrastructure, and other road users). Since intrinsic factors are inherent characteristics of the driver and difficult to change in the short term, the widespread focus in controlling driving stress is on creating a low-stress road environment by optimizing extrinsic factors to improve safety.

[0004] Identifying road environment factors that influence driving stress is crucial for creating low-stress road environments and has been a hot research topic. For example, increased road curvature exacerbates driving stress and impairs drivers' lane-keeping ability. Complex traffic conditions, such as high density of surrounding vehicles or slow-moving road users (like cyclists), also increase driving stress. Furthermore, abundant greenery along roadsides and well-designed median strips have been shown to help reduce driving stress. Currently, most research focuses on the correlation between driving stress and one or a few individual road environment factors, lacking a comprehensive analysis of these factors. Summary of the Invention

[0005] The purpose of this invention is to provide an interpretable prediction method for driving stress based on a layered road environment scene map. This method can comprehensively consider various road environment factors under the driver's visual perception and provide interpretable prediction and analysis of driving stress. It provides important insights for road design, traffic management and optimization of driving assistance systems, and helps to reduce driving stress and improve road safety.

[0006] To achieve the above objectives, this invention provides an interpretable prediction method for driving stress based on a layered road environment scene map, comprising the following steps:

[0007] S1. Extract visual road environment information;

[0008] S2. Based on the acquired visual road environment information, construct a multi-layer scene map of the road environment;

[0009] S3. Establish an attention-enhanced spatiotemporal graph convolutional LSTM network, and use the constructed multi-layer scene graph as input to output interpretable driving stress prediction results.

[0010] Preferably, the visual information extracted in S1 includes four categories: visual road geometry information extraction based on curve fitting, visual semantic information extraction based on panoramic segmentation, visual depth information extraction based on depth estimation, and visual sensitivity information extraction based on eye-tracking data.

[0011] Preferably, the visual road geometry information extraction based on curve fitting specifically involves: using the lower left corner of the road environment image as the origin of the coordinate system, and using Catmull-Rom spline curves to fit the left and right visual road boundaries from the driver's visual perspective. The left spline curve is composed of four control points P. 1L P 2L P 3L P 4L By definition, the spline curve on the right is derived from P 1R P 2R P 3R P 4R Definition; Each control point contains three types of information: pixel position coordinates X i Pixel cumulative length S iL or S iR and the tangent angle θ iL or θ (i+1)R In addition, three parameters were calculated to reflect the geometric segment information formed by adjacent control points, specifically including the visual curve width vW. i Visual curve length vs iL , vs iR and visual curve curvature vC iL , vC iR The formulas for calculating these parameters are as follows:

[0012] vW i =X i+1 -X i ;

[0013] vS iL =S (i+1)L -S iL ;

[0014] vS iR =S (i+1)R -S iR ;

[0015]

[0016] Preferably, the visual semantic information extraction based on panoramic segmentation specifically involves: using a panoramic segmentation neural network composed of feature extraction, feature fusion, semantic segmentation, and instance segmentation to extract semantic information of the facility environment and traffic participants; first, the feature extraction module preserves the features of objects in the road environment image, then these features are input into a feature pyramid network for feature fusion; finally, the semantic segmentation branch identifies the semantic category and pixel distribution of the facility environment, while the instance segmentation branch classifies traffic participants and distinguishes individual instances within each category; the outputs of semantic segmentation and instance segmentation are fused to generate the panoramic segmentation result.

[0017] Preferably, the visual depth information extraction based on depth estimation specifically involves: using PSMNet to estimate the depth of the road environment from a stereo image containing left and right images; processing the left and right images of the facility environment in parallel to extract their respective features, then using these features to construct a cost volume, and regularizing and regressing the cost volume through a 3D CNN to complete the depth estimation.

[0018] Preferably, the extraction of visually sensitive information based on eye-tracking data specifically involves: extracting fixation point information based on eye-tracking data, including pixel coordinates, fixation time, and fixation duration; cleaning the extracted fixation point information to remove noise and invalid data points; representing the driver's attention level to different areas using a heatmap, with different colors in the heatmap indicating the fixation duration of each area of ​​the road environment; and displaying the driver's attention sequence to different areas using a trajectory map, where circles represent fixation points and numbers indicate the order of fixation.

[0019] Preferably, in S2, a visual road geometry map is constructed based on the obtained visual road geometry information; a visual facility environment map is constructed based on the obtained visual semantic, depth, and sensitivity information; and a visual traffic participant map is constructed based on the obtained visual geometry, semantic, depth, and sensitivity information. By establishing the relationship between the road geometry nodes in the visual road geometry map, the traffic participant instance nodes in the visual traffic participant map, and the road surface semantic nodes in the visual facility environment map, the visual road geometry map, the visual facility environment map, and the visual traffic participant map are integrated to form a multi-layer scene map of the road environment under the driver's visual perception.

[0020] Preferably, the attention-enhanced spatiotemporal graph convolutional LSTM network in S3 consists of three parts: a spatial attention-based GCN model, a temporal attention-based LSTM model, and a support vector machine.

[0021] In the spatial attention-based GCN model, the node embedding set and its adjacency information serve as the model input. For the l-th layer of graph convolution, the nodes... Embedded feature representation The update formula is as follows:

[0022]

[0023] Where: N(i) is the node The set of neighbors; W (l) c is the learnable weight matrix of the l-th layer; i It is the normalization constant; σ(·) represents the ReLU activation function; where, The current spatial relationships between nodes are fixed and cannot reflect changes in these relationships or their impact on the final result. Therefore, a self-attention mechanism is introduced to capture the spatial relationships between nodes, as shown in the following formula:

[0024] γ ij =softmax j (LeakyReLU(a T [WH i ||WH j ]));

[0025] Where: a is the attention vector, || denotes the vector concatenation operation, LeakyReLU is an activation function, and softmax is a function of the same type. j It is for nodes A function that normalizes the neighboring nodes;

[0026] For each node and his neighbors Calculate attention score γ ij Attention score γ of node i i It is obtained by summing the attention scores of all its neighboring nodes;

[0027] Final node representation with attention The calculation is as follows:

[0028]

[0029] The GCN layer is set to 2 layers, with 100 hidden units. After each layer of MR-GCN, the scene graph G... ml The set of node embeddings is represented as X ml ; Obtain each G through graph readout operation ml Graph-level embedding H G The readout operation is sum pooling, mean pooling, or max pooling, which corresponds to summing, averaging, or selecting the maximum value for all nodes embedded in each feature dimension, respectively.

[0030] Preferably, in a time-attention-based LSTM model, the graph-level embedding sequence within a time period T is represented as follows: The sequence is input into an attention-based LSTM model; an LSTM unit consists of a forget gate, an input gate, and an output gate, and for each timet t, the LSTM updates its unit state C. t and hidden state p t The calculation company is as follows:

[0031]

[0032] Add an attention layer to the LSTM and use attention weights δ t Calculate the weighted context vector c t The calculation formula is as follows:

[0033]

[0034] The hidden state p of LSTM t and unit state C t Based on context vector c t Input image embedding In addition, the previously hidden state and unit state are updated using the following formula:

[0035]

[0036] After the LSTM processes all scene graph embeddings, the resulting output sequence (ie, p1, p2, ..., p) is processed. T Perform time readout operations to calculate the final spatiotemporal embedding Z;

[0037] In the support vector machine, a support vector machine with radial basis function kernels is used to predict driving stress Y based on spatiotemporal embedding Z; driving stress prediction is a binary classification task, where Y=1 indicates that the driver is experiencing driving stress, and Y=0 indicates that there is no stress.

[0038] The preferred attention-enhanced spatiotemporal graph convolutional LSTM network was developed using the PyTorch deep learning framework, with Adam selected as the network optimizer. Three initial learning rates were tested: 0.0001, 0.0005, and 0.001. The training process consisted of 300 epochs, with a batch size of 16. The dataset was randomly divided into training and test sets in a 3:1 ratio. The model was first trained on the training set and then evaluated on the test set. The effectiveness of the model was evaluated using accuracy and the area under the ROC curve.

[0039] Therefore, the present invention employs the above-mentioned interpretable prediction method for driving stress based on a layered road environment scene map, which can comprehensively consider various road environment factors under the driver's visual perception and provide interpretable prediction and analysis of driving stress. This provides important insights for road design, traffic management and optimization of driving assistance systems, and helps to reduce driving stress and improve road safety.

[0040] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0041] Figure 1 This is a flowchart of an interpretable prediction method for driving stress based on a layered road environment scene map according to the present invention.

[0042] Figure 2 This is a schematic diagram of a road geometry information extraction method according to an embodiment of the driving pressure interpretable prediction method based on layered road environment scene map of the present invention;

[0043] Figure 3 This is a schematic diagram of a visual semantic information extraction method according to an embodiment of the driving pressure interpretable prediction method based on layered road environment scene map of the present invention;

[0044] Figure 4 This is a schematic diagram of a visual depth information extraction method according to an embodiment of the driving pressure interpretable prediction method based on layered road environment scene map of the present invention;

[0045] Figure 5 This is a schematic diagram of a visual dynamic information extraction method according to an embodiment of the driving pressure interpretable prediction method based on layered road environment scene map of the present invention;

[0046] Figure 6 This is a schematic diagram of a method for constructing a layered road environment scene map, which is an embodiment of the driving stress interpretable prediction method based on a layered road environment scene map according to the present invention.

[0047] Figure 7 These are application case diagrams of an embodiment of the driving stress interpretable prediction method based on layered road environment scene map of the present invention, wherein (a) is application case one, (b) is application case two, and (c) is application case three. Detailed Implementation

[0048] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0049] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0050] Example 1

[0051] This embodiment includes a natural driving experiment conducted on two-way, two-lane rural roads in four Chinese provinces: Tibet Autonomous Region, Shanghai Municipality, Jiangxi Province, and Zhejiang Province, covering a total distance of over 30,000 kilometers. A total of 21 experienced drivers (14 men and 7 women) participated, ranging in age from 21 to 55 years (mean age 33.7 years, standard deviation 8.3 years). All participants had at least three years of driving experience and were familiar with the experimental road environment. Data provided by each driver included at least 1,000 kilometers of driving data.

[0052] The experiment employed a dual-camera system, including a Garmin GDR35 and a Garmin GBC30 dashcam, to capture the road environment as perceived by the driver. These dashcams were mounted on the vehicle's windshield at eye level. Additionally, a GPS locator and a three-axis accelerometer were used to acquire vehicle kinematic information, with data acquisition sampling at a rate of 1 Hz.

[0053] This embodiment assesses a driver's driving stress state from both subjective and objective perspectives. Subjectively, drivers are asked to verbally report any feelings of psychological stress during the experiment. Objectively, drivers wear heart rate monitors throughout the experiment, continuously collecting heart rate data and recording heart rate variability (HRV). HRV is a commonly used stress measurement indicator, reflecting the variation in the interval between consecutive R-wave peaks (i.e., RR interval); a lower HRV indicates a higher stress level. According to Ahmed and Rony's research, an HRV value below 23.45 milliseconds indicates that the driver is under objective driving stress. Based on geometric features, continuous rural roads are divided into multiple road segments. If a driver's subjective verbal report indicates driving stress in a specific road segment, or if their objective HRV data reaches a stress threshold, they are considered to be under stress in that road segment.

[0054] After careful screening and processing, more than 5,000 valid datasets were obtained. Each dataset contains stereo image pairs of the driver's visual point, the corresponding geographic coordinates, vehicle speed, three-axis acceleration, steering angle, and the driver's driving stress state. The ratio of driving stress state to non-stress state in the dataset is approximately 1:1.

[0055] like Figure 1 As shown, this embodiment provides an interpretable prediction method for driving stress based on a layered road environment scene map, including the following steps:

[0056] S1. Extract visual road environment information.

[0057] The extracted visual information includes four categories: visual road geometry information extraction based on curve fitting, visual semantic information extraction based on panoramic segmentation, visual depth information extraction based on depth estimation, and visual sensitivity information extraction based on eye-tracking data.

[0058] Among them, such as Figure 2 As shown, the extraction of visual road geometry information based on curve fitting specifically involves: using the lower left corner of the road environment image as the origin of the coordinate system, and fitting the left and right visual road boundaries from the driver's perspective using Catmull-Rom spline curves. The left spline curve consists of four control points P. 1L P 2L P 3L P 4L By definition, the spline curve on the right is derived from P 1R P 2R P 3R P 4R Definition. Each control point contains three types of information: pixel position coordinates ((X...) iL ,Y iL ) or (X iR ,Y iR)), pixel cumulative length (S) iL Or L iR and tangent angle (θ) iL Or θ iR (i = 1, 2, 3, 4). In addition, three parameters were calculated to reflect the geometric segment information formed by adjacent control points, specifically including the visual curve width vW. i Visual curve length vs iL , vs iR and visual curve curvature vC iL , vC iR The formulas for calculating these parameters are as follows:

[0059] vW i =X i+1 -X i ;

[0060] vS iL =S (i+1)L -S iL ;

[0061] vS iR =S (i+1)R -S iR ;

[0062]

[0063] Visual semantic information reflects a driver's perception of the semantic composition of rural roads. Road environment semantics can be divided into two categories: facility environment (including road facilities and surrounding landscape) and traffic participants (including vehicles, non-motorized vehicles and pedestrians).

[0064] The visual semantic information extraction based on panoramic segmentation specifically involves using a panoramic segmentation neural network, composed of feature extraction, feature fusion, semantic segmentation, and instance segmentation, to extract semantic information about the facility environment and traffic participants. For example... Figure 3 As shown, firstly, the features of objects in the road environment image are preserved through a feature extraction module (i.e., a residual neural network ResNet). Then, these features are input into a feature pyramid network (FPN) for feature fusion. Finally, the semantic segmentation branch identifies the semantic categories and pixel distribution of the facility environment, while the instance segmentation branch classifies traffic participants and distinguishes individual instances within each category. The outputs of semantic segmentation and instance segmentation are fused to generate a panoramic segmentation result.

[0065] Visual depth information enables drivers to accurately assess the distance of objects in the road environment, thus forming a comprehensive, three-dimensional perception of the road environment. Visual depth information extraction based on depth estimation specifically involves using PSMNet (Pyramid Stereo Matching Network) to estimate the depth of the road environment from a stereo image containing left and right images. PSMNet is a deep learning-based neural network specifically designed for depth estimation. It consists of two weight-sharing pipelines, each containing a convolutional neural network (CNN) for computing feature maps, a spatial pyramid pooling module (SPP) for feature extraction, and a convolutional layer for feature fusion. The left and right images of the road environment are processed in parallel to extract their respective features. These features are then used to construct a cost volume, which is then regularized and regressed using a 3D CNN to complete the depth estimation.

[0066] The network structure of PSMNet is as follows: Figure 4 As shown, the visual depth information of the road environment is represented in grayscale image form, with each pixel value representing the estimated depth, ranging from 0 to 255. The higher the pixel value, the greater the estimated depth. In this embodiment, the depth of the facility environment is classified as "near view" from 0 to 85, "mid view" from 85 to 170, and "far view" from 170 to 255.

[0067] Visually sensitive information refers to the degree and order in which a driver pays attention to different areas of the road environment. The extraction of visually sensitive information based on eye-tracking data specifically involves: extracting fixation point information, including pixel coordinates, fixation time, and fixation duration, based on eye-tracking data; cleaning the extracted fixation point information to remove noise and invalid data points; using a heatmap to represent the driver's attention to different areas, with different colors indicating the fixation duration for each area of ​​the road environment; and displaying the driver's order of attention to different areas using a trajectory graph, where circles represent fixation points and numbers indicate the order of fixation. The process of extracting visually sensitive information based on eye-tracking data is as follows: Figure 5 As shown.

[0068] S2. Based on the acquired visual road environment information, construct a multi-layered scene map of the road environment. This scene map consists of three sub-maps: a visual road geometry map, a visual facility environment map, and a visual traffic participant map.

[0069] Among them, a visual road geometry map was constructed based on the obtained visual road geometry information, such as... Figure 6 As shown, this diagram integrates information from control points and geometric segments, and reflects the topological relationships between control points. The visual road geometry is represented by a directed graph G. g It is represented as (V) g E g The four control points P on the spline curve on the left. 1L P2L P 3L P 4L and the four control points P on the spline curve on the right. 1R P 2R P 3R P 4R They are respectively labeled as nodes 1 to 8, i.e. Each node is accessed via a vector. This indicates that the vector includes the cumulative length S. j and tangent angle θ j Information. Consisting of two adjacent control points. The different geometric segments formed by the edges Connect. If two control points are adjacent points on the same side of the spline curve, then It can be one of four types: "Long Curve (LC)", "Long Smooth (LS)", "Short Curve (SC)", or "Short Smooth (SS)". When the length of the visual curve exceeds one-third of the total visual curve length on that side, the edge is considered "long"; when the curvature of the visual curve is greater than 0.1, the edge is considered "curved". Conversely, lengths less than one-third are considered "short", and curvatures less than 0.1 are considered "smooth". If the two control points are symmetrical control points on the left and right sides of the spline curve, then... It can be one of two types: "W" or "N" (narrow). Specifically, from near to far, if the width of the visual curve exceeds 80%, 60%, 40%, or 20% of the image width, it is classified as "W"; otherwise, it is considered "Narrow".

[0070] Based on the acquired visual semantic, depth, and sensitive information, a visual facility environment map was constructed, such as... Figure 6 As shown. This diagram integrates multiple information types, reflecting semantic categories, their distribution in three-dimensional space, and the driver's attention patterns. The visual facility environment map uses a directed graph G. f It is represented as (V) f E f Based on the output of panoramic segmentation, facility environment semantics can be automatically identified. These semantics are divided into six categories: road surface, road markings, traffic signs, roadside protection facilities, natural landscape, and roadside buildings. Therefore, a visual facility environment map can contain up to six facility environment semantic nodes, represented as follows: and Each node's attributes include semantic regions (A m ) and the driver's attention to this semantic (α) m A m It is the total number of pixels in the facility environment semantics m, α mThis represents the average gaze duration of the driver towards semantic m. Additionally, three representations of near view (v) were created. near ), medium shot (v mid ) and vision (v far The nodes of the facility environment. Based on visual depth information, each facility environment semantic node is defined by an "in" edge (i.e., These three nodes are connected to describe the visual depth of the facility's environmental semantics. Different facility environmental semantic nodes are connected by directed edges. Connected, marked "next to (B)", its direction is determined by the driver's gaze order, pointing from the earlier observed semantic category to the later observed category.

[0071] like Figure 6 As shown, a visual traffic participant map was constructed based on the acquired visual geometric, semantic, depth, and sensitivity information. This map reflects the presence of traffic participants in the road environment, their relative positional relationships with the vehicle, and the driver's attention patterns towards these participants. The visual traffic participant map is constructed using a directed graph G. p It is represented as (V) p E p Based on the output of the panoramic segmentation, N motor vehicle instances, M non-motor vehicle instances, and L pedestrian instances are extracted from the road environment. Including vehicle nodes, the visual traffic participant graph has a total of N+M+L+1 traffic participant nodes, represented as follows. Each node's attribute is the driver's attention level (α) to that instance. r ), representing the average gaze duration of a driver on instance r. Different traffic participant instance nodes are connected by directed edges. Interconnected. Based on the relative positions of traffic participant instances, It can be divided into three categories: "Same direction (Sa)", "Op" and "Surrounding direction (Su)". If and If all instances are motor vehicles traveling in the same direction, the edge is labeled "Sa"; if they travel in opposite directions, it is labeled "Op". If a vehicle instance is outside the road, it will be connected to other vehicles via an edge labeled "Su". Edges between non-motorized vehicle and pedestrian instances and other nodes are all considered "Su". The 3D Euclidean distance is calculated based on the centroid coordinates and depth values ​​of the traffic participant instances. They can also be categorized as "far (D)", "near (CL)", and "very near (VC)". Considering both relative position and distance, there are nine types of edges between traffic participant instance nodes: Sa-D, Sa-CL, Sa-VC, Op-D, Op-CL, Op-VC, Su-D, Su-CL, and Su-VC. Furthermore, traffic participant instance nodes are connected by "inner" edges (i.e.,... ) and close-up node (v near ), mid-range node (v mid ) and vision nodes (v far (Connections are used to describe the visual depth information of traffic participants.)

[0072] The road surface is the most important semantic element in the road environment, connecting the semantic elements of other infrastructure environments and also relating to the road's geometric features and traffic participants. This is achieved by establishing road geometric nodes (i.e.,...) in the visual road geometry map. ) and traffic participant instance nodes in the visual traffic participant graph (i.e. ) and road surface semantic nodes in the visual facility environment map (i.e. The three sub-graphs are integrated to form a multi-layered scene map of the road environment as perceived by the driver's vision, such as... Figure 6 As shown. Edge Node With nodes Connecting the eight control points, we associate them as features (F) with the road surface semantics. Node With nodes Connections describe the relationship between traffic participants and the road surface, with attributes indicating whether the participant is on or off the road surface. Following this, a multi-layered scene diagram of the road environment as perceived by the driver's vision is generated. The graph can then be constructed; it consists of node V. ml and edge E ml Composition. Each node E ml Having an initial feature vector Where M n It is the number of nodes, F n It represents the attribute dimensions of each node.

[0073] S3. Establish an attention-enhanced spatiotemporal graph convolutional LSTM network, and use the constructed multi-layer scene graph as input to output interpretable driving stress prediction results.

[0074] Using an attention-enhanced spatiotemporal graph convolutional LSTM network (i.e., AST-GCN-LSTM), interpretable driver stress predictions are provided based on multi-layer scene maps of the road environment as perceived by the driver's vision. For example... Figure 7As shown, the network architecture of AST-GCN-LSTM is illustrated. This network consists of three parts: a GCN model based on spatial attention, an LSTM model based on temporal attention, and a support vector machine (SVM).

[0075] Spatial attention-based graph convolutional (GCN) models aim to extract spatial features from multi-layer scene graphs (MLSG). Node embedding sets and their adjacency information serve as the model's input. For the l-th layer of the graph convolution, the nodes... Embedded feature representation The update formula is as follows:

[0076]

[0077] Where: N(i) is the node The set of neighbors; W (l) c is the learnable weight matrix of the l-th layer; i It is the normalization constant; σ(·) represents the ReLU activation function; where, The current spatial relationships between nodes are fixed and cannot reflect changes in these relationships or their impact on the final result. Therefore, a self-attention mechanism is introduced to capture the spatial relationships between nodes, as shown in the following formula:

[0078] γ ij =softmax j (LeakyReLU(a T [WH i ||WH j ]));

[0079] Where: a is the attention vector, || denotes the vector concatenation operation, LeakyReLU is an activation function, and softmax is a function of the same type. j It is for nodes A function that normalizes the neighboring nodes;

[0080] For each node and his neighbors Calculate attention score γ ij Attention score γ of node i i It is obtained by summing the attention scores of all its neighboring nodes;

[0081] Final node representation with attention The calculation is as follows:

[0082]

[0083] The GCN layer is set to 2 layers, with 100 hidden units. After each layer of MR-GCN, the scene graph G... ml The set of node embeddings is represented as Xml ; Obtain each G through graph readout operation ml Graph-level embedding H G The readout operation is sum pooling, mean pooling, or max pooling, which corresponds to summing, averaging, or selecting the maximum value for all nodes embedded in each feature dimension, respectively.

[0084] In a time-attention-based LSTM model, the graph-level embedding sequence within a time interval T is represented as follows: The sequence is input into an attention-based LSTM model; an LSTM unit consists of a forget gate, an input gate, and an output gate, and for each timet t, the LSTM updates its unit state C. t and hidden state p t The calculation company is as follows:

[0085]

[0086] Add an attention layer to the LSTM and use attention weights δ t Calculate the weighted context vector c t The calculation formula is as follows:

[0087]

[0088] The hidden state p of LSTM t and unit state C t Based on context vector c t Input image embedding In addition, the previously hidden state and unit state are updated using the following formula:

[0089]

[0090] After the LSTM processes all scene graph embeddings, the resulting output sequence (ie, p1, p2, ..., p) is processed. T Perform time readout operations to calculate the final spatiotemporal embedding Z;

[0091] In Support Vector Machines (SVMs), a radial basis function (RBF) kernel is used to predict driving stress Y based on the spatiotemporal embedding Z. Driving stress prediction is a binary classification task, where Y = 1 indicates that the driver is experiencing driving stress, and Y = 0 indicates no stress. SVMs are particularly effective when dealing with high-dimensional and non-linear data, especially with the RBF kernel, which maps the data to a higher-dimensional space to improve classification performance.

[0092] In this embodiment, the attention-enhanced spatiotemporal graph convolutional LSTM network was developed using the PyTorch deep learning framework (version 1.13.1), with Adam selected as the network optimizer. Three initial learning rates were tested: 0.0001, 0.0005, and 0.001. The training process consisted of 300 epochs, with a batch size of 16. The dataset was randomly divided into training and test sets in a 3:1 ratio. The model was first trained on the training set and then evaluated on the test set. The effectiveness of the model was evaluated using accuracy and the area under the ROC curve.

[0093] The specific experimental data are as follows:

[0094] The final AMC-GC-LSTM network achieved an accuracy of 0.952 and an AUC of 0.980, reflecting its high performance and effectiveness in predicting driving stress. Furthermore, an ablation study was conducted on the AMC-GC-LSTM network to evaluate the contribution of each component; the experimental results are shown in Table 1.

[0095] Table 1. Experimental Data for Different Models

[0096]

[0097] Model 1 is the final model, consisting of three components: a spatial attention-based GCN model, a temporal attention-based LSTM model, and an SVM. Model 2 represents the most basic version of the model, without advanced spatial or temporal processing components or attention mechanisms. In Model 2, MLSG attributes are extracted, averaged over the temporal domain, and then fed into the SVM for stress prediction. Model 2 performed the worst, with an accuracy of 0.820 and an AUC of 0.903.

[0098] When using the GCN model for spatial modeling (i.e., Model 3), the accuracy improved from 0.820 in Model 2 to 0.901, and the AUC increased from 0.903 to 0.955. Introducing an LSTM layer for temporal modeling (i.e., Model 4) also led to improved model performance, with accuracy rising to 0.898 and AUC to 0.932 compared to Model 2. When the model included both GCN and LSTM (i.e., Model 5), the accuracy reached 0.941 and the AUC reached 0.976. These results demonstrate that combining GCN and LSTM models for spatial and temporal modeling can improve the performance of driving stress prediction, highlighting the importance of each component in the AMC-GC-LSTM model.

[0099] Furthermore, the effectiveness of spatial and temporal attention mechanisms in the AMC-GC-LSTM model was evaluated. Compared to Model 5, which lacks spatial and temporal attention mechanisms, Model 6, which introduces a temporal attention mechanism, achieved an accuracy of 0.943 and an AUC of 0.978. Model 7, which introduces a spatial attention mechanism combined with GCN, achieved an accuracy of 0.950 and an AUC of 0.982. The introduction of spatial and temporal attention mechanisms improved the performance of driving stress prediction to some extent.

[0100] Driving stress can be explained and analyzed as follows:

[0101] Introducing a spatiotemporal attention mechanism into the AMC-GC-LSTM network can not only improve the performance of driving stress prediction but also enhance the model's interpretability. In this embodiment, the attention score γ... i and δ t It was used to identify key factors contributing to driving stress in the spatial and temporal domains. In the time domain, δ t The significance at different time steps is shown. If the attention weights are high in some frames, it indicates that the events in these frames are particularly important for stress prediction. In the spatial domain, higher γ... i This indicates that node i has a greater impact on driving stress prediction. Figure 7 Several sequences of rural road environments depicting driving stress were presented, with temporal attention scores δ for each sequence. t The images are visualized using bar charts. Furthermore, this embodiment includes road environment images of key sequences and their corresponding MLSGs. To enhance the readability of the graph, only the MLSG portions contributing to driving stress are retained. Specifically, nodes with spatial attention scores greater than 0.300 are retained.

[0102] like Figure 7 As shown in (a), a 30-second sequence of road environment is displayed, where the temporal attention score δ t The stress began to rise at 12 seconds and remained high between 17 and 21 seconds. Road environment images and key nodes influencing driving stress were provided at 6, 17, 19, and 22 seconds. The four figures show that traffic participant instance nodes contributed the most to driving stress in these sequences. At 6 seconds, in addition to the driver, there was one traffic participant instance node in the oncoming lane relatively close to the driver. This sequence contributed less to stress prediction, and the spatial attention score of the traffic participant instance node was low. At 17 seconds, in addition to the driver, there were four traffic participant instance nodes: three oncoming vehicles, two very close to the driver, and one surrounding a non-motorized vehicle. These vehicles then passed the driver in turn, while the non-motorized vehicle remained in front. During this process, these sequences showed high temporal and spatial attention scores for the traffic participant instance nodes.

[0103] In addition, roadside buildings in the foreground or midground are also important nodes influencing driving stress prediction, although their impact is slightly less than that of traffic participants. Many studies have shown that high traffic density is generally considered a significant source of driving stress, especially in complex rural road environments. Driving assistance functions such as distance keeping and lane keeping have been found to be effective in reducing driving stress in such situations. Furthermore, research suggests providing stress-reducing voice prompts (such as "Pay attention and slow down appropriately" or "Don't worry, just drive normally") when a large number of traffic participants are detected to alleviate some of the unavoidable driving stress.

[0104] like Figure 7 As shown in (b), a 16-second road environment sequence is presented, where the time attention score remains high between 6 and 14 seconds. This sequence covers the process of a driver approaching and navigating a curve. Road environment images at 7 and 10 seconds are provided, along with key nodes influencing driving stress. At 7 seconds, in addition to surrounding non-motorized vehicles, four road facility semantic nodes (i.e., traffic signs, roadside buildings, road surface, and surrounding landscape) significantly contribute to driving stress prediction. At 10 seconds, as the driver approaches the curve, all three types of nodes (i.e., road geometry nodes, facility environment semantic nodes, and traffic participant instance nodes) influence driving stress.

[0105] Specifically, road surface semantic nodes and road geometry nodes have the greatest impact on driving stress prediction, while oncoming vehicles and surrounding parked non-motorized vehicles also play a significant role. In this sequence, drivers navigate in complex environments with dense and diverse facility environmental semantics in the near or mid-range. Research indicates that increased environmental complexity raises the information load in the driving environment, potentially leading to increased stress and decreased driving performance. Therefore, drivers may struggle to react promptly to sudden traffic participants and curves, thus increasing driving stress. Rural roads with environmental complexity should be optimized by reducing visual clutter from signs, advertisements, and roadside objects to create a safer and more comfortable driving environment.

[0106] like Figure 7 As shown in (c), an 18-second road environment sequence is presented, where the temporal attention score is significantly higher between 12 and 15 seconds. This sequence depicts a driver approaching and navigating a bend in a scenic rural road. Road environment images at 11 and 14 seconds and key nodes influencing driving stress are provided. At 11 seconds, road geometry nodes contribute the most to driving stress. At 14 seconds, both road geometry nodes and traffic participant instance nodes have a high impact on driving stress prediction as the driver approaches the bend. Unlike road environment sequence 2, the road infrastructure and surrounding environment in this sequence are simplified, including only the road surface and surrounding vegetation.

[0107] However, drivers still experience driving stress due to curves and oncoming vehicles. In this sequence, dense vegetation almost completely occupies the field of vision, which can affect the driver's visibility and perception of curves. Furthermore, a uniform surrounding landscape may reduce driver alertness, leaving them unprepared for suddenly appearing oncoming vehicles, thus increasing driving stress. Prominent traffic signs should be placed before curves to help drivers anticipate and respond to potential hazards, although this may affect the aesthetics of the environment.

[0108] Therefore, the present invention employs the above-mentioned interpretable prediction method for driving stress based on a layered road environment scene map, which can comprehensively consider various road environment factors under the driver's visual perception and provide interpretable prediction and analysis of driving stress. This provides important insights for road design, traffic management and optimization of driving assistance systems, and helps to reduce driving stress and improve road safety.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for interpretable prediction of driving stress based on layered road environment scene maps, characterized in that: Includes the following steps: S1. Extract visual road environment information; S2. Based on the acquired visual road environment information, construct a multi-layer scene map of the road environment; S3. Establish an attention-enhanced spatiotemporal graph convolutional LSTM network, and use the constructed multi-layer scene graph as input to output interpretable driving stress prediction results; The visual information extracted in S1 includes four categories: visual road geometry information extraction based on curve fitting, visual semantic information extraction based on panoramic segmentation, visual depth information extraction based on depth estimation, and visual sensitivity information extraction based on eye-tracking data. In S2, a visual road geometry map is constructed based on the obtained visual road geometry information. A visual facility environment map is constructed based on the obtained visual semantic, depth, and sensitivity information. A visual traffic participant map is constructed based on the obtained visual geometry, semantic, depth, and sensitivity information. By establishing the relationship between the road geometry nodes in the visual road geometry map, the traffic participant instance nodes in the visual traffic participant map, and the road surface semantic nodes in the visual facility environment map, the visual road geometry map, the visual facility environment map, and the visual traffic participant map are integrated to form a multi-layer scene map of the road environment under the driver's visual perception. The attention-enhanced spatiotemporal graph convolutional LSTM network in S3 consists of three parts: a spatial attention-based GCN model, a temporal attention-based LSTM model, and a support vector machine. In the spatial attention-based GCN model, the node embedding set and its adjacency information serve as the model input for the graph convolution. Layers, nodes Embedded feature representation The update formula is as follows: ； in: It is a node The set of neighbors; It is the learnable weight matrix of the l-th layer; It is a normalization constant; Represents the activation function ReLU; where, The current spatial relationships between nodes are fixed and cannot reflect changes in these relationships or their impact on the final result. Therefore, a self-attention mechanism is introduced to capture the spatial relationships between nodes, as shown in the following formula: ； in: It is an attention vector. This represents the vector concatenation operation. It is an activation function. It is for nodes A function that normalizes the neighboring nodes; For each node and his neighbors Calculate attention score Attention score of node i It is obtained by summing the attention scores of all its neighboring nodes; Final node representation with attention The calculation is as follows: ； The GCN layer is set to 2 layers, with 100 hidden units. After each layer of MR-GCN, the scene graph... The set of node embeddings is represented as ; Obtain each through graph readout operation Graph-level embedding The readout operation is sum pooling, mean pooling, or max pooling, which correspond to summing, averaging, or selecting the maximum value for all nodes embedded in each feature dimension, respectively. In time-attention-based LSTM models, during time periods The graph-level embedding sequence within is represented as The sequence is input into an attention-based LSTM model; an LSTM unit consists of a forget gate, an input gate, and an output gate, and for each timet t, the LSTM updates its unit state. and hidden state The calculation formula is as follows: ； Add an attention layer to the LSTM and use attention weights Calculate the weighted context vector The calculation formula is as follows: = ； = ； Hidden state of LSTM and unit state Based on context vectors Input image embedding In addition, the previously hidden state and unit state are updated using the following formula: ； After the LSTM processes all scene graph embeddings, a time readout operation is performed on the output sequence to calculate the final spatiotemporal embedding Z. In support vector machines, a support vector machine with radial basis function kernels is used to predict driving stress Y based on spatiotemporal embedding Z; driving stress prediction is a binary classification task, where Y=1 indicates that the driver is experiencing driving stress, and Y=0 indicates that there is no stress.

2. The interpretable prediction method for driving stress based on a layered road environment scene map according to claim 1, characterized in that, The extraction of visual road geometry information based on curve fitting is specifically as follows: taking the lower left corner of the road environment image as the origin of the coordinate system, a Catmull-Rom spline curve is used to fit the left and right visual road boundaries from the driver's visual perspective. The left spline curve consists of four control points. , , , By definition, the spline curve on the right is formed by... , , , Definition; Each control point contains three types of information: pixel position coordinates Pixel cumulative length or and tangent angle or In addition, three parameters were calculated to reflect the geometric segment information formed by adjacent control points, specifically including the visual curve width. Visual curve length , and visual curve curvature , The formulas for calculating these parameters are as follows: ； ； ； ； 。 3. The interpretable prediction method for driving stress based on a layered road environment scene map according to claim 1, characterized in that, The visual semantic information extraction based on panoramic segmentation is specifically as follows: a panoramic segmentation neural network consisting of feature extraction, feature fusion, semantic segmentation, and instance segmentation is used to extract semantic information of the facility environment and traffic participants. First, the feature extraction module preserves the features of objects in the road environment image. Then, these features are input into a feature pyramid network for feature fusion. Finally, the semantic segmentation branch identifies the semantic category and pixel distribution of the facility environment, while the instance segmentation branch classifies traffic participants and distinguishes individual instances in each category. The outputs of semantic segmentation and instance segmentation are fused to generate the panoramic segmentation result.

4. The interpretable prediction method for driving stress based on a layered road environment scene map according to claim 1, characterized in that, The visual depth information extraction based on depth estimation is specifically as follows: PSMNet is used to estimate the depth of the road environment from a stereo image containing left and right images; the left and right images of the facility environment are processed in parallel to extract their respective features, and then these features are used to construct the cost volume. The cost volume is then regularized and regressed through a 3D CNN to complete the depth estimation.

5. The interpretable prediction method for driving stress based on a layered road environment scene map according to claim 1, characterized in that, The extraction of visually sensitive information based on eye-tracking data specifically involves: extracting fixation point information based on eye-tracking data, including pixel coordinates, fixation time, and fixation duration; cleaning the extracted fixation point information to remove noise and invalid data points; representing the driver's attention level to different areas using a heatmap, with different colors in the heatmap indicating the fixation duration of each area of ​​the road environment; and displaying the driver's attention sequence to different areas using a trajectory map, where circles represent fixation points and numbers indicate the order of fixation.

6. The interpretable prediction method for driving stress based on a layered road environment scene map according to claim 1, characterized in that: The attention-enhanced spatiotemporal graph convolutional LSTM network was developed using the PyTorch deep learning framework, with Adam selected as the network optimizer. Three initial learning rates were tested: 0.0001, 0.0005, and 0.

001. The training process consisted of 300 epochs, with a batch size of 16. The dataset was randomly divided into training and test sets in a 3:1 ratio. The model was first trained on the training set and then evaluated on the test set. The effectiveness of the model was evaluated using accuracy and the area under the ROC curve.

Images