Driving pressure interpretable prediction method based on layered road environment scene graph

Abstract

The invention discloses a driving pressure interpretable prediction method based on a layered road environment scene graph, and belongs to the technical field of intelligent driving assistance, and the method comprises the following steps: S1, extracting visual road environment information; s2, constructing a multi-layer scene graph of the road environment based on the obtained visual road environment information; and S3, establishing an attention-enhanced space-time diagram convolution LSTM network, taking the constructed multi-layer scene graph as input, and outputting an interpretable driving pressure prediction result. The driving pressure interpretable prediction method based on the layered road environment scene graph can provide important opinions for road design, traffic management and optimization of a driving assistance system, and is helpful for reducing the driving pressure and improving the road safety.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving assistance technology, and in particular to an interpretable prediction method for driving stress based on a layered road environment scene graph. Background Art

[0002] Traffic accidents are one of the major causes of death among people of all ages. More than 90% of traffic accidents are attributed to driver-related errors, and driving stress is one of the key factors. According to the Australian Traffic Accident Report, increased driving stress increases the probability of traffic accidents by nearly ten times. Driving stress refers to the physical and psychological tension caused when the cognitive demands on the driver exceed expectations, usually accompanied by negative emotions such as anxiety. High driving stress can impair the driver's decision-making ability and lead to attention errors, thereby affecting driving performance.

[0003] The factors that cause driving stress can be divided into internal factors (such as the driver's age, personality, physical condition and driving experience) and external factors (i.e. road environment factors, including road geometry, facility environment and traffic participants). Since internal factors are inherent characteristics of drivers and difficult to change in the short term, the widespread focus of controlling driving stress is to create a low-stress road environment by optimizing external factors to improve safety.

[0004] Identifying road environment factors that affect driving stress is the key to creating a low-stress road environment and has always been a hot topic of research. For example, increasing road curvature can increase driving stress and weaken the driver's lane keeping ability. Complex traffic conditions, such as a high density of surrounding vehicles or slow traffic participants (such as bicycles), can also increase driving stress. In addition, abundant greenery on both sides of the road and properly designed central isolation strips have been shown to help reduce driving stress. Currently, most studies focus on the correlation between driving stress and one or several single road environment factors, and lack a comprehensive analysis of these factors. Summary of the invention

[0005] The purpose of the present invention is to provide an interpretable prediction method for driving stress based on a hierarchical road environment scene graph, which can comprehensively consider various road environment factors under the driver's visual perception and provide interpretable driving stress prediction and analysis, providing important insights for road design, traffic management and optimization of driving assistance systems, helping to reduce driving stress and improve road safety.

[0006] To achieve the above object, the present invention provides an interpretable prediction method for driving stress based on a layered road environment scene graph, comprising the following steps:

[0007] S1, extracting visual road environment information;

[0008] S2, constructing a multi-layer scene graph of the road environment based on the acquired visual road environment information;

[0009] S3. Establish an attention-enhanced spatiotemporal graph convolutional LSTM network, take the constructed multi-layer scene graph as input, and output interpretable driving stress prediction results.

[0010] Preferably, the visual information extracted in S1 includes four categories, namely, visual road geometry information extraction based on curve fitting, visual semantic information extraction based on panoramic segmentation, visual depth information extraction based on depth estimation, and visual sensitivity information extraction based on eye movement data.

[0011] Preferably, the visual road geometry information extraction based on curve fitting is specifically as follows: taking the lower left corner of the road environment image as the origin of the coordinate system, using the Catmull-Rom spline curve to fit the left and right visual road boundaries of the road from the driver's visual perspective, and the left spline curve consists of four control points P 1L , P 2L , P 3L , P 4L Definition, the right spline curve is defined by P 1R , P 2R , P 3R , P 4R Definition; Each control point contains three types of information: pixel position coordinate X i , pixel cumulative length S iL or S iR and the tangent angle θ iL or θ (i+1)R In addition, three parameters are calculated to reflect the geometric segment information formed by adjacent control points, including the visual curve width vW i , Visual curve length vS iL 、vS iR and the visual curve curvature vC iL 、vC iR , the calculation formulas of these parameters are as follows:

[0012] vW i =X i+1 -X i ;

[0013] v iL =S (i+1)L -S iL ;

[0014] v iR =S (i+1)R -S iR ;

[0015]

[0016] Preferably, the visual semantic information extraction based on panoramic segmentation is specifically as follows: a panoramic segmentation neural network composed of feature extraction, feature fusion, semantic segmentation and instance segmentation is used to extract the semantic information of the facility environment and traffic participants; first, the features of the objects in the road environment image are retained by the feature extraction module, and then these features are input into the feature pyramid network for feature fusion, and finally, the semantic segmentation branch identifies the semantic category and pixel distribution of the facility environment, while the instance segmentation branch classifies the traffic participants and distinguishes the individual instances in each category; the output results of the semantic segmentation and instance segmentation are fused to generate a panoramic segmentation result.

[0017] Preferably, the visual depth information extraction based on depth estimation is specifically as follows: PSMNet is used to estimate the depth of the road environment through a stereo image pair containing left and right images; the left image and the right image of the facility environment are processed in parallel to extract their respective features, and then these features are used to construct a cost volume, and the cost volume is regularized and regressed through 3D CNN to complete the depth estimation.

[0018] Preferably, the extraction of visual sensitive information based on eye movement data is specifically as follows: extracting gaze point information based on eye movement data, including pixel coordinates, gaze time and gaze duration; cleaning the extracted gaze point information to remove noise and invalid data points; using a heat map to represent the driver's attention level to different areas, with different colors in the heat map indicating the gaze duration of each area of ​​the road environment; the driver's attention order to different areas is displayed through a trajectory map, in which circles represent gaze points and numbers indicate the gaze order.

[0019] Preferably, in S2, a visual road geometry map is constructed based on the obtained visual road geometry information, a visual facility environment map is constructed based on the obtained visual semantics, depth and sensitive information, and a visual traffic participant map is constructed based on the obtained visual geometry, semantics, depth and sensitive information. By establishing the relationship between the road geometry nodes in the visual road geometry map, the traffic participant instance nodes in the visual traffic participant map and the road surface semantic nodes in the visual facility environment map, the visual road geometry map, the visual facility environment map and the visual traffic participant map are integrated to form a multi-layer scene map of the road environment under the driver's visual perception.

[0020] Preferably, the attention-enhanced spatiotemporal graph convolutional LSTM network in S3 consists of three parts: a GCN model based on spatial attention, a LSTM model based on temporal attention, and a support vector machine;

[0021] In the GCN model based on spatial attention, the node embedding set and its adjacency information are used as the input of the model. For the lth layer of graph convolution, the node Embedded feature representation The update formula is as follows:

[0022]

[0023] Where: N(i) is the node The neighbor set of W (l) is the learnable weight matrix of layer l; c i is the normalization constant; σ(·) represents the activation function ReLU; where, is fixed and cannot reflect the changes in different spatial relationships between nodes and their impact on the final result; the self-attention mechanism is introduced to capture the spatial association between nodes. The formula is as follows:

[0024] γ ij =softmax j (LeakyReLU(a T [WH i ||WH j ]));

[0025] Among them: a is the attention vector, || represents the concatenation operation of the vector, LeakyReLU is an activation function, softmax j It is a node A function to normalize the neighbor nodes of ;

[0026] For each node and its neighbors Calculate the attention score γ ij ; Attention score γ of node i i Obtained by summing the attention scores of all its neighboring nodes;

[0027] Final node representation with attention The calculation is as follows:

[0028]

[0029] The number of GCN layers is set to 2, and the number of hidden units is 100. After each layer of MR-GCN, the scene graph G ml The node embedding set is denoted as X ml ; Through the graph read operation, obtain each G ml The graph-level embedding H G , the readout operation is sum pooling, mean pooling or max pooling, which corresponds to summing, averaging or selecting the maximum value of all node embeddings in each feature dimension respectively.

[0030] Preferably, in the LSTM model based on time attention, the graph-level embedding sequence within time period T is represented as This sequence is input into the attention-based LSTM model; an LSTM unit consists of a forget gate, an input gate, and an output gate. For each timestamp t, the LSTM updates its unit state C t and the hidden state p t , calculate the company as follows:

[0031]

[0032] Add an attention layer to the LSTM and use the attention weight δ t Calculate the weighted context vector c t , the calculation formula is as follows:

[0033]

[0034] The hidden state p of LSTM t and cell status C t Based on the context vector c t , input graph embedding As well as the previous hidden state and unit state, the formula is as follows:

[0035]

[0036] After LSTM processes all scene graph embeddings, the resulting output sequence (ie, p1, p2, ..., p T ) performs a temporal readout operation to calculate the final spatiotemporal embedding Z;

[0037] In the support vector machine, a support vector machine with a radial basis function kernel is used to predict driving stress Y based on the spatiotemporal embedding Z; driving stress prediction is a binary classification task, where Y=1 indicates that the driver is experiencing driving stress and Y=0 indicates no stress.

[0038] Preferably, the attention-enhanced spatiotemporal graph convolutional LSTM network is developed using the PyTorch deep learning framework, Adam is selected as the network optimizer, and three initial learning rates are tested: 0.0001, 0.0005 and 0.001; the training process consists of 300 epochs, and the batch size is set to 16; the data set is randomly divided into a training set and a test set in a ratio of 3:1. The model is first trained on the training set and then evaluated on the test set. The effectiveness of the model is evaluated by the accuracy and the area under the ROC curve.

[0039] Therefore, the present invention adopts the above-mentioned explainable prediction method of driving stress based on the layered road environment scene graph, which can comprehensively consider various road environment factors under the driver's visual perception, and provide explainable driving stress prediction and analysis, providing important insights for road design, traffic management and optimization of driving assistance systems, which helps to reduce driving stress and improve road safety.

[0040] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of a method for interpretable prediction of driving stress based on a layered road environment scene graph of the present invention;

[0042] Figure 2 It is a schematic diagram of a road geometry information extraction method according to an embodiment of a method for interpretable prediction of driving stress based on a layered road environment scene graph of the present invention;

[0043] Figure 3 It is a schematic diagram of a visual semantic information extraction method of an embodiment of a method for interpretable prediction of driving stress based on a layered road environment scene graph of the present invention;

[0044] Figure 4 It is a schematic diagram of a visual depth information extraction method of an embodiment of a driving stress interpretable prediction method based on a layered road environment scene graph of the present invention;

[0045] Figure 5 It is a schematic diagram of a visual dynamic information extraction method of an embodiment of a method for interpretable prediction of driving stress based on a layered road environment scene graph of the present invention;

[0046] Figure 6 It is a schematic diagram of a method for constructing a layered road environment scene graph according to an embodiment of a method for interpretable prediction of driving stress based on a layered road environment scene graph of the present invention;

[0047] Figure 7 It is an application case diagram of an embodiment of an explainable prediction method for driving stress based on a layered road environment scene diagram of the present invention, wherein (a) is application case one, (b) is application case two, and (c) is application case three. DETAILED DESCRIPTION

[0048] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0049] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0050] Embodiment 1

[0051] In this embodiment, a natural driving experiment was conducted on two-way two-lane rural roads in four provinces in China, including the Tibet Autonomous Region, Shanghai, Jiangxi Province and Zhejiang Province, covering a total mileage of more than 30,000 kilometers. A total of 21 experienced drivers (14 men and 7 women) participated in the experiment, with an age range of 21 to 55 years (average age of 33.7 years and standard deviation of 8.3 years). All participants had at least three years of driving experience and were familiar with the experimental road environment. The data provided by each driver contained at least 1,000 kilometers of driving data.

[0052] The experiment used a dual-camera system, including GARMIN GDR35 and GARMIN GBC30 dashcams, to capture the road environment as perceived by the driver. These dashcams were mounted on the vehicle windshield, in line with the driver's line of sight. In addition, a GPS locator and a three-axis accelerometer were used to obtain the vehicle's kinematic information, with a data acquisition sampling rate of 1 Hz.

[0053] This embodiment evaluates the driver's driving stress state from both subjective and objective perspectives. In terms of subjectivity, the driver is required to verbally report any feelings of psychological stress immediately during the experiment. In terms of objectivity, the driver wears a heart rate monitor throughout the entire process, continuously collects heart rate data, and records heart rate variability (HRV). HRV is a commonly used stress measurement indicator that reflects the interval changes between consecutive R wave peaks (i.e., RR intervals). A lower HRV indicates a higher stress level. According to the research of Ahmed and Rony, when the HRV value is lower than 23.45 milliseconds, it indicates that the driver is in an objective driving stress state. Based on geometric features, continuous rural roads are divided into multiple sections. If the driver's subjective verbal report in a specific section indicates driving stress, or his objective HRV data reaches the stress threshold, he is considered to be in a stress state in that section.

[0054] After careful screening and processing, more than 5,000 valid data sets were obtained. Each data set contains a stereo image pair of the driver's visual point, the corresponding geographic coordinates, vehicle speed, three-axis acceleration, steering angle, and the driver's driving stress state. The ratio of driving stress state to non-stress state in the data set is about 1:1.

[0055] like Figure 1 As shown, this embodiment provides an interpretable prediction method for driving stress based on a layered road environment scene graph, comprising the following steps:

[0056] S1. Extract visual road environment information.

[0057] The extracted visual information includes four categories: visual road geometry information extraction based on curve fitting, visual semantic information extraction based on panoramic segmentation, visual depth information extraction based on depth estimation, and visual sensitivity information extraction based on eye movement data.

[0058] Among them, Figure 2 As shown in Figure 1, the visual road geometry information extraction based on curve fitting is as follows: the lower left corner of the road environment image is used as the origin of the coordinate system, and the Catmull-Rom spline curve is used to fit the left and right visual road boundaries of the road from the driver's visual perspective. The left spline curve consists of four control points P 1L , P 2L , P 3L , P 4L Definition, the right spline curve is defined by P 1R , P 2R , P 3R , P 4R Definition. Each control point contains three types of information: pixel position coordinates ((X iL ,Y iL ) or (X iR ,Y iR)), pixel cumulative length (S iL or L iR ) and the tangent angle (θ iL Or θ iR In addition, three parameters are calculated to reflect the geometric segment information formed by adjacent control points, including the visual curve width vW i , Visual curve length vS iL 、vS iR and the visual curve curvature vC iL 、vC iR , the calculation formulas of these parameters are as follows:

[0059] vW i =X i+1 -X i ;

[0060] v iL =S (i+1)L -S iL ;

[0061] v iR =S (i+1)R -S iR ;

[0062]

[0063] Visual semantic information reflects the driver's perception of the semantic composition of rural roads. The road environment semantics can be divided into two categories: facility environment (including road facilities and surrounding landscape) and traffic participants (including vehicles, non-motor vehicles and pedestrians).

[0064] The visual semantic information extraction based on panoptic segmentation is specifically: using a panoptic segmentation neural network composed of feature extraction, feature fusion, semantic segmentation and instance segmentation to extract the semantic information of facility environment and traffic participants. Figure 3 As shown in the figure, first, the features of objects in the road environment image are retained through the feature extraction module (i.e., residual neural network ResNet), and then these features are input into the feature pyramid network (FPN) for feature fusion. Finally, the semantic segmentation branch identifies the semantic category and pixel distribution of the facility environment, while the instance segmentation branch classifies traffic participants and distinguishes individual instances in each category. The output results of semantic segmentation and instance segmentation are fused to generate a panoramic segmentation result.

[0065] Visual depth information enables drivers to accurately assess the distance of objects in the road environment, thereby forming a comprehensive, three-dimensional perception of the road environment. The extraction of visual depth information based on depth estimation is specifically: PSMNet (i.e., pyramid stereo matching network) is used to estimate the depth of the road environment through a stereo image pair containing left and right images. PSMNet is a deep learning-based neural network designed specifically for depth estimation. It consists of two weight-sharing pipelines, each of which contains a convolutional neural network (CNN) for calculating feature maps, a spatial pyramid pooling module (SPP) for feature extraction, and a convolutional layer for feature fusion. The left and right images of the facility environment are processed in parallel to extract their respective features, and then these features are used to construct the cost volume, and the cost volume is regularized and regressed through 3D CNN to complete the depth estimation.

[0066] The network structure of PSMNet is as follows Figure 4 As shown, the visual depth information of the road environment is represented in the form of a grayscale image, and each pixel value represents an estimated depth, ranging from 0 to 255. The higher the pixel value, the greater the estimated depth. In this embodiment, the depth of the facility environment is classified from 0 to 85 as "near view", 85 to 170 as "mid view", and 170 to 255 as "distant view".

[0067] Visually sensitive information refers to the degree of attention a driver pays to different areas of the road environment and the order in which they pay attention. The extraction of visually sensitive information based on eye movement data is specifically as follows: extracting gaze point information based on eye movement data, including pixel coordinates, gaze time, and gaze duration. Cleaning the extracted gaze point information to remove noise and invalid data points. A heat map is used to represent the degree of attention a driver pays to different areas, and different colors in the heat map indicate the gaze duration of each area of ​​the road environment. The order in which a driver pays attention to different areas is shown in a trajectory map, in which circles represent gaze points and numbers indicate the order of gaze. The process of extracting visually sensitive information based on eye movement data is as follows: Figure 5 shown.

[0068] S2. Based on the acquired visual road environment information, a multi-layer scene graph of the road environment is constructed. The scene graph consists of three sub-graphs: a visual road geometry graph, a visual facility environment graph, and a visual traffic participant graph.

[0069] Among them, based on the obtained visual road geometry information, a visual road geometry map is constructed, such as Figure 6 As shown in Figure 1. This graph integrates the information of control points and geometric segments and reflects the topological relationship between control points. The visual road geometry graph is represented by a directed graph G. g It is defined as (V g ,E g ). The four control points P on the left spline curve 1L , P2L , P 3L , P 4L and the four control points P on the right spline curve 1R , P 2R , P 3R , P 4R They are marked as nodes 1 to 8 respectively, that is, Each node is connected by vector Indicates that the vector includes the cumulative length S j and the tangent angle θ j Information. By two adjacent control points The different geometric segments formed by the edge If two control points are adjacent points on the same side of the spline, then Can be one of four types: "Long Curved (LC)", "Long Smooth (LS)", "Short Curved (SC)", or "Short Smooth (SS)". An edge is considered "long" when the visual curve length exceeds one-third of the total visual curve length on that side, and "curved" when the visual curve curvature is greater than 0.1. Conversely, an edge is considered "short" when the length is less than one-third, and "smooth" when the curvature is less than 0.1. If the two control points are symmetrical control points on the left and right sides of the spline, then Can be one of two types: "Wide (W)" or "Narrow (N)". Specifically, from near to far, if the visual curve width exceeds 80%, 60%, 40%, or 20% of the image width, it is classified as "Wide", otherwise it is considered "Narrow".

[0070] Based on the acquired visual semantics, depth and sensitive information, a visual facility environment map is constructed, such as Figure 6 As shown in Figure 1, the graph integrates multiple types of information, reflecting semantic categories, their distribution in three-dimensional space, and the driver's attention pattern. f It is defined as (V f , E f ). Based on the output of panoptic segmentation, facility environment semantics can be automatically identified. These semantics are divided into six categories: road surface, road markings, traffic signs, roadside protection facilities, natural landscapes, and roadside buildings. Therefore, the visual facility environment graph can contain up to six facility environment semantic nodes, represented as and The attributes of each node include semantic region (A m ) and the driver’s attention to the semantics (α m ). m is the total number of pixels of the facility environment semantics m, α mrepresents the average fixation time of the driver on semantic m. In addition, three near-view (v near )、Mid-shot(v mid ) and vision (v far ) nodes. According to the visual depth information, each facility environment semantic node is connected through the “in” edge (i.e. ) are connected to these three nodes to describe the visual depth of the facility environment semantics. Different facility environment semantic nodes are connected through directed edges. connected, labeled “beside (B)”, with the direction determined by the driver’s gaze order, pointing from the earlier observed semantic category to the later observed category.

[0071] like Figure 6 As shown in Figure 1, a visual traffic participant graph is constructed based on the acquired visual geometry, semantics, depth and sensitive information. This graph reflects the existence of traffic participants in the road environment, their relative position relationship with the vehicle, and the driver's attention pattern to these participants. The visual traffic participant graph is represented by a directed graph G p It is defined as (V p , E p ). According to the output of panoptic segmentation, N motor vehicle instances, M non-motor vehicle instances, and L pedestrian instances are extracted from the road environment. Including the vehicle node, the visual traffic participant graph has a total of N+M+L+1 traffic participant nodes, expressed as The attribute of each node is the driver's attention to the instance (α r ), represents the average fixation time of the driver on instance r. Different traffic participant instance nodes are connected by directed edges. Based on the relative positions of traffic participant instances, There are three types: "Same Direction (Sa)", "Opposite Direction (Op)" and "Around (Su)". and If they are all motor vehicle instances and they are traveling in the same direction, the edge is labeled "Sa"; if they are traveling in opposite directions, it is labeled "Op". If a vehicle instance is outside the road, it will be connected to the vehicle and other vehicles through the "Su" edge. The edges of non-motor vehicle and pedestrian instances with other nodes are all considered "Su". Based on the three-dimensional Euclidean distance calculated from the centroid coordinates and depth values ​​of the traffic participant instances, It can also be classified into: "Distant (D)", "Close (CL)" and "Very Close (VC)". Considering the relative position and distance, there may be 9 types of edges between traffic participant instance nodes: Sa-D, Sa-CL, Sa-VC, Op-D, Op-CL, Op-VC, Su-D, Su-CL and Su-VC. In addition, traffic participant instance nodes are connected through "inside" edges (i.e. ) and the near-view node (v near )、Mid-ground node(v mid ) and the vision node (v far ) to describe the visual depth information of traffic participants.

[0072] The road surface is the most important semantics in the road environment, connecting other facility environment semantics, and also related to the geometric features of the road and traffic participants. ) and the traffic participant instance nodes in the visual traffic participant graph (i.e. ) and the road semantic nodes in the visual facility environment graph (i.e. ), the three sub-graphs are integrated to form a multi-layer scene graph of the road environment under the driver's visual perception, such as Figure 6 Side The node With Node The eight control points are connected as features (i.e., F) to be associated with the road semantics. The node With Node The connection describes the relationship between the traffic participant and the road surface, and the attribute indicates whether the participant is on the road surface or off the road surface. After that, the multi-layer scene graph of the road environment under the driver's visual perception The construction is completed. The graph consists of node V ml and edge E ml Each node E ml With an initial eigenvector Among them, M n is the number of nodes, F n is the attribute dimension of each node.

[0073] S3. Establish an attention-enhanced spatiotemporal graph convolutional LSTM network, take the constructed multi-layer scene graph as input, and output interpretable driving stress prediction results.

[0074] Using the attention-enhanced spatiotemporal graph convolutional LSTM (i.e., AST-GCN-LSTM) network, we provide interpretable driving stress prediction based on the multi-layer scene graph of the road environment under the driver’s visual perception. Figure 7As shown in the figure, the network architecture of AST-GCN-LSTM is shown, which consists of three parts: a GCN model based on spatial attention, a LSTM model based on temporal attention, and a support vector machine (SVM).

[0075] The graph convolution (GCN) model based on spatial attention is designed to extract spatial features from the multi-layer scene graph (MLSG). The node embedding set and its adjacency information are used as the input of the model. For the lth layer of the graph convolution, the node Embedded feature representation The update formula is as follows:

[0076]

[0077] Where: N(i) is the node The neighbor set of W (l) is the learnable weight matrix of layer l; c i is the normalization constant; σ(·) represents the activation function ReLU; where, is fixed and cannot reflect the changes in different spatial relationships between nodes and their impact on the final result; the self-attention mechanism is introduced to capture the spatial association between nodes. The formula is as follows:

[0078] γ ij =softmax j (LeakyReLU(a T [WH i ||WH j ]));

[0079] Among them: a is the attention vector, || represents the concatenation operation of the vector, LeakyReLU is an activation function, softmax j It is a node A function to normalize the neighbor nodes of ;

[0080] For each node and its neighbors Calculate the attention score γ ij ; Attention score γ of node i i Obtained by summing the attention scores of all its neighboring nodes;

[0081] Final node representation with attention The calculation is as follows:

[0082]

[0083] The number of GCN layers is set to 2, and the number of hidden units is 100. After each layer of MR-GCN, the scene graph G ml The node embedding set is denoted as Xml ; Through the graph read operation, obtain each G ml The graph-level embedding H G , the readout operation is sum pooling, mean pooling or max pooling, which corresponds to summing, averaging or selecting the maximum value of all node embeddings in each feature dimension respectively.

[0084] In the temporal attention-based LSTM model, the graph-level embedding sequence within time period T is represented as This sequence is input into the attention-based LSTM model; an LSTM unit consists of a forget gate, an input gate, and an output gate. For each timestamp t, the LSTM updates its unit state C t and the hidden state p t , calculate the company as follows:

[0085]

[0086] Add an attention layer to the LSTM and use the attention weight δ t Calculate the weighted context vector c t , the calculation formula is as follows:

[0087]

[0088] The hidden state p of LSTM t and cell status C t Based on the context vector c t , input graph embedding As well as the previous hidden state and unit state, the formula is as follows:

[0089]

[0090] After LSTM processes all scene graph embeddings, the resulting output sequence (ie, p1, p2, ..., p T ) performs a temporal readout operation to calculate the final spatiotemporal embedding Z;

[0091] In the support vector machine, a support vector machine with a radial basis function kernel is used to predict driving stress Y based on the spatiotemporal embedding Z; driving stress prediction is a binary classification task, where Y = 1 means the driver is experiencing driving stress and Y = 0 means no stress. SVM is particularly effective in processing high-dimensional and nonlinear data, especially through the RBF kernel, which maps the data to a higher dimensional space to improve classification performance.

[0092] In this embodiment, the attention-enhanced spatiotemporal graph convolutional LSTM network was developed using the PyTorch deep learning framework (version 1.13.1), Adam was selected as the network optimizer, and three initial learning rates were tested: 0.0001, 0.0005, and 0.001. The training process lasted for 300 epochs, and the batch size was set to 16. The data set was randomly divided into a training set and a test set with a ratio of 3:1. The model was first trained on the training set and then evaluated on the test set. The effectiveness of the model was evaluated by accuracy and area under the ROC curve.

[0093] The specific experimental data are as follows:

[0094] The final AMC-GC-LSTM network achieved an accuracy of 0.952 and an AUC of 0.980, reflecting its high performance and effectiveness in predicting driving stress. In addition, an ablation study was conducted on the AMC-GC-LSTM network to evaluate the contribution of each component, and the experimental results are shown in Table 1.

[0095] Table 1 Experimental data of different models

[0096]

[0097] Model 1 is the final model, which consists of three components: a GCN model based on spatial attention, an LSTM model based on temporal attention, and an SVM. Model 2 represents the most basic version of the model, without advanced spatial or temporal processing components or attention mechanisms. In Model 2, the MLSG attributes are extracted, averaged over the temporal domain, and then fed into the SVM for pressure prediction. Model 2 performed the worst with an accuracy of 0.820 and an AUC of 0.903.

[0098] When using the GCN model for spatial modeling (i.e., Model 3), the accuracy increased from 0.820 in Model 2 to 0.901, and the AUC increased from 0.903 to 0.955. The introduction of the LSTM layer for temporal modeling (i.e., Model 4) also led to improved model performance, with the accuracy rising to 0.898 and the AUC rising to 0.932 compared to Model 2. When the model included both GCN and LSTM (i.e., Model 5), the accuracy reached 0.941 and the AUC reached 0.976. These results show that combining the GCN and LSTM models for spatial and temporal modeling can improve the performance of driving stress prediction, highlighting the importance of each component in the AMC-GC-LSTM model.

[0099] In addition, the effectiveness of the spatial and temporal attention mechanisms in the AMC-GC-LSTM model was evaluated. Compared with model 5, which lacks the spatial and temporal attention mechanisms, model 6 introduces the temporal attention mechanism, with an accuracy of 0.943 and an AUC of 0.978. Model 7 introduces the spatial attention mechanism combined with GCN, with an accuracy of 0.950 and an AUC of 0.982. The introduction of spatial and temporal attention mechanisms improves the performance of driving stress prediction to a certain extent.

[0100] Driving stress can be explained by the following analysis:

[0101] Introducing the spatiotemporal attention mechanism in the AMC-GC-LSTM network can not only improve the performance of driving stress prediction, but also improve the interpretability of the model. In the embodiment, the attention score γ i and δ t is used to identify the key factors that contribute to driving stress in both spatial and temporal domains. In the temporal domain, δ t The saliency of different time steps is shown. If the attention weights of some frames are high, it means that the events in these frames are particularly important for pressure prediction. In the spatial domain, higher γ i This indicates that node i has a greater impact on the prediction of driving stress. Figure 7 Several sequences of rural road environments where drivers experience driving stress are shown. In each sequence, the temporal attention score δ t It is visualized with a bar chart. In addition, this embodiment also includes some key sequences of road environment images and corresponding MLSG. In order to enhance the readability of the graph, only the MLSG part that contributes to driving stress is retained. Specifically, nodes with a spatial attention score greater than 0.300 are retained.

[0102] like Figure 7 As shown in (a), a 30-second road environment sequence is shown, where the temporal attention score δ t It starts to rise at 12 seconds and remains high from 17 to 21 seconds. Images of the road environment and key nodes that affect driving stress are provided at 6 seconds, 17 seconds, 19 seconds, and 22 seconds. The four figures show that in these sequences, the traffic participant instance nodes contribute the most to driving stress. At 6 seconds, in addition to the ego vehicle, there is also a traffic participant instance node in the opposite lane that is close to the driver. This sequence contributes less to stress prediction, and the spatial attention score of the traffic participant instance node is low. At 17 seconds, in addition to the ego vehicle, there are four traffic participant instance nodes: three oncoming vehicles, two of which are very close to the driver, and the other is a surrounding non-motorized vehicle. These vehicles then pass the driver in turn, and the non-motorized vehicle stays in front. In this process, these sequences show high temporal and spatial attention scores of the traffic participant instance nodes.

[0103] In addition, roadside buildings in the near or medium distance are also important nodes that affect the prediction of driving stress, although their impact is slightly lower than that of traffic participants. Many studies have shown that high traffic density is generally considered to be an important source of driving stress, especially in complex rural road environments. Driving assistance functions such as keeping distance and lane keeping have been found to be effective in reducing driving stress in this situation. In addition, the study recommends providing decompression voice prompts (such as "pay attention and slow down appropriately" or "don't worry, just drive normally") when a large number of traffic participants are detected to alleviate some inevitable driving stress.

[0104] like Figure 7 (b) shows a 16-second road environment sequence, in which the temporal attention score remains high from 6 to 14 seconds. The sequence covers the process of the driver approaching and passing a curve. Road environment images at 7 seconds and 10 seconds and key nodes affecting driving stress are provided. At 7 seconds, in addition to the surrounding non-motorized vehicles, four road facility semantic nodes (i.e., traffic signs, roadside buildings, road surfaces, and surrounding landscapes) contribute significantly to the prediction of driving stress. At 10 seconds, as the driver approaches the curve, all three types of nodes (i.e., road geometry nodes, facility environment semantic nodes, and traffic participant instance nodes) have an impact on driving stress.

[0105] Specifically, road surface semantic nodes and road geometry nodes have the greatest impact on driving stress prediction, while oncoming vehicles and surrounding parked non-motorized vehicles also play an important role. In this sequence, the driver is driving in a complex environment with dense and diverse facility environment semantics in the near or mid-ground. Studies have shown that increased environmental complexity increases the information load in the driving environment, which may lead to increased stress and decreased driving performance. As a result, drivers may have difficulty reacting to sudden traffic participants and curves in a timely manner, increasing driving stress. Rural roads with environmental complexity should be optimized by reducing visual clutter of signs, advertisements, and roadside objects to create a safer and more comfortable driving environment.

[0106] like Figure 7 (c) shows an 18-second road environment sequence, in which the temporal attention score is significantly higher between 12 and 15 seconds. The sequence shows the process of the driver approaching and passing a scenic rural road curve. Road environment images and key nodes affecting driving stress are provided at 11 and 14 seconds. At 11 seconds, the road geometry node contributes the most to driving stress. At 14 seconds, as the driver approaches the curve, the road geometry node and traffic participant instance node have a high impact on driving stress prediction. Unlike road environment sequence 2, the road facilities and surrounding environment in this sequence are relatively simplified, including only the road surface and surrounding vegetation.

[0107] However, drivers still feel driving stress due to the curve and oncoming vehicles. In this sequence, dense vegetation occupies almost the entire field of view, which may affect the driver's vision and perception of the curve. In addition, the uniform surrounding landscape may reduce the driver's vigilance and make them less prepared for the sudden appearance of oncoming vehicles, thereby increasing driving stress. Prominent traffic signs should be set up before the curve to help drivers predict and respond to potential dangers, although this may affect the aesthetics of the environment.

[0108] Therefore, the present invention adopts the above-mentioned explainable prediction method of driving stress based on the layered road environment scene graph, which can comprehensively consider various road environment factors under the driver's visual perception, and provide explainable driving stress prediction and analysis, providing important insights for road design, traffic management and optimization of driving assistance systems, which helps to reduce driving stress and improve road safety.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for interpretable prediction of driving stress based on a hierarchical road environment scene graph, characterized in that: The following steps are involved: S1, extracting visual road environment information; S2, constructing a multi-layer scene graph of the road environment based on the acquired visual road environment information; S3. Establish an attention-enhanced spatiotemporal graph convolutional LSTM network, take the constructed multi-layer scene graph as input, and output interpretable driving stress prediction results.

2. The method for interpretable prediction of driving stress based on a hierarchical road environment scene graph according to claim 1, characterized in that: The visual information extracted in S1 includes four categories, namely, visual road geometry information extraction based on curve fitting, visual semantic information extraction based on panoramic segmentation, visual depth information extraction based on depth estimation, and visual sensitivity information extraction based on eye movement data.

3. The method for interpretable prediction of driving stress based on a hierarchical road environment scene graph according to claim 2, characterized in that: The specific process of extracting visual road geometry information based on curve fitting is as follows: the lower left corner of the road environment image is used as the origin of the coordinate system, and the Catmull-Rom spline curve is used to fit the left and right visual road boundaries of the road from the driver's visual perspective. The left spline curve consists of four control points P 1L , P 2L , P 3L , P 4L Definition, the right spline curve is defined by P 1R , P 2R , P 3R , P 4R Definition; Each control point contains three types of information: pixel position coordinate X i , pixel cumulative length S iL or S iR and the tangent angle θ iL or θ (i+1)R In addition, three parameters are calculated to reflect the geometric segment information formed by adjacent control points, including the visual curve width vW i , Visual curve length vS iL 、vS iR and the visual curve curvature vC iL 、vC iR , the calculation formulas of these parameters are as follows: vW i =X i+1 -X i ; vS iL =S (i+1)L -S iL ; vS iR =S (i+1)R -S iR ; 4. The method for interpretable prediction of driving stress based on a hierarchical road environment scene graph according to claim 2, characterized in that: The specific method of visual semantic information extraction based on panoramic segmentation is as follows: a panoramic segmentation neural network composed of feature extraction, feature fusion, semantic segmentation and instance segmentation is used to extract the semantic information of facility environment and traffic participants; first, the features of objects in the road environment image are retained through the feature extraction module, and then these features are input into the feature pyramid network for feature fusion; finally, the semantic segmentation branch identifies the semantic category and pixel distribution of the facility environment, while the instance segmentation branch classifies traffic participants and distinguishes individual instances in each category; the output results of semantic segmentation and instance segmentation are fused to generate a panoramic segmentation result.

5. The method for interpretable prediction of driving stress based on a hierarchical road environment scene graph according to claim 2, characterized in that: The visual depth information extraction based on depth estimation is specifically as follows: PSMNet is used to estimate the depth of the road environment through a stereo image pair containing left and right images; the left and right images of the facility environment are processed in parallel to extract their respective features, and then these features are used to construct a cost volume, and the cost volume is regularized and regressed through 3DCNN to complete the depth estimation.

6. The method for interpretable prediction of driving stress based on a hierarchical road environment scene graph according to claim 2, characterized in that: The extraction of visual sensitive information based on eye movement data is specifically as follows: extracting gaze point information based on eye movement data, including pixel coordinates, gaze time and gaze duration; cleaning the extracted gaze point information to remove noise and invalid data points; using a heat map to represent the driver's attention to different areas, with different colors in the heat map indicating the gaze duration of each area of ​​the road environment; the driver's attention order to different areas is displayed through a trajectory map, in which circles represent gaze points and numbers indicate the gaze order.

7. The method for interpretable prediction of driving stress based on a hierarchical road environment scene graph according to claim 2, characterized in that: In S2, a visual road geometry map is constructed based on the obtained visual road geometry information, a visual facility environment map is constructed based on the obtained visual semantics, depth and sensitive information, and a visual traffic participant map is constructed based on the obtained visual geometry, semantics, depth and sensitive information. By establishing the relationship between the road geometry nodes in the visual road geometry map, the traffic participant instance nodes in the visual traffic participant map and the road surface semantic nodes in the visual facility environment map, the visual road geometry map, the visual facility environment map and the visual traffic participant map are integrated to form a multi-layer scene map of the road environment under the driver's visual perception.

8. The method for interpretable prediction of driving stress based on a hierarchical road environment scene graph according to claim 2, characterized in that: The attention-enhanced spatiotemporal graph convolutional LSTM network in S3 consists of three parts: a GCN model based on spatial attention, an LSTM model based on temporal attention, and a support vector machine; In the GCN model based on spatial attention, the node embedding set and its adjacency information are used as the input of the model. For the lth layer of graph convolution, the node Embedded feature representation The update formula is as follows: Where: N(i) is the node The neighbor set of W (l) is the learnable weight matrix of layer l; c i is the normalization constant; σ(·) represents the activation function ReLU; where, is fixed and cannot reflect the changes in different spatial relationships between nodes and their impact on the final result; the self-attention mechanism is introduced to capture the spatial association between nodes. The formula is as follows: γ ij =softmax j (LeakyReLU(a T [WH i ||WH j ])); Among them: a is the attention vector, || represents the concatenation operation of the vector, LeakyReLU is an activation function, softmax j It is a node A function to normalize the neighbor nodes of ; For each node and its neighbors Calculate the attention score γ ij ; Attention score γ of node i i Obtained by summing the attention scores of all its neighboring nodes; Final node representation with attention The calculation is as follows: The number of GCN layers is set to 2, and the number of hidden units is 100. After each layer of MR-GCN, the scene graph G ml The node embedding set is denoted as X ml ; Through the graph read operation, obtain each G ml The graph-level embedding H G , the readout operation is sum pooling, mean pooling or max pooling, which corresponds to summing, averaging or selecting the maximum value of all node embeddings in each feature dimension respectively.

9. The method for interpretable prediction of driving stress based on a hierarchical road environment scene graph according to claim 8, characterized in that: In the temporal attention-based LSTM model, the graph-level embedding sequence within time period T is represented as This sequence is input into the attention-based LSTM model; an LSTM unit consists of a forget gate, an input gate, and an output gate. For each timestamp t, the LSTM updates its unit state C t and the hidden state p t , calculate the company as follows: Add an attention layer to the LSTM and use the attention weight δ t Calculate the weighted context vector c t , the calculation formula is as follows: The hidden state p of LSTM t and cell status C t Based on the context vector c t , input graph embedding As well as the previous hidden state and unit state, the formula is as follows: After LSTM processes all scene graph embeddings, the resulting output sequence (ie, p1, p2, ..., p T ) performs a temporal readout operation to calculate the final spatiotemporal embedding Z; In the support vector machine, a support vector machine with a radial basis function kernel is used to predict driving stress Y based on the spatiotemporal embedding Z; driving stress prediction is a binary classification task, where Y=1 indicates that the driver is experiencing driving stress and Y=0 indicates no stress.

10. The method for interpretable prediction of driving stress based on a hierarchical road environment scene graph according to claim 1, characterized in that: The attention-enhanced spatiotemporal graph convolutional LSTM network was developed using the PyTorch deep learning framework. Adam was selected as the network optimizer, and three initial learning rates were tested: 0.0001, 0.0005, and 0.

001. The training process lasted for 300 epochs, and the batch size was set to 16. The dataset was randomly divided into a training set and a test set with a ratio of 3:

1. The model was first trained on the training set and then evaluated on the test set. The effectiveness of the model was evaluated by the accuracy and the area under the ROC curve.

Images