A pedestrian crossing intention prediction method based on human-vehicle interaction modeling and multimodal fusion

Abstract

This invention discloses a method for predicting pedestrian crossing intentions based on human-vehicle interaction modeling and multimodal fusion, belonging to the field of intelligent transportation environment perception technology. It includes: S1, data acquisition; S2, data processing; S3, data partitioning; S4, model construction; S5, model training; and S6, model testing and visualization. This invention quantifies the dynamic interaction relationship between pedestrians and autonomous vehicles through human-vehicle interaction modeling, and effectively improves the accuracy of the model in predicting pedestrian crossing intentions through multimodal fusion based on an attention mechanism.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation environment perception technology, and more specifically to a method for predicting pedestrian crossing intentions based on human-vehicle interaction modeling and multimodal fusion. Background Technology

[0002] In recent years, autonomous driving technology, as a key component of intelligent transportation systems, has developed rapidly and is considered an important way to improve road traffic safety and operational efficiency. Compared to human drivers, autonomous vehicles exhibit higher stability and consistency in perception, decision-making, and control. However, autonomous vehicles still face many challenges in interacting with pedestrians, especially in predicting whether pedestrians intend to cross the street in the short term.

[0003] Pedestrian crossing behavior is influenced by a combination of factors, including vehicle speed, traffic rules, and individual risk preferences, resulting in high uncertainty and individual variability. Therefore, in scenarios with significant risks of human-vehicle conflicts, such as intersections without traffic lights or unauthorized crossings, accurately predicting pedestrians' crossing intentions is crucial for autonomous vehicles to develop appropriate response strategies and improve the safety and efficiency of pedestrian interactions.

[0004] Early pedestrian crossing prediction methods often used motion feature extraction (such as movement trajectory and posture) in conjunction with traditional machine learning models to identify pedestrians' crossing intentions. These methods have advantages such as low computational cost and strong interpretability, but they are difficult to capture the complex nonlinear relationships and contextual semantics in high-dimensional image data.

[0005] With the development of deep learning technology, various deep learning models have been widely used in pedestrian intention prediction tasks. These models typically predict pedestrians' intention to cross the street by extracting and encoding information such as pedestrian motion features, external environmental features (such as traffic signals, road attributes, and weather conditions), and local visual features of pedestrians (such as body orientation and hand gestures in images).

[0006] However, existing research still has significant shortcomings in modeling the dynamic interactions between pedestrians and vehicles. When pedestrians decide whether to cross the road in unsignalized areas, their decisions are largely influenced by the speed and distance of nearby vehicles. However, current research mostly considers vehicle speed and rarely encodes the spatial relationship between pedestrians and vehicles in the model. This oversight makes the model prone to prediction lag or misjudgment when predicting pedestrian intentions. In addition, since the input multimodal data have different structural characteristics, how to effectively fuse these multimodal features has become another major challenge facing this task.

[0007] Therefore, proposing a pedestrian crossing intention prediction method based on human-vehicle interaction modeling and multimodal fusion to solve the difficulties of existing technologies is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0008] In view of this, the present invention provides a method for predicting pedestrian crossing intentions based on human-vehicle interaction modeling and multimodal fusion. The method quantifies the dynamic interaction relationship between pedestrians and autonomous vehicles through human-vehicle interaction modeling, and effectively improves the accuracy of the model in predicting pedestrian crossing intentions through multimodal fusion based on attention mechanism.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] A pedestrian crossing intention prediction method based on human-vehicle interaction modeling and multimodal fusion includes the following steps:

[0011] S1. Data Acquisition: Acquire source data, including video data streams captured using the vehicle's front-facing high-definition camera and vehicle speed data obtained from the vehicle speed sensor.

[0012] S2. Data Processing: Preprocess the video data stream and vehicle speed data to obtain preprocessed data;

[0013] S3. Data partitioning: Divide the preprocessed data into training set, validation set and test set;

[0014] S4. Model Construction: Construct an intent prediction network model, which includes a pedestrian pose feature extraction module, an environmental context feature extraction module, a pedestrian-vehicle interaction encoder, a multimodal feature fusion module, and a classification head;

[0015] S5. Model Training: Input the training set into the intent prediction network model and train the intent prediction network model; update the network weight parameters through the loss function; after several training iterations, obtain the trained intent prediction network model and validate it using the validation set.

[0016] S6. Model Testing and Visualization: Input the test set into the trained intent prediction network model to obtain the pedestrian crossing intent prediction results, and then visualize the prediction results.

[0017] Optionally, in S2, the video data stream and vehicle speed data are preprocessed to obtain the following preprocessed data:

[0018] The Yolov5 target detection model is used to detect targets in the images in the acquired video data stream and obtain the target pedestrian bounding boxes. Based on the improved CrowdPose 2D pose estimation model and fused with the BTS depth estimation model, the 3D pose of the target pedestrian is detected according to the range of the target pedestrian bounding box.

[0019] A pre-trained DeepLabV3 model is used to process images in the video data stream to generate semantic segmentation maps;

[0020] A pre-trained ZoeDepth model is used to process images in the video data stream to generate depth maps;

[0021] The vehicle speed data is normalized and stitched together.

[0022] Optionally, an intent prediction network model is constructed in S4. This model includes a pedestrian pose feature extraction module, an environmental context feature extraction module, a pedestrian-vehicle interaction encoder, a multimodal feature fusion module, and a classification head.

[0023] The pedestrian pose feature extraction module based on adaptive graph convolution includes three cascaded graph convolution-temporal convolution units for extracting pedestrian pose features.

[0024] The environmental context feature extraction module includes two parallel channels, which are used for local scene feature extraction and global scene feature extraction, respectively.

[0025] The pedestrian-vehicle interactive encoder includes an embedding layer, an interactive coding layer, and a GRU, which is used to fuse vehicle speed data and relative position information between pedestrians and vehicles;

[0026] The multimodal feature fusion module, including the GRU-attention module and the GCN-attention module, is used to process different types of neural network channels in parallel and fuse multimodal feature vectors.

[0027] The classification head, implemented using a fully connected layer, is used to output the final prediction result of pedestrians' crossing intentions.

[0028] Optionally, the environment context feature extraction module includes two parallel channels, used for local scene feature extraction and global scene feature extraction respectively.

[0029] The input to the local scene channel is the cropped original image, while the input to the global scene channel is a resized semantic segmentation map.

[0030] Both parallel channels integrate CNN and RNN modules. The CNN module uses the VGG19 model pre-trained on the ImageNet dataset, selecting the fourth pooling layer of the VGG19 model and all convolutional and pooling layers before it. The RNN module uses a gated recurrent unit (GRU) as its core component.

[0031] Optionally, the relative position of the pedestrian and vehicle in the pedestrian-vehicle interactive encoder includes the horizontal distance represented by the pedestrian bounding box coordinates and the vertical distance represented by the depth.

[0032] Optionally, the pedestrian-vehicle interactive encoder includes an embedding layer, an interactive coding layer, and a GRU; wherein,

[0033] The embedding layer consists of three independent multilayer perceptrons;

[0034] The interactive encoding layer uses the principle of the Cross-attention mechanism to fuse the three feature vectors output by the embedding layer;

[0035] GRU is used to further capture the temporal characteristics of these interactions.

[0036] Optionally, in S5, the training set is input into the intent prediction network model, and the specific details of training the intent prediction network model are as follows:

[0037] The video data was filtered, and the valid pedestrian samples obtained from the filtering were divided into training set, validation set and test set;

[0038] Accuracy, AUC, F1 score, and precision were used as performance evaluation metrics for the intent prediction network model.

[0039] The pedestrian crossing intention prediction model is trained using the training set and validated using the validation set during the training process.

[0040] The trained model is used to perform inference on the test set. Based on the model evaluation and debugging results, the hyperparameters of the model are continuously adjusted and optimized, and finally an optimal set of hyperparameters is determined.

[0041] Optionally, in S6, the test set is input into the trained intent prediction network model to obtain the pedestrian crossing intent prediction results, and the specific content of visualizing the prediction results is as follows:

[0042] The test set is input to evaluate the best-performing intent prediction network model, and the output is the pedestrian crossing intent prediction result.

[0043] The image of the pedestrian crossing intention prediction result is displayed on the screen, the target pedestrian is marked with a bounding box, and the predicted pedestrian crossing intention result is marked with text near the bounding box.

[0044] As can be seen from the above technical solution, compared with the prior art, the present invention provides a method for predicting pedestrian crossing intentions based on human-vehicle interaction modeling and multimodal fusion, which has the following beneficial effects:

[0045] (1) This invention quantifies the dynamic interaction relationship between pedestrians and self-driving vehicles through human-vehicle interaction modeling, and effectively improves the accuracy of the model in predicting pedestrians’ intentions to cross the street through multimodal fusion based on attention mechanism;

[0046] (2) This invention combines multiple technologies such as target detection, pose estimation, semantic segmentation, and depth estimation to provide a high-precision method for predicting pedestrian crossing intentions; it constructs a pedestrian-vehicle interaction encoder, which combines information such as the speed of the autonomous vehicle, the vertical distance between the pedestrian and the vehicle, and the horizontal distance between the pedestrian and the vehicle to capture the dynamic interaction between the pedestrian and the vehicle in all aspects; it uses image depth estimation technology to obtain depth information without the need for a depth camera, which effectively reduces equipment costs; by designing different neural network algorithm channels, it extracts features for different modalities, which can extract multimodal information more efficiently; it designs a multimodal feature fusion module based on different attention mechanisms to enhance the model's ability to capture key information and its ability to fuse multimodal features. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0048] Figure 1 The flowchart of the pedestrian crossing intention prediction method based on human-vehicle interaction modeling and multimodal fusion provided by the present invention is shown below.

[0049] Figure 2 A flowchart of pedestrian detection and pose estimation, as well as cross-modal feature conversion of image data, provided by the present invention;

[0050] Figure 3 A schematic diagram of the structure of 19 pedestrian posture key points obtained by posture estimation according to the present invention;

[0051] Figure 4 This is a structural diagram of the pedestrian crossing intention prediction model provided by the present invention;

[0052] Figure 5 This is a structural diagram of the pedestrian-vehicle interactive encoder provided by the present invention;

[0053] Figure 6This is a comparison diagram of the model structure of different human-vehicle interaction feature fusion methods in the pedestrian-vehicle interaction encoder provided by the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] See Figure 1 As shown, this invention discloses a method for predicting pedestrian crossing intentions based on human-vehicle interaction modeling and multimodal fusion, comprising the following steps:

[0056] S1. Data Acquisition: Acquire source data, including video data streams captured using the vehicle's front-facing high-definition camera and vehicle speed data obtained from the vehicle speed sensor.

[0057] S2. Data Processing: Preprocess the video data stream and vehicle speed data to obtain preprocessed data;

[0058] S3. Data partitioning: Divide the preprocessed data into training set, validation set and test set;

[0059] S4. Model Construction: Construct an intent prediction network model, which includes a pedestrian pose feature extraction module, an environmental context feature extraction module, a pedestrian-vehicle interaction encoder, a multimodal feature fusion module, and a classification head;

[0060] S5. Model Training: Input the training set into the intent prediction network model and train the intent prediction network model; update the network weight parameters through the loss function; after several training iterations, obtain the trained intent prediction network model and validate it using the validation set.

[0061] S6. Model Testing and Visualization: Input the test set into the trained intent prediction network model to obtain the pedestrian crossing intent prediction results, and then visualize the prediction results.

[0062] Specifically, the S1 uses a 1920×1080 resolution in-vehicle front-facing high-definition camera, and is also equipped with two types of vehicle speed sensors, including an on-board diagnostic system sensor and a GPS sensor.

[0063] Furthermore, in S2, the video data stream and vehicle speed data are preprocessed to obtain the following preprocessed data:

[0064] The Yolov5 target detection model is used to detect targets in the images in the acquired video data stream and obtain the target pedestrian bounding boxes. Based on the improved CrowdPose 2D pose estimation model and fused with the BTS depth estimation model, the 3D pose of the target pedestrian is detected according to the range of the target pedestrian bounding box.

[0065] A pre-trained DeepLabV3 model is used to process images in the video data stream to generate semantic segmentation maps;

[0066] A pre-trained ZoeDepth model is used to process images in the video data stream to generate depth maps;

[0067] The vehicle speed data is normalized and stitched together.

[0068] Specifically, for vehicle speed data, if a single sensor is used for collection, the data is directly scaled and normalized; if multiple sensors are used for collection, the data collected by each sensor is first normalized separately, and then the normalized data is stitched together and integrated into one dimension.

[0069] Specifically, such as Figure 2 As shown, the pedestrian bounding box in pedestrian detection and pose estimation is a rectangular box that encloses the complete human body. The detected bounding box information has a dimension of 30×4, where 30 represents the number of video frames and 4 represents the x and y pixel coordinates of the top-left and bottom-right corners of the bounding box in the image. The 19 pedestrian pose keypoints obtained through pose estimation are shown below. Figure 3 As shown.

[0070] Specifically, the target detection method using the Yolov5 target detection model for human detection and pose estimation is as follows:

[0071] Yolov5 is used to perform target detection on independent image frames extracted from video clips to obtain the bounding box annotation information of all targets identified as pedestrians in the image;

[0072] Using the inference scripts provided by CrowdPose, pose estimation is performed on all annotated images. CrowdPose records the bounding box coordinates, 2D pose keypoint coordinates, and confidence scores of all detected pedestrians in each image.

[0073] The observation time window for all pedestrians was set to one second, and the posture information of each pedestrian sample was organized into a sequence of posture key points from 2 seconds to 1 second before the prediction time point.

[0074] The BTS depth estimation model is adopted, and the corresponding depth value is matched according to the pixel position of each 2D key point. The value is then integrated into the third dimension of the pedestrian pose data, resulting in 3D pedestrian pose data with dimensions of 30×19×4. Here, 30 represents the number of video frames, 19 corresponds to the total number of pose key points, and 4 represents the three-dimensional spatial coordinates and confidence score of each key point.

[0075] Specifically, the cross-modal feature transformation of image data is performed using pre-trained DeepLabV3 and ZoeDepth models as follows:

[0076] Using a DeepLabV3 model pre-trained on the Cityscapes dataset, each pixel in the image is classified into 19 different categories to generate a semantic segmentation map of the image;

[0077] Key traffic objects were selected and other non-key or similar objects were merged. The scene was finally simplified into four categories: roads, pedestrians, vehicles and background. The semantic segmentation maps of all image frames were uniformly adjusted to a size of 224×224 pixels.

[0078] The trained ZoeD-M12-NK model was used to estimate the depth of the image sequence for each pedestrian sample within a set observation time window. The model was first pre-trained on 12 datasets for relative depth estimation, and then fine-tuned on the NYUdepth v2 and KITTI datasets for absolute depth.

[0079] Select the depth value corresponding to the center point of the target pedestrian and use it as the vertical distance between the pedestrian and the self-driving vehicle. Then, organize the depth information into an input format with a size of 30×1 dimension.

[0080] Furthermore, such as Figure 4 As shown, an intent prediction network model is constructed in S4. This model includes a pedestrian pose feature extraction module, an environmental context feature extraction module, a pedestrian-vehicle interaction encoder, a multimodal feature fusion module, and a classification head.

[0081] The pedestrian pose feature extraction module based on adaptive graph convolution includes three cascaded graph convolution-temporal convolution units for extracting pedestrian pose features.

[0082] The environmental context feature extraction module includes two parallel channels, which are used for local scene feature extraction and global scene feature extraction, respectively.

[0083] The pedestrian-vehicle interactive encoder includes an embedding layer, an interactive coding layer, and a GRU, which is used to fuse vehicle speed data and relative position information between pedestrians and vehicles;

[0084] The multimodal feature fusion module, including the GRU-attention module and the GCN-attention module, is used to process different types of neural network channels in parallel and fuse multimodal feature vectors.

[0085] The classification head, implemented using a fully connected layer, is used to output the final prediction result of pedestrians' crossing intentions.

[0086] Furthermore, the environmental context feature extraction module includes two parallel channels, specifically for local scene feature extraction and global scene feature extraction, respectively.

[0087] The input to the local scene channel is the cropped original image, while the input to the global scene channel is a resized semantic segmentation map.

[0088] Both parallel channels integrate CNN and RNN modules. The CNN module uses the VGG19 model pre-trained on the ImageNet dataset, selecting the fourth pooling layer of the VGG19 model and all convolutional and pooling layers before it. The RNN module uses a gated recurrent unit (GRU) as its core component.

[0089] Furthermore, the relative positions of the pedestrian and vehicle in the pedestrian-vehicle interactive encoder include the horizontal distance represented by the pedestrian bounding box coordinates and the vertical distance represented by the depth.

[0090] Furthermore, the pedestrian-vehicle interactive encoder includes an embedding layer, an interactive coding layer, and a GRU; among which,

[0091] The embedding layer consists of three independent multilayer perceptrons;

[0092] The interactive encoding layer uses the principle of the Cross-attention mechanism to fuse the three feature vectors output by the embedding layer;

[0093] GRU is used to further capture the temporal characteristics of these interactions.

[0094] Specifically, for each graph convolution-temporal convolution unit in the pedestrian pose feature extraction module, the input is a 3D pedestrian pose sequence. The output is the spatiotemporal features of pedestrian pose. In this unit, the adaptive adjacency matrix of 3D is first applied based on the pedestrian pose keypoint connection graph. Parameter initialization: All connection weights and directions between nodes are set to learnable parameters; among them, In this context, 3 represents the 3D adjacency matrix, and N represents the number of keypoints for each pedestrian pose; subsequently, ... Perform L2 regularization to avoid and Multiplication changes the proportion of the eigenvectors.

[0095] Next, using the spectral propagation rule of graph convolution, a linear transformation is performed on the scalar product of the input feature map and the adjacency matrix to obtain the output of the graph convolution layer:

[0096]

[0097] in, Let be the adjacency matrix of the i-th graph convolutional layer; The input feature map is the i-th graph convolutional layer; Let be the linear transformation weight matrix of the i-th graph convolutional layer.

[0098] By introducing a residual connection mechanism, the input signal can be directly transmitted across layers, mitigating the gradient vanishing or exploding problem caused by increasing network depth. Building upon this, a batch normalization layer and a ReLU activation function are added, together forming the final output of the adaptive graph convolutional layer.

[0099]

[0100] in, is the ReLU activation function; BN is the batch normalization layer.

[0101] The output feature map of the adaptive graph convolutional layer is convolved in the time dimension to obtain the output of the temporal convolutional layer:

[0102]

[0103] in, For the first i The output of the layer graph convolutional layer; For the first i The linear transformation weight matrix of the time-series convolutional layer.

[0104] The output feature map of the temporal convolutional layer is batch normalized and residually connected, and then activated by ReLU to obtain the output of the graph convolution-temporal convolution unit.

[0105]

[0106] Three graph convolutional-temporal convolutional units are concatenated sequentially, with their output channel numbers configured to 32, 64, and 128 respectively. Dropout at a ratio of 0.25 is applied before the module header to mitigate overfitting and enhance the model's generalization ability. For input 3D pedestrian pose data with dimensions 30×19×4, the pedestrian pose feature extraction module processes the data to obtain a final output feature map with dimensions 30×19×128.

[0107] Specifically, for the environmental context feature extraction module, two parallel channels were designed to process the local scene presented as a cropped image and the global scene presented as a semantic segmentation map, respectively. When inputting into the module, both the cropped image and the semantic segmentation map were uniformly adjusted to a size of 224×224. Each channel integrates a CNN and an RNN module, where the CNN module is used to extract feature details from the image, and the RNN module is used to capture the temporal contextual dependencies between consecutive image feature maps.

[0108] In the CNN module, the VGG19 model pre-trained on the ImageNet dataset is used for image feature extraction. The fourth pooling layer of this model and all the convolutional and pooling layers before it are selected to construct the CNN module. Finally, the module outputs a feature vector of size 30×512.

[0109] In the RNN module, a unidirectional GRU is used, with its hidden layer size set to 128. During processing, feature vectors extracted from local and global scene information are used as inputs and fed into the corresponding GRUs. After GRU processing, a hidden layer output vector sequence with a dimension of 30×128 and a final time step hidden layer output vector with a dimension of 1×128 are obtained from two channels.

[0110] Specifically, such as Figure 5 As shown, the pedestrian-vehicle interaction encoder integrates the speed information of the autonomous vehicle with the horizontal and vertical distances between the pedestrian and the vehicle to model the dynamic interaction between them. In this encoder, features are first extracted from the input using an embedding layer. The embedding layer consists of three independent MLPs used to extract features from the autonomous vehicle's speed V. i Vertical distance D i and horizontal distance H i Features are extracted from the input layer. Each MLP includes two linear transformation processes, sequentially connecting the input layer to the hidden layer and the hidden layer to the output layer. After processing by their respective MLPs, V... i Di and H i Output feature vectors of sizes 64, 32, and 32 respectively. The specific calculation formula for MLP is:

[0111]

[0112] in, This is the output feature vector of the embedding layer; and These are the weight matrices from the input layer to the hidden layer and from the hidden layer to the output layer of the MLP, respectively. and These are the bias vectors for the hidden layer and the output layer, respectively; ReLU is the activation function for the hidden layer; LN represents layer normalization, used to stabilize the training process of the neural network.

[0113] Subsequently, the feature vectors generated by the embedding layer enter the core of this encoder, namely, the interactive encoding layer. This encoding layer, based on the cross-attention mechanism, mines the latent relationships between input features. First, weights are assigned to these features to promote the network's adaptive learning. Specifically, the embedding vector of the autonomous vehicle speed is linearly transformed to obtain α; the embedding vectors of the vertical and horizontal distances are concatenated and then subjected to two different linear transformations to obtain β and γ, each with different weights and biases, calculated using the following formula:

[0114]

[0115] Where α, β, and γ are the output feature vectors of different linear transformations; W (1) W (2) and W (3) b is the weight matrix for the linear transformation; (1) b (2) and b (3) V is the deviation vector of the linear transformation; v V d and V h V i D i and H i The vector obtained after the embedding layer; concat is the vector concatenation operation.

[0116] The calculated α, β, and γ dimensions are consistent, denoted as hs. Next, a scaled dot product is performed on α and β, and the result is divided by... To prevent numerical instability and gradient vanishing issues that might arise from excessively large dot product results, the scaled result is then multiplied by γ and normalized using the Softmax function. Next, a Dropout layer with a ratio of 0.5 is introduced. Finally, a GRU is used to extract temporal features, yielding the output feature vector of the pedestrian-vehicle interactive encoder. The calculation formula is:

[0117]

[0118] in, This is the output of the pedestrian-vehicle interactive encoder; α is the transpose of β; hs is the dimension of α, β and γ.

[0119] Specifically, for the multimodal feature fusion module, this invention designs two attention modules, GRU-attention and GCN-attention, to process different types of neural network channels in parallel in order to achieve efficient feature fusion.

[0120] The GRU-attention module is applied to process the feature set of the GRU output, including the extracted environmental context features and the output of the pedestrian-vehicle interaction encoder. This module is embedded after each GRU unit. GRU-attention allows the model to effectively capture important information in the sequence data by dynamically assigning attention weights to relevant parts of the input sequence. Its output vector is calculated using the following formula:

[0121]

[0122] in, h is the output vector of the GRU-attention module. s The sequence of hidden layer output vectors of a GRU can be represented as: h e This is the hidden layer output vector at the final time step of the GRU.

[0123] The GCN-attention module processes the features output by the pedestrian pose feature extraction module. Unlike GRU-attention, which focuses on capturing key information from the contextual hidden states of sequential data, the GCN-attention module, based on the self-attention mechanism, has the unique advantage of directly extracting key information from the input data. In the GCN-attention module, the dimensions of the pose features are first rearranged to 128×570. Then, three independent two-dimensional convolution operations are used to generate Q, K, and V vectors, calculated as follows:

[0124]

[0125] Where Conv2d_q, Conv2d_k, and Conv2d_v are the two-dimensional convolutions that generate Q, K, and V vectors, respectively; x p Input pedestrian pose features.

[0126] Next, the Softmax function, combined with dot product operations, is used to aggregate the three vectors Q, K, and V. Furthermore, a learnable decay parameter is introduced. The information flow is controlled through a gating mechanism. Finally, the result is added to the input features to obtain the output vector of the GCN-attention module. The calculation formula for this process is as follows:

[0127]

[0128] in, This is the output vector of the GCN-attention module; This is a learnable decay parameter.

[0129] Furthermore, in S5, the training set is input into the intent prediction network model, and the specific details of training the intent prediction network model are as follows:

[0130] The video data was filtered, and the valid pedestrian samples obtained from the filtering were divided into training set, validation set and test set;

[0131] Accuracy, AUC, F1 score, and precision were used as performance evaluation metrics for the intent prediction network model.

[0132] The pedestrian crossing intention prediction model is trained using the training set and validated using the validation set during the training process.

[0133] The trained model is used to perform inference on the test set. Based on the model evaluation and debugging results, the hyperparameters of the model are continuously adjusted and optimized, and finally an optimal set of hyperparameters is determined.

[0134] Specifically, regarding the data filtering principle, this invention filters based on the length of the pedestrian sequence, setting the time elapsed from the last observed frame to the start of the pedestrian crossing event to 1 to 2 seconds, i.e., 30 to 60 frames, and setting the length of the sliding observation window to 1 second, excluding samples with a pedestrian sequence length of less than 3 seconds.

[0135] Specifically, in the model training phase, binary cross-entropy loss is chosen as the classification loss function, and the AdamW optimizer is employed, along with a 1×10⁻⁶ ohmmeter. -3 The weight decay coefficient is determined, and a OneCycleLR learning rate strategy is introduced, setting the initial learning rate to 2.1 × 10⁻⁶. -4 The entire training process is set to an iteration cycle of 50, with 64 samples processed in each batch.

[0136] The PIE dataset was selected as the target dataset for model training and evaluation. The PIE dataset contains 6 hours of high-definition driving video, covering data from 1842 pedestrians, and providing detailed crossing information for each pedestrian. Pedestrian crossing intentions were categorized into three labels: crossing the street, not crossing the street, and irrelevant pedestrians. To simplify the classification problem, this invention classifies the "irrelevant" category into the "not crossing the street" category, and the model's final prediction result is a binary output.

[0137] Specifically, Table 1 shows the comparison results of the impact of different channels on model performance on the PIE dataset.

[0138] Table 1. Comparison of the impact of different channels on model performance on the PIE dataset.

[0139]

[0140] Table 1 presents the impact of different channels (including pedestrian pose, local scene cropped image, global scene semantic segmentation map, and pedestrian-vehicle interaction information) on model performance. Using pedestrian pose as the fixed baseline input, experiments were conducted by sequentially disabling one or more other channels. The results show that using only one channel results in poor model performance. Combining two channels significantly improves model performance. The model achieving optimal performance is the one that fully integrates all four channel types. Based on this, the model with all four channel types was chosen.

[0141] Specifically, the performance evaluation results of the pedestrian-vehicle interactive encoder are shown in Table 2 and... Figure 6 As shown, a comparative analysis was conducted on the pedestrian-vehicle interactive encoder and three different feature fusion methods. The first method employed a squeezing and excitation network, performing 2D pooling operations on the vertical and horizontal distances and adding a Sigmoid activation function, followed by multiplication with the autonomous vehicle speed, as shown. Figure 6 As shown in (a). The second method involves concatenating the vertical and horizontal distances into a feature vector, and then performing matrix multiplication with the vehicle speed, as shown in (a). Figure 6 As shown in (b). The third method transforms the feature dimension of the autonomous vehicle's speed through a pooling layer to align it with the feature dimensions of vertical and horizontal distances, and then uses matrix element multiplication for fusion, as shown in (b). Figure 6 As shown in (c). The last method is the interactive coding layer in the pedestrian-vehicle interactive encoder proposed in this invention, the structure of which is as follows: Figure 6 As shown in (d).

[0142] Table 2 Comparison of different feature fusion methods for pedestrian-vehicle interaction information

[0143]

[0144] The results show that method (a) performs the worst, mainly due to the loss of feature information during the squeezing operation, which weakens the model's expressive power. The fusion strategy of methods (b) and (c) improves performance and outperforms method (a). Method (d) demonstrates the best performance on all evaluation metrics, especially achieving significant advantages in accuracy and F1 score.

[0145] Furthermore, by comparing and analyzing the impact of different combinations of inputs—vehicle speed, vertical distance, and horizontal distance—on model performance, the effectiveness of pedestrian-vehicle interaction information in the model was explored in depth. Table 3 shows that the model achieved optimal performance when all information sources were used simultaneously. This result indicates that using only one type of interaction information cannot fully capture the complex relationship between pedestrians and vehicles, while combining vehicle speed, vertical distance, and horizontal distance can effectively uncover their inherent correlations, thereby improving the model's predictive ability.

[0146] Table 3. Comparison of the impact of different types of pedestrian-vehicle interaction information on model performance

[0147]

[0148] Specifically, a comparative experiment was conducted between commonly used feature fusion methods and the multimodal feature fusion module based on different attention mechanisms proposed in this invention. Table 4 shows the performance of the model when using different feature fusion strategies.

[0149] Table 4 Comparison of the impact of different feature fusion strategies on model performance

[0150]

[0151] Furthermore, in S6, the test set is input into the trained intent prediction network model to obtain pedestrian crossing intent prediction results, and the specific content of visualizing the prediction results is as follows:

[0152] The test set is input to evaluate the best-performing intent prediction network model, and the output is the pedestrian crossing intent prediction result.

[0153] The image of the pedestrian crossing intention prediction result is displayed on the screen, the target pedestrian is marked with a bounding box, and the predicted pedestrian crossing intention result is marked with text near the bounding box.

[0154] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0155] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A pedestrian crossing intention prediction method based on human-vehicle interaction modeling and multi-modal fusion, characterized in that, The method comprises the following steps: S1, data collection: collecting source data, the source data including capturing video data stream using a front-mounted high-definition camera on the vehicle and obtaining vehicle speed data based on a vehicle speed sensor; S2, data processing: preprocessing the video data stream and the vehicle speed data to obtain preprocessed data; S3, data division: dividing the preprocessed data into a training set, a validation set, and a test set; S4, model construction: constructing an intention prediction network model, the intention prediction network model including a pedestrian posture feature extraction module, an environment context feature extraction module, a pedestrian-vehicle interaction encoder, a multi-modal feature fusion module, and a classification head; S5, model training: inputting the training set into the intention prediction network model to train the intention prediction network model; updating the network weight parameters through a loss function, and obtaining a trained intention prediction network model after several training times, and verifying the trained intention prediction network model using the validation set; S6, model testing and visualization: inputting the test set into the trained intention prediction network model to obtain a pedestrian crossing intention prediction result, and visualizing the prediction result; In S4, the intention prediction network model includes a pedestrian posture feature extraction module, an environment context feature extraction module, a pedestrian-vehicle interaction encoder, a multi-modal feature fusion module, and a classification head; wherein, The pedestrian posture feature extraction module based on adaptive graph convolution includes three graph convolution-time convolution units connected in series, and is used for extracting pedestrian posture features; The environment context feature extraction module includes two parallel channels, which are respectively used for local scene feature extraction and global scene feature extraction; The pedestrian-vehicle interaction encoder includes an embedding layer, an interaction encoding layer, and a GRU, and is used for fusing vehicle speed data and relative position information between the pedestrian and the vehicle; The relative position between the pedestrian and the vehicle in the pedestrian-vehicle interaction encoder includes a horizontal distance represented by pedestrian bounding box coordinates and a vertical distance represented by depth; The multi-modal feature fusion module includes a GRU-attention attention module and a GCN-attention attention module, and is used for parallel processing of different types of neural network channels and fusing multi-modal feature vectors; The classification head is realized by a fully connected layer, and is used for outputting a final pedestrian crossing intention prediction result; The pedestrian-vehicle interaction encoder includes an embedding layer, an interaction encoding layer, and a GRU; wherein, The embedding layer is composed of three independent multilayer perceptrons, which are respectively used for extracting vectors from the vehicle speed, the vertical distance, and the horizontal distance to obtain three feature vectors; The interaction encoding layer uses the principle of Cross-attention mechanism to fuse the three feature vectors output by the embedding layer; The GRU is used to extract time sequence features to obtain an output feature vector of the pedestrian-vehicle interaction encoder.

2. The pedestrian crossing intention prediction method based on human-vehicle interaction modeling and multi-modal fusion according to claim 1, wherein In S2, the specific content of preprocessing the video data stream and the vehicle speed data to obtain preprocessed data is: The Yolov5 target detection model is used for target detection on the image in the collected video data stream, and a target pedestrian bounding box is obtained; based on the improved CrowdPose 2D pose estimation model, the BTS depth estimation model is fused, and the 3D pose of the target pedestrian is detected according to the target pedestrian bounding box range; The pre-trained DeepLabV3 model is used for image processing in the video data stream to generate a semantic segmentation map; The pre-trained ZoeDepth model is used for image processing in the video data stream to generate a depth map; The speed data of the ego vehicle is normalized and spliced.

3. The pedestrian crossing intention prediction method based on human-vehicle interaction modeling and multi-modal fusion according to claim 1, characterized in that an environmental context feature extraction module includes two parallel channels for local scene feature extraction and global scene feature extraction, and the specific content is: The input of the local scene channel is the cropped original image, and the input of the global scene channel is the size-adjusted semantic segmentation map; Both parallel channels fuse CNN modules and RNN modules, wherein the CNN module uses the pre-trained VGG19 model on the ImageNet dataset, and selects the fourth pooling layer and all convolutional and pooling layers before it of the VGG19 model; the RNN module uses a gated recurrent unit GRU as a core component.

4. The pedestrian crossing intention prediction method based on human-vehicle interaction modeling and multi-modal fusion according to claim 1, characterized in that the specific content of training the intention prediction network model by inputting the training set in S5 is: Filter the video data, and divide the effective pedestrian samples obtained by filtering into a training set, a validation set, and a test set; Use accuracy, AUC, F1 score, and precision as performance evaluation indicators of the intention prediction network model; Use the training set to train the constructed pedestrian crossing intention prediction model, and use the validation set to verify during the training process; Use the trained model to perform inference on the test set, continuously adjust and optimize the hyperparameters of the model according to the model evaluation and debugging results, and finally determine a set of optimal hyperparameter combinations.

5. The pedestrian crossing intention prediction method based on human-vehicle interaction modeling and multi-modal fusion according to claim 1, characterized in that the specific content of inputting the test set into the trained intention prediction network model in S6 to obtain pedestrian crossing intention prediction results and visually presenting the prediction results is: Input the test set into the intention prediction network model with the optimal performance, and output the pedestrian crossing intention prediction results; Display the pedestrian crossing intention prediction results on the display, mark the target pedestrian with a bounding box, and mark the predicted pedestrian crossing intention results with text near the bounding box. ​ ​ ​

Images