Pedestrian crossing intention prediction method, model and device based on double channels

Abstract

The application relates to a kind of pedestrian crossing intention prediction method, model and device based on double channel, method includes: obtaining and pedestrian crossing related observation frame;Pedestrian skeleton information X of target pedestrian is extracted from the observation frame using posture estimation extraction algorithm SimCC;Extract the space-time behavior feature Ks from the pedestrian skeleton information X;From the last picture of the observation frame, the traffic scene object interaction relationship feature Us related to the target pedestrian is extracted;Robust pedestrian crossing intention prediction is realized by using adaptive average pooling layer AAP to deeply fuse the space-time behavior feature Ks and the traffic scene object interaction relationship feature Us.The application can accurately predict the crossing intention of pedestrian.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of crossing intention prediction, and particularly relates to a pedestrian crossing intention prediction method based on a double channel. BACKGROUND

[0002] Human-vehicle interaction is a hot issue in the field of intelligent transportation. Effective human-vehicle interaction can prevent human-vehicle conflicts and protect the safety of traffic participants. In complex urban roads, especially on both sides of roads without crossroads, vehicles cannot make scientific decisions in time when facing sudden behaviors of pedestrians during driving. At the same time, pedestrians will change their original behavior actions and movement directions in a very short time when encountering unexpected situations during crossing, resulting in complex and unpredictable crossing behaviors. If the driver misjudges the crossing behavior of the pedestrian, it will cause traffic congestion and property loss, and even threaten the life of the pedestrian.

[0003] Therefore, accurately predicting the crossing intention of the pedestrian and assisting the driver to slow down and brake can protect the safety of the pedestrian and provide a more comfortable driving experience. It is urgent to design a method that can accurately predict the crossing intention of the pedestrian. SUMMARY

[0004] The technical problem to be solved by the present application is to overcome the defects of the prior art and provide a pedestrian crossing intention prediction method based on a double channel, which can accurately predict the crossing intention of the pedestrian.

[0005] To solve the above technical problems, the technical scheme of the present application is as follows: a pedestrian crossing intention prediction method based on a double channel, comprising:

[0006] Obtaining an observation frame related to pedestrian crossing;

[0007] Extracting pedestrian skeleton information X of a target pedestrian from the observation frame by using a pose estimation extraction algorithm SimCC;

[0008] Extracting a spatiotemporal behavior feature Ks from the pedestrian skeleton information X;

[0009] Extracting a traffic scene object interaction relationship feature Us related to the target pedestrian from the last frame of the observation frame;

[0010] Deeply fusing the spatiotemporal behavior feature Ks and the traffic scene object interaction relationship feature Us by using an adaptive average pooling layer AAP to realize robust pedestrian crossing intention prediction.

[0011] Further, the step of extracting the spatiotemporal behavior feature Ks from the pedestrian skeleton information X comprises:

[0012] Obtaining the spatiotemporal behavior feature Ks by using two layers of multi-scale graph convolution layers MS_GCL in sequence.

[0013] Furthermore, the extraction process of each multi-scale graph convolution layer MS_GCL includes:

[0014] Use the multi-scale channel refinement graph convolutional layer MCR_GCL to extract spatial features from the input information;

[0015] The multi-scale temporal convolutional layer MS_TCL is used to enrich the temporal information of the spatial features.

[0016] Furthermore, the multi-scale channel refinement graph convolutional layer MCR_GCL is used to extract spatial features from the input information, including:

[0017] First parameterize a shared adjacency matrix Serves as the prior spatial matrix for all channels and provides the intrinsic spatial properties between joint points;

[0018] Then, the spatial features between the joints of different channels in different motion modes are learned through the multi-scale refined dynamic shared adjacency matrix D;

[0019] Finally, the spatial features of each channel and the input data T(X) after feature transformation are aggregated;

[0020] The specific formula is:

[0021]

[0022] f MCR_GCL is the output of the multi-scale channel refinement graph convolutional layer MCR_GCL, represents the weight of the multi-scale channel refinement graph convolutional layer MCR_GCL, λ represents the trainable parameter, and σ is the sigmoid() function.

[0023] Furthermore, the calculation formula of the multi-scale temporal convolution layer MS_TCL is:

[0024]

[0025] Among them, W i TCL Represents the weights of the multi-scale temporal convolution layer MS_TCL.

[0026] Furthermore, extracting the traffic scene object interaction relationship feature Us related to the target pedestrian from the last frame of the observation frame includes:

[0027] First, use the feature pyramid to extract the image features of the last frame of the observation frame;

[0028] Then, the attribute network is used to extract attribute features based on the visual features of each detected object O in the image features.

[0029] Then use the interaction relationship fusion function U i,j (·) Take any two objects O i and O j Attribute feature A i and A j As input, output relation features in,

[0030] R i,j (A i ,A j )=ReLU(W i A i +W j A j )-(W i A i -W j A j )⊙(W i A i -W j A j )

[0031] Indicates O i , O j Feature space mapping of the target object;

[0032] Finally, all the relationship features R i,j (A i ,A j ) aggregation to generate object interaction relationship features

[0033] The specific formula is:

[0034]

[0035] represents the interaction relationship fusion, f ij Indicates O i , O j The relationship distribution vector between target object categories, m represents the number of objects O.

[0036] Furthermore, the adaptive average pooling layer AAP is used to deeply fuse the spatiotemporal behavior feature Ks and the traffic scene object interaction relationship feature Us to achieve robust pedestrian crossing intention prediction, including:

[0037] First, the adaptive average pooling layer AAP is used to pool the traffic scene object interaction relationship features Us and normalize them using sigmoid();

[0038] Then, the attention mechanism is used to focus on the spatiotemporal behavior feature Ks, so that the spatiotemporal behavior feature Ks has a greater weight;

[0039] Based on the output of the attention mechanism, the traffic scene object interaction relationship feature Us and the spatiotemporal behavior feature Ks are fused to obtain the fusion feature in,

[0040]

[0041] Finally, the fully connected layer is used to predict the target pedestrian’s crossing intention Y P ;in,

[0042] Set the threshold to 0.5, and after discretization, P P Mapped to Y P ;

[0043] SiLU(·) represents the activation function, Att(·) represents the attention-based transformation function, and W a represents the learnable attention weight, and Linear(·) is used to classify and output the probability of crossing / no crossing intention.

[0044] SiLU(·) represents the activation function, Att(·) represents the attention-based transformation function, and W a represents the learnable attention weight, and Linear(·) is used to classify and output the crossing / no crossing intention label.

[0045] The present invention also provides a dual-channel-based pedestrian crossing intention prediction model, which is used to implement a dual-channel-based pedestrian crossing intention prediction method.

[0046] Furthermore, the model uses the loss function Loss during the training process;

[0047] Loss=-Y T ·log(P P )-(1-Y T )·log(1-P P )

[0048] Y T represents the true label of the pedestrian sample S, P P Represents the model's predicted probability of pedestrian crossing intention.

[0049] The present invention also provides a pedestrian crossing intention prediction device based on a dual channel, which is characterized in that:

[0050] include:

[0051] An acquisition module, used to acquire observation frames related to pedestrian crossing;

[0052] A posture estimation and extraction module is used to extract the pedestrian skeleton information X of the target pedestrian from the observation frame using the posture estimation and extraction algorithm SimCC;

[0053] A pedestrian action encoding module is used to extract spatiotemporal behavior features Ks from the pedestrian skeleton information X;

[0054] A traffic scene object interaction module is used to extract the traffic scene object interaction relationship feature Us related to the target pedestrian from the last frame of the observation frame;

[0055] The intention prediction module is used to use the adaptive average pooling layer AAP to deeply fuse the spatiotemporal behavior features Ks and the traffic scene object interaction relationship features Us to achieve robust pedestrian crossing intention prediction.

[0056] After adopting the above technical solution, the present invention has the following beneficial effects:

[0057] (1) The present invention combines the temporal and spatial behavior characteristics of pedestrians with the interactive relationship characteristics of traffic scene objects to accurately predict pedestrian crossing intentions;

[0058] (2) To address the problems of incomplete human skeletons and single graph convolutional network structures in low-resolution images, we use the human joint vector representation method and the multi-scale graph convolutional layer MS_GCL to learn the relative position information and time change information of the skeleton between different channels, respectively, to achieve the motion characteristics of different motion modes of pedestrians in low-resolution images, and alleviate the challenges of low-resolution images for action recognition;

[0059] (3) To address the problem of superficial scene understanding, the interaction between target pedestrians and related traffic objects is established based on the local context and global semantic graph, which allows for an in-depth understanding of high-level semantic clues in traffic scenes and improves the accuracy of pedestrian crossing intentions. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 4 is a structural diagram of a dual-channel pedestrian crossing intention prediction model according to the present invention;

[0061] Figure 2 This is a sampling diagram of pedestrian samples in the dataset;

[0062] Figure 3 Pedestrian pose estimation results in four complex traffic environments in the JAAD and PIE datasets;

[0063] Figure 4 These are the qualitative results of the model in this invention, PCPA, and Pedestrian Graph Plus models on the JAAD and PIE datasets. DETAILED DESCRIPTION

[0064] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.

[0065] Example 1

[0066] like Figure 1 As shown, a dual-channel pedestrian crossing intention prediction method includes:

[0067] Obtain observation frames related to pedestrian crossing;

[0068] Extracting pedestrian skeleton information X of the target pedestrian from the observation frame using a posture estimation extraction algorithm SimCC;

[0069] Extracting spatiotemporal behavior features Ks from the pedestrian skeleton information X;

[0070] Extracting the traffic scene object interaction relationship feature Us related to the target pedestrian from the last frame of the observation frame;

[0071] The adaptive average pooling layer AAP is used to deeply fuse the spatiotemporal behavior features Ks and the traffic scene object interaction relationship features Us to achieve robust pedestrian crossing intention prediction.

[0072] like Figure 1 As shown in the figure, F1…Fn are the pictures in the observation frame, Fn is the last frame of the input picture, and P1…Pt are the key point information extracted after SimCC, that is, the pedestrian skeleton information X.

[0073] Pedestrian pose estimation is fundamental to predicting pedestrian crossing intentions, and the pedestrian skeleton provides the most accurate description of their pose. Accurately identifying pedestrian skeletons in dynamic environments and complex backgrounds not only impacts pedestrian action recognition results but also the subsequent prediction of pedestrian crossing intentions. However, in complex traffic scenarios, due to the large number of people, traditional heatmap methods suffer from high overhead and long inference time. Furthermore, external environmental factors such as time, weather, lighting, and distance can also affect pose estimation results. Low-resolution images can hinder pose estimation performance in inclement weather conditions and when detecting distant pedestrians. Traditional low-resolution image processing methods, such as upsampling, have limited computational efficiency and low accuracy. Therefore, research is necessary to accurately estimate pedestrian pose in these harsh environments. In addition to accurate pose estimation, pedestrian action encoding methods are also crucial. The reliability of pose-based action encoding significantly impacts the accuracy of pedestrian crossing intention prediction. Skeleton-based action recognition methods have strong learning capabilities for complex traffic scenarios and dynamic pedestrian features. However, traditional action recognition methods rely on recurrent neural networks to learn temporal information while ignoring spatial relationships. Convolutional neural networks, while extracting spatial features, suffer from poor robustness and insufficient generalization performance. Graph convolutional networks have achieved remarkable results in skeleton-based action recognition. For example, ST-GCN constructs a spatiotemporal graph based on a human skeleton sequence, and the adaptive graph convolutional network flexibly learns action information through an attention mechanism. However, due to the complexity and diversity of pedestrian motion patterns, it is difficult for a shared topology to learn the diverse relationships between joints under different motion patterns. Skeleton information of different actions can be independently modeled by parameterizing a multi-channel graph convolutional network, but there are disadvantages such as high parameter overhead and long inference time. Therefore, real-time, dynamic, and effective action encoding remains a challenging task. This embodiment combines the spatiotemporal behavior characteristics of pedestrians with the interaction relationship characteristics of traffic scene objects to accurately predict the pedestrian's intention to cross the street, and introduces the AAP layer to fuse the spatiotemporal action features and scene object interaction features to accurately predict the pedestrian's intention to cross the street.

[0074] In one embodiment, Figure 1 As shown, the extraction of spatiotemporal behavior features Ks from the pedestrian skeleton information X includes:

[0075] The spatiotemporal behavior features Ks are obtained by sequentially extracting using two layers of multi-scale graph convolutional layers MS_GCL.

[0076] When pedestrians intend to cross the street, they will perform corresponding actions, such as turning their heads, turning around, lifting their feet, and looking. Accurately capturing pedestrian movement information can improve the accuracy of crossing intention prediction. The pedestrian skeleton is a compact representation of posture, and the pedestrian skeleton can be regarded as a graph structure data G = (V, E). Where V = {v1, v2, ..., v N} is a set of N joint points, E={e ij|i=1,2,...N,j=1,2,...,N,i≠j} is the torso set between joints. On this basis, this embodiment uses two layers of multi-scale graph convolution layers MS_GCL to reduce the amount of computation and the spatiotemporal overhead, thereby improving the accuracy of pedestrian spatiotemporal motion encoding.

[0077] To alleviate the challenges of action recognition in low-resolution images, this implementation uses a human keypoint vector representation and introduces a multi-scale graph convolutional layer (MS-GCL) to extract deep spatiotemporal information from the skeleton, thereby learning the motion characteristics of different pedestrians in low-resolution images. Addressing the bottleneck of information interaction between traffic objects, this method proposes to establish interactions between the target pedestrian and related traffic objects in the scene, achieving a deeper understanding of higher-level semantic elements related to crossing intentions.

[0078] Among them, each layer of MS_GCL contains a multi-scale channel refinement graph convolution layer MCR_GCL and a multi-scale temporal convolution layer MS_TCL. The extraction process of each multi-scale graph convolution layer MS_GCL includes:

[0079] Use the multi-scale channel refinement graph convolutional layer MCR_GCL to extract spatial features from the input information;

[0080] The multi-scale temporal convolutional layer MS_TCL is used to enrich the temporal information of the spatial features.

[0081] Aiming at the problem of single adaptive graph convolution network structure and redundant parameterized multi-channel graph convolution information, such as Figure 1 As shown, the multi-scale channel refinement graph convolution layer MCR_GCL is used to extract spatial features from the input information, including:

[0082] First parameterize a shared adjacency matrix Serves as a priori spatial matrix for all channels and provides inherent spatial properties between general joint points;

[0083] Then, the spatial features between the joints of different channels in different motion modes are learned through the multi-scale refined dynamic shared adjacency matrix D;

[0084] Finally, the spatial features of each channel and the input data T(X) after feature transformation are aggregated to better learn the multi-category action patterns of pedestrians;

[0085] The specific formula is:

[0086]

[0087] f MCR_GCL is the output of the multi-scale channel refinement graph convolutional layer MCR_GCL, represents the weight of the multi-scale channel refinement graph convolutional layer MCR_GCL, λ represents the trainable parameter, and σ is the sigmoid() function.

[0088] The calculation formula of the multi-scale temporal convolution layer MS_TCL is:

[0089]

[0090] Among them, W i TCL Represents the weights of the multi-scale temporal convolution layer MS_TCL.

[0091] like Figure 1 As shown in the figure, in each layer of MS_GCL, the output of MCR_GCL is processed by BN+ReLU and then used as the input of MS_TCL. BN (Batch Normalization) and ReLU (Rectified Linear Unit) can be used together to improve the performance of the neural network. BN can make the input of ReLU more stable, thereby improving the expressiveness and generalization ability of ReLU.

[0092] like Figure 1 As shown, the Sum+ReLU operation is performed on the output of each layer of MS_GCL, which is a residual connection method used for cross-layer connections in deep neural networks to improve network performance and stability.

[0093] Exploring the interactive relationship between objects in traffic scenes is of great significance for understanding traffic scenes, and can assist pedestrian action features in better predicting the intention to cross the street. To this end, this embodiment uses IRN to model the interactive relationship between scene objects. First, Faster R-CNN is used to extract input image features. Then, a relational feature extractor is used to extract visual features in the bounding box of each traffic object. Finally, the relational feature extractor is used to extract the interactive relationship features of paired traffic objects based on paired bounding boxes and corresponding visual features. Figure 1 As shown, the specific process is as follows:

[0094] First, use the feature pyramid to extract the image features of the last frame of the observation frame;

[0095] Then, the attribute network is used to extract attribute features based on the visual features of each detected object O in the image features.

[0096] Then use the interaction relationship fusion function U i,j (·) Take any two objects O i and O j Attribute feature A i and A j As input, output relation features in,

[0097] R i,j (A i ,A j )=ReLU(W i A i +W j A j )-(W i A i -W j A j )⊙(W i A i -W j A j )

[0098] representing the feature space mapping of the target object O i , O j ;

[0099] Finally, all the relationship features R i,j (A i ,A j ) are aggregated to generate the object interaction relationship feature

[0100] The specific formula is:

[0101]

[0102] representing the interaction relationship fusioner, f ij representing the relationship distribution vector between the target object categories O i , O j , and m representing the number of objects O.

[0103] In one embodiment, as shown in Figure 1 , the adaptive average pooling layer AAP is used to deep fuse the spatio-temporal behavior feature Ks and the traffic scene object interaction relationship feature Us, to realize robust pedestrian crossing intention prediction, which includes:

[0104] First, the adaptive average pooling layer AAP is used to pool the traffic scene object interaction relationship feature Us, so that the pooled Us and Ks have the same channel structure, and are normalized by sigmoid(); AAP can convert any size feature map into a fixed size vector, thereby reducing the number of parameters of the fully connected layer, avoiding overfitting, and improving the generalization ability of the network;

[0105] Then, the attention mechanism is used to focus on the spatio-temporal behavior feature Ks, so that the spatio-temporal behavior feature Ks has a greater weight;

[0106] Based on the output of the attention mechanism, the traffic scene object interaction relationship feature Us and the spatiotemporal behavior feature Ks are fused to obtain the fusion feature in,

[0107]

[0108] The output of the intention mechanism is a set of weights that represent the importance of each element in the input sequence. These weights can be used to perform a weighted average on the input sequence to obtain a vector representing the input sequence.

[0109] Finally, the fully connected layer is used to predict the target pedestrian’s crossing intention Y P ;in,

[0110] Set the threshold to 0.5, and after discretization, P P Mapped to Y P ;

[0111] SiLU(·) represents the activation function, Att(·) represents the attention-based transformation function, and W a represents the learnable attention weight, and Linear(·) is used to classify and output the probability of crossing / no crossing intention.

[0112] In order to balance the dual-channel fusion encoding feature M, this embodiment introduces an attention mechanism to highlight the action feature K S Impact on predicting pedestrian crossing intention. To prevent overfitting, the dropout value is set to 0.5, and a fully connected layer is used to predict the pedestrian S’s crossing intention.

[0113] Example 2

[0114] like Figure 1 As shown, a dual-channel pedestrian crossing intention prediction model is used to implement the dual-channel pedestrian crossing intention prediction method described in Example 1.

[0115] Specifically, the dual-channel pedestrian crossing intention prediction model is pre-trained to learn the posture of pedestrian samples in the frame before the crossing event Event and the interaction relationship features of traffic scene objects, and predict whether the pedestrian has the intention to cross the street at the time of the crossing event Event.

[0116] Among them, the model adopts the loss function Loss during the training process;

[0117] Loss=-Y T ·log(P P )-(1-Y T )·log(1-P P )

[0118] Y T represents the true label of the pedestrian sample S, P P Represents the model's predicted probability of pedestrian crossing intention.

[0119] Pedestrian crossing intention prediction is a binary classification task that includes crossing intention (C) and no crossing intention (NC). This embodiment uses a binary cross-entropy loss function to evaluate model performance and assist in the pedestrian action feature and traffic scene object interaction feature feature tasks.

[0120] Example 3

[0121] A pedestrian crossing intention prediction device based on dual channels, comprising:

[0122] An acquisition module, used to acquire observation frames related to pedestrian crossing;

[0123] A posture estimation and extraction module is used to extract the pedestrian skeleton information X of the target pedestrian from the observation frame using the posture estimation and extraction algorithm SimCC;

[0124] A pedestrian action encoding module is used to extract spatiotemporal behavior features Ks from the pedestrian skeleton information X;

[0125] A traffic scene object interaction module is used to extract the traffic scene object interaction relationship feature Us related to the target pedestrian from the last frame of the observation frame;

[0126] The intention prediction module is used to use the adaptive average pooling layer AAP to deeply fuse the spatiotemporal behavior features Ks and the traffic scene object interaction relationship features Us to achieve robust pedestrian crossing intention prediction.

[0127] The following describes the process of training, testing, and verifying the model in Example 2 in conjunction with a specific data set.

[0128] 1. Experimental Dataset

[0129] (1) JAAD: The JAAD dataset studies the behavior of pedestrians and drivers and the impact of other factors on them in the context of autonomous driving in everyday urban environments. To this end, the JAAD dataset provides 346 video clips cropped from more than 240 hours of driving footage. Each video is 5 to 10 seconds long and contains rich pedestrian attributes, appearance, and behavior labels. The JAAD dataset also includes a pedestrian behavior label dataset JAAD_beh, various weather conditions (such as cloudy, sunny, rainy, and snowy) at multiple locations, and pedestrian occlusion ratio labels, reflecting real driving scenarios.

[0130] (2) PIE: The PIE dataset is similar to the JAAD dataset and also studies pedestrian behavior in traffic scenes. The PIE dataset provides driving footage from Toronto, Canada, over six consecutive hours in clear weather. Unlike the JAAD dataset, the PIE dataset does not include driving footage in inclement weather. It provides ego-motion vehicle information labels from the OBD sensor, providing both traffic object attributes and location bounding box information.

[0131] 2. Data Preprocessing

[0132] The JAAD and PIE datasets use the same data partitioning as the benchmark. In the JAAD dataset, 323 videos are split into 177, 117, and 29 videos for training, testing, and validation, respectively. To meet the pedestrian observation sample length requirements, valid pedestrian samples are screened based on whether the pedestrian sample length meets the observation frame and TTE. The number of valid pedestrian samples in JAAD_all is 783, 612, and 115, respectively. The number of valid pedestrian samples in JAAD_beh is 194, 22, and 171, respectively. In the PIE dataset, sets 01, 02, and 06 are used as training sets, set 03 is used as a test set, and sets 04 and 05 are used as validation sets. The number of valid pedestrian samples in PIE is 795, 222, and 636, respectively.

[0133] like Figure 2 As shown in the figure, an event is defined as a sign that distinguishes pedestrians with intention to cross (C) from those without intention to cross (NC). Pedestrians with intention to cross are considered to have an event based on the start frame of the crossing behavior, while pedestrians with no intention to cross are considered to have an event based on the last observable frame. To provide short-term warnings of sudden pedestrian crossings and learn pedestrian behavior, the TTE (Time to Event) is set to 1–2 seconds (30–60 frames). Each pedestrian observation sample is 16 frames long and overlapped within the TTE. The sample overlap ratios for JAAD and PIE are 0.8 and 0.6, respectively.

[0134] The format of pedestrian observation sample data is (T, N, C). Among them, T represents the length of the pedestrian observation frame, N represents the number of pedestrian joints, and C represents the joint dimension of (x, y, d, sc). Among them, x and y represent the 2D coordinates of the joints, d represents the depth of the joints, and sc represents the confidence of the joints. In order to understand the posture information of pedestrians in complex traffic scenes, the present invention uses SimCC to extract pedestrian skeleton data, and introduces the R-MSFM monocular depth estimator pre-trained on the KITTI dataset to extract the depth information of the pedestrian skeleton in each frame of the observation sequence. In addition, when preprocessing the pedestrian skeleton, an MLP-based human posture prediction module is added. By learning the pedestrian observation sequence, the pedestrian posture of the next 30 frames is predicted [Pn+1 n n+2 ..., P n+30 ], to improve the performance of the pedestrian crossing intention prediction model. The traffic scene object interaction relationship feature assists the pedestrian action information, and deeper understanding of the scene high-level semantic information. The application uses the last frame of the observation frame as the traffic scene, and extracts the traffic object interaction relationship feature. The PIE dataset provides the bounding box coordinates of the traffic objects (Traffic Light, Vehicle, Sign, Crosswalk, Transit station and Pedestrian) in each frame. Unlike the PIE dataset, the JAAD dataset only provides the number label of whether there is a traffic object in the scene, and does not have the position information of the traffic object. Therefore, the application introduces Fast-RCNN to detect the traffic object target, and selects the four traffic objects closest to the pedestrian bounding box center and frequently interacting according to the pedestrian bounding box center to model the interaction relationship.

[0135] III. Evaluation index

[0136] The intention prediction module is a binary classification intention prediction model, and the pedestrian crossing intention has two labels: C represents crossing intention, and NC represents no crossing intention. For the problem of data set sample imbalance, the application follows the balanced data method in the Benchmark, and enhances the data by mirror flipping and random deletion. For the enhanced data, the Precision, Recall, Accuracy, F1 score and ROC_AUC of each label are additionally compared to evaluate the performance of the classifier. These evaluation indexes can be expressed by true positive (TP), false positive (FP), true negative (TN) and false negative (FN):

[0137]

[0138]

[0139]

[0140]

[0141] IV. Experimental setting

[0142] The experimental platform used in the present application is equipped with a Nvidia 3090 graphics card and an Intel I7 CPU, and the experimental environment is Ubuntu 18.04 system and Pytorch framework. The network is trained using the Adamw optimizer, and the batch size is 128 each time, and the training epoch is 30. The initial learning rate of JAAD is 0.005, and the initial learning rate of PIE is 0.003, and the CosineAnnealingLR learning strategy is used, and the learning rate is trained in a sinusoidal manner. In addition, in order to prevent overfitting and accelerate network convergence, the present application standardizes the data of the joint. The 2D coordinates of the joint are divided by the pixels 1920 and 1080 of the input picture respectively. At the same time, in the STORNet network, the last frame of the observed frame traffic scene picture is normalized channel by channel, and the mean is [102.9801, 115.9465, 122.7717], and the standard deviation is [1.0, 1.0, 1.0]. In addition, in order to obtain repeatable results for training and testing, the seed is set to 42.

[0143] V. Comparative test

[0144] The present application uses a pedestrian crossing intention prediction benchmark to compare different crossing intention prediction models. Table 1 shows the comparison results on the JAAD and PIE datasets. There are three types of trends in the mainstream pedestrian crossing intention prediction methods, convolutional models, recurrent models, and graph convolutional models. ATGC and ConvLSTM are early proposed to use convolutional models to realize the crossing intention prediction task, using convolutional neural networks to analyze the scene and predict the crossing behavior of pedestrians. Recurrent models model the dependency between visual features of consecutive video frames, such as SingleRNN, MultiRNN, StackedRNN, HierarchicalRNN, and SFRNN, which use image stacks as input and use stacked RNNs to layer-by-layer fuse visual features to improve pedestrian crossing intention prediction performance. 3D convolutional models can capture temporal information, such as C3D, I3D, and PCPA, which use 3D convolutional branches to encode visual information and use RNN branches to process appearance features provided by the dataset in parallel. Graph convolutional models model the structural properties of graphs and node feature information, such as Pedestrian Graph+, which introduces a graph convolutional network to model pedestrian pose data and uses convolutional modules to process images and vehicle speeds. DPCIAN has a model with a scene object interaction module and an action encoding module, and the PIE dataset has greatly improved results. Acc and F1 have improved by 2%. Although, the JAAD dataset does not provide traffic object bounding box coordinates, but through Fast-RCNN detection positioning, it is difficult to accurately obtain the attribute and location information of the traffic object. Moreover, the JAAD driving environment is more complex, with low visibility in scenes such as rain, snow, and night. The scene object interaction module has difficulty obtaining interactive information that affects pedestrian crossing. However, DPCIAN can also capture high-level interactive semantic perception and achieve better results than attribute semantic perception. The JAAD_all dataset has improved Acc by 3%. The JAAD_beh dataset has improved Acc by 1% and F1 by 2%, and other indicators are not inferior to SOTA models.

[0145] Table 1 Comparison experiment

[0146]

[0147] Six, qualitative analysis of pose estimation

[0148] Figure 3The pedestrian pose estimation results of four complex traffic environments in the JAAD and PIE datasets. In order to compare SimCC and Alphapose fairly, the number of pedestrian joint points in the application is uniformly set to 17, and the two facial joint points in Alphapose that have the least influence on the pedestrian crossing action are removed. At the same time, in order to better learn the pedestrian action information, the application distinguishes the four limbs of the pedestrian. Among them, the right arm skeleton in each sample is orange, the left arm skeleton is yellow, the right leg skeleton is light blue, and the left leg skeleton is dark blue. Figure 3 In the (a) and (b) sample parts, the pedestrian in the frame is blocked by other traffic objects (such as pedestrians and vehicles), but the skeleton fitting degree of SimCC can still achieve good results whether it is sunny or rainy. Figure 3 In (c) and (d), the skeleton information extraction is affected by rain, snow, and dim environment before sunrise, etc. SimCC uses a vector representation for key points, divides each pixel uniformly into multiple regions, and the skeleton fitting degree of the two dim environment samples is 4.1% and 11.2% higher than Alphapose, respectively. Through qualitative analysis of pose estimation, SimCC can accurately estimate the pedestrian pose in scenes with low resolution caused by distant target objects and bad weather, and provide effective pedestrian pose information for the action encoding module.

[0149] Seven, intention prediction qualitative analysis

[0150] Figure 4 The qualitative results of the model in the application, PCPA, and Pedestrian Graph Plus in the JAAD and PIE datasets. In the figure, the target pedestrian of each sample is marked with a red frame, the true label and the prediction result consistent with the true label are represented with red font, and the prediction result inconsistent with the true label is represented with blue font. In the provided pedestrian samples, the model in the application can accurately predict the crossing intention of the pedestrian. However, the application and Pedestrian Graph Plus have prediction errors in some samples. By analyzing the sample observation sequence, the reasons are summarized as follows from the pedestrian action information and scene information.(1) Figure 4 (a), Figure 4 (b), Figure 4 (c), and Figure 4 (e) samples show that in complex weather traffic scenes such as rain, snow, fog, and night, the pedestrian pose estimation and action encoding performance of the model in the application and Pedestrian Graph Plus decrease significantly. However, the pedestrian action encoding module in the application provides effective pedestrian action spatiotemporal features in low resolution scenes, improving the pedestrian action encoding ability.(2) In some special pedestrian samples, such as Figure 4 (b) and Figure 4(c) The pedestrian is far away from the road and has no intention to cross the street, even though he is moving to cross. The traffic scene object interaction relationship module in the present invention models the global traffic objects, provides the relative position information of the traffic objects, and assists the pedestrian action encoding module to complete the scene understanding, thereby reducing the prediction error of the long-distance pedestrian's intention to cross the street. (3) Figure 4 (b) represents an unrelated pedestrian sample. In most cases, unrelated pedestrians will not affect the normal operation of the autonomous vehicle. However, unrelated pedestrians have the intention to cross the street. Moreover, as key participants in the intelligent traffic scene, unrelated pedestrians will affect the crossing intentions of other pedestrians. Based on road safety as the criterion, this invention models the interaction between scene objects, improving the accuracy of pedestrian crossing intention prediction while also enhancing road safety.

[0151] With the above-described preferred embodiments of the present invention as a guide, and with reference to the above description, relevant personnel are fully capable of making various changes and modifications without departing from the technical scope of this invention. The technical scope of this invention is not limited to the contents of the specification and must be determined according to the scope of the claims.

Claims

1. A dual-channel pedestrian crossing intention prediction method, characterized in that: include: Obtain observation frames related to pedestrian crossing; Extracting pedestrian skeleton information X of the target pedestrian from the observation frame using a posture estimation extraction algorithm SimCC; Extracting spatiotemporal behavior features Ks from the pedestrian skeleton information X; Extracting the traffic scene object interaction relationship feature Us related to the target pedestrian from the last frame of the observation frame; The adaptive average pooling layer AAP is used to deeply fuse the spatiotemporal behavior feature Ks and the traffic scene object interaction relationship feature Us to achieve robust pedestrian crossing intention prediction; The method of using the adaptive average pooling layer AAP to deeply fuse the spatiotemporal behavior feature Ks and the traffic scene object interaction relationship feature Us to achieve robust pedestrian crossing intention prediction includes: First, the adaptive average pooling layer AAP is used to pool the traffic scene object interaction relationship features Us and normalize them using sigmoid(); Then, the attention mechanism is used to focus on the spatiotemporal behavior feature Ks, so that the spatiotemporal behavior feature Ks has a greater weight; Based on the output of the attention mechanism, the traffic scene object interaction relationship feature Us and the spatiotemporal behavior feature Ks are fused to obtain the fusion feature in, Finally, the fully connected layer is used to predict the target pedestrian’s crossing intention Y P ;in, Set the threshold to 0.5, and after discretization, P P Mapped to Y P ; SiLU(·) represents the activation function, Att(·) represents the attention-based transformation function, and W a represents the learnable attention weight, Linear(·) is used to classify and output the probability of crossing / no crossing intention; σ is the sigmoid() function.

2. The method for predicting pedestrian crossing intention based on dual channels according to claim 1, characterized in that: The step of extracting the spatiotemporal behavior feature Ks from the pedestrian skeleton information X includes: The spatiotemporal behavior features Ks are obtained by sequentially extracting using two layers of multi-scale graph convolutional layers MS_GCL.

3. The dual-channel pedestrian crossing intention prediction method according to claim 2, characterized in that: The extraction process of each multi-scale graph convolution layer MS_GCL includes: Use the multi-scale channel refinement graph convolutional layer MCR_GCL to extract spatial features from the input information; The multi-scale temporal convolutional layer MS_TCL is used to enrich the temporal information of the spatial features.

4. The dual-channel pedestrian crossing intention prediction method according to claim 3, characterized in that: The multi-scale channel refinement graph convolution layer MCR_GCL is used to extract spatial features from input information, including: First parameterize a shared adjacency matrix Serves as the prior spatial matrix for all channels and provides the intrinsic spatial properties between joint points; Then, the spatial features between the joints of different channels in different motion modes are learned through the multi-scale refined dynamic shared adjacency matrix D; Finally, the spatial features of each channel and the input data T(X) after feature transformation are aggregated; The specific formula is: f MCR_GCL is the output of the multi-scale channel refinement graph convolutional layer MCR_GCL, represents the weight of the multi-scale channel refinement graph convolutional layer MCR_GCL, and λ represents a trainable parameter.

5. The dual-channel pedestrian crossing intention prediction method according to claim 4, characterized in that: The calculation formula of the multi-scale temporal convolution layer MS_TCL is: Among them, W i TCL Represents the weights of the multi-scale temporal convolution layer MS_TCL.

6. The dual-channel pedestrian crossing intention prediction method according to claim 1, characterized in that: The extracting of the traffic scene object interaction relationship feature Us related to the target pedestrian from the last frame of the observation frame includes: First, use the feature pyramid to extract the image features of the last frame of the observation frame; Then, the attribute network is used to extract attribute features based on the visual features of each detected object O in the image features. Then use the interaction relationship fusion function U i,j (·) Take any two objects O i and O j Attribute feature A i and A j As input, output relation features in, R i,j (A i ,A j )=ReLU(W i A i +W j A j )-(W i A i -W j A j )⊙(W i A i -W j A j ) Indicates O i , O j Feature space mapping of the target object; Finally, all the relationship features R i,j (A i ,A j ) aggregation to generate object interaction relationship features The specific formula is: represents the interaction relationship fusion, f ij Indicates O i , O j The relationship distribution vector between target object categories, m represents the number of objects O.

7. A dual-channel pedestrian crossing intention prediction model, characterized by: Used to implement the dual-channel pedestrian crossing intention prediction method described in any one of claims 1-6.

8. The dual-channel pedestrian crossing intention prediction model according to claim 7, characterized in that: It uses the loss function Loss during the training process; Loss=-Y T ·log(P P )-(1-Y T )·log(1-P P ) Y T represents the true label of the pedestrian sample S, P P Represents the model's predicted probability of pedestrian crossing intention.

9. A dual-channel pedestrian crossing intention prediction device, characterized in that: include: An acquisition module, used to acquire observation frames related to pedestrian crossing; A posture estimation and extraction module is used to extract the pedestrian skeleton information X of the target pedestrian from the observation frame using the posture estimation and extraction algorithm SimCC; A pedestrian action encoding module is used to extract spatiotemporal behavior features Ks from the pedestrian skeleton information X; A traffic scene object interaction module is used to extract the traffic scene object interaction relationship feature Us related to the target pedestrian from the last frame of the observation frame; The intention prediction module is used to use the adaptive average pooling layer AAP to deeply fuse the spatiotemporal behavior feature Ks and the traffic scene object interaction relationship feature Us to achieve robust pedestrian crossing intention prediction; wherein, The method of using the adaptive average pooling layer AAP to deeply fuse the spatiotemporal behavior feature Ks and the traffic scene object interaction relationship feature Us to achieve robust pedestrian crossing intention prediction includes: First, the adaptive average pooling layer AAP is used to pool the traffic scene object interaction relationship features Us and normalize them using sigmoid(); Then, the attention mechanism is used to focus on the spatiotemporal behavior feature Ks, so that the spatiotemporal behavior feature Ks has a greater weight; Based on the output of the attention mechanism, the traffic scene object interaction relationship feature Us and the spatiotemporal behavior feature Ks are fused to obtain the fusion feature in, Finally, the fully connected layer is used to predict the target pedestrian’s crossing intention Y P ;in, Set the threshold to 0.5, and after discretization, P P Mapped to Y P ; SiLU(·) represents the activation function, Att(·) represents the attention-based transformation function, and W a represents the learnable attention weight, Linear(·) is used to classify and output the probability of crossing / no crossing intention; σ is the sigmoid() function.

Images