Pedestrian crossing intention recognition method based on multi-source information fusion

Abstract

The application relates to the field of automatic driving. A pedestrian crossing intention recognition method based on multi-source information fusion is characterized by comprising the following steps: step one, pedestrian traffic feature extraction: according to the classification of people-vehicles-roads, features obviously affecting the recognition of pedestrian intention are screened out, including road environment features, traffic features and pedestrian behavior data features, and a pedestrian traffic feature extraction model is constructed; step two, encoding all feature information extracted in step one: static, dynamic and time feature coding networks are respectively built to recode the extracted feature information, and deep information related to pedestrian crossing actions is obtained; step three, recognizing the pedestrian crossing intention through multi-feature fusion: a layered and step-by-step feature mixed fusion architecture is determined based on the correlation between feature coding and pedestrian intention recognition, an attention mechanism is introduced, a pedestrian crossing intention recognition network model is established, and the probability of the target pedestrian crossing action is calculated to judge the pedestrian crossing intention.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving, the field of automobile driver assistance technology and the field of computer vision, and in particular to a pedestrian crossing intention recognition method based on multi-source information fusion, a device and a storage medium. BACKGROUND

[0002] With the research of automatic driving car technology, the development of automatic driving car brings certain safety hazards to vulnerable traffic participants such as pedestrians and cyclists, and people are full of worries about whether the automatic driving car can consider these problems when driving on public roads. Therefore, it is crucial for the automatic driving car to be able to detect, classify and predict the intention of vulnerable traffic participants in real time, and take necessary actions to ensure the traffic safety of other road users.

[0003] When a vehicle drives in an urban traffic scene, it is a very common scene and a scene where traffic accidents are most likely to occur that there are pedestrians crossing the road transversely in front of the vehicle, and the automatic driving car must be able to accurately identify the crossing pedestrian intention and then take measures to avoid the pedestrian. Pedestrian crossing intention recognition is a key step between environmental perception and decision planning for an automatic driving car, and is the most important basis for safe driving of an automatic driving car, and has important research significance.

[0004] Currently, the research on pedestrian intention recognition is mainly divided into three categories based on dynamic motion modeling, planning model and data-driven.

[0005] The method based on dynamic motion modeling requires high accuracy of the model, and assumes that all trajectories have similar dynamics, but the actual situation is not the case. Due to the individual differences of pedestrians, their motion state is extremely irregular, and there is no consistent dynamic model for pedestrian motion trajectory, so the method based on dynamic motion modeling has low accuracy in pedestrian behavior intention recognition.

[0006] The intention recognition method based on planning model needs the destination information of the pedestrian as a constraint condition, but it is difficult for an automatic driving car to effectively and quickly determine the destination of the pedestrian, so the method based on planning model is not suitable for automatic driving vehicles to determine the crossing intention of pedestrians in urban road scenes.

[0007] The data-driven deep learning method shows excellent performance in the research on pedestrian intention recognition, but existing research tends to use a single feature source for intention recognition research, and there are fewer studies using multiple feature information. In order to improve the recognition accuracy, as the number of feature sources used by the research increases, how to balance the utilization weight of each feature information, fuse different feature information and find feature information that can more accurately express the pedestrian intention becomes the focus of the research. SUMMARY

[0008] The technical problem to be solved by the present application is to provide a pedestrian crossing intention recognition method, device and storage medium based on multi-source information fusion, which realizes the fusion of different feature information and finds a method and device that can more accurately express the pedestrian intention.

[0009] Technical scheme

[0010] The pedestrian crossing intention recognition method based on multi-source information fusion comprises the following steps:

[0011] Step one, pedestrian traffic feature extraction: according to the classification of people-vehicles-roads, the features that have a relatively obvious influence on the recognition of pedestrian intention are screened out, including road environment features, traffic features and pedestrian behavior data features, and a pedestrian traffic feature extraction model is constructed;

[0012] Step two, encoding all the feature information extracted in step one: a static, dynamic and time feature encoding network is respectively built to re-encode the extracted feature information, and deep information related to pedestrian crossing actions is obtained;

[0013] Step three, identifying pedestrian crossing intention through multi-feature fusion: a hierarchical and step-by-step feature mixed fusion architecture is determined based on the correlation between feature encoding and pedestrian intention recognition, an attention mechanism is introduced, a pedestrian crossing intention recognition network model is established, the probability of the target pedestrian crossing action is calculated, and the pedestrian crossing intention is judged.

[0014] Further, in step one, by analyzing the behavior characteristics of pedestrians in urban road scenes and the situation awareness of dynamic decision-making of autonomous vehicles on urban roads, the road environment is structured according to the function of functional traffic elements, a traffic environment hierarchical framework model is built, the traffic features in urban scenes are analyzed layer by layer based on the environment hierarchical framework model, the traffic features to be extracted are determined, the corresponding feature extraction module is built, the pedestrian traffic feature extraction model is established, and the feature extraction is completed.

[0015] Further, the pedestrian traffic feature extraction model comprises pedestrian bounding box feature data obtained by target detection, pedestrian skeleton pose feature data obtained by pose estimation, pedestrian appearance feature data obtained by image cropping, global context environment feature data obtained by image semantic segmentation, and vehicle information feature data obtained from vehicle sensors.

[0016] In step two, feature encoding networks are established for different traffic characteristics to encode information related to pedestrian crossing actions. Through temporal analysis of traffic scenes, optical flow is used to supplement the temporal information of the scene. A 3D convolutional I3D network with dilation and separable convolution is introduced to optimize dilation and build a lightweight temporal feature extraction network to encode optical flow. Based on the VGG19 network, a spatially separable convolutional attention module is introduced to build a static feature encoding network to encode context and pedestrian appearance attribute information. Based on gated recurrent units, a bidirectional GRU dynamic feature encoding network is built to encode the context, pedestrian state, action information, and vehicle information bidirectionally, capturing the human-vehicle interaction relationship and obtaining corresponding deep feature information.

[0017] The lightweight temporal feature extraction network consists of twelve convolutional layers, four max pooling layers, and one global average pooling layer. It is divided into five modules according to the pooling process. The post-processing part includes flattening layers and fully connected layers. Modules 1 and 2 use basic convolutional and max pooling layers, modules 3 and 4 use improved Inception convolutional and max pooling layers, and module 5 uses separable convolutional and global average pooling layers.

[0018] The improved Inception convolution uses a 2D convolution kernel and introduces dilated convolution, including four branches. The first branch propagates the temporal dimension information backward separately, with the convolution kernel set to 3×1×1. The second and third branches first use 1×1×1 convolution to achieve feature dimensionality reduction and parameter reduction, and then use 1×3×3 convolution to ensure that the temporal dimension information continues to propagate backward while continuing to convolve the spatial dimension information. The pooling and convolution in the fourth branch ensure that the input and output sizes are consistent, and the number of channels in the network remains matched.

[0019] The separable convolution is based on the improved Inception convolution, while introducing dilated convolution and depthwise separable convolution. The first branch of the improved Inception convolution remains unchanged. After the 3D convolutional layers of the second and third branches, a 1D convolutional layer is added. The 2D convolutional layer with a kernel size of 1×3×3 samples spatial dimension information, and the 1D convolutional layer with a kernel size of 3×1×1 samples temporal dimension information. The max pooling layer of the fourth branch has a pooling kernel size of 3×3×3, which downsamples the temporal dimension while improving the ability to perceive temporal information with larger weights.

[0020] Furthermore, the layered and step-by-step feature fusion architecture in step three is as follows: First, pedestrian action encoding and temporal feature encoding are concatenated and fused and input into the temporal attention module to obtain the first attention vector. Next, the first attention vector is concatenated and fused with pedestrian trajectory encoding and input into the temporal attention module to obtain the second attention vector. Then, it is concatenated and fused with vehicle speed encoding to obtain the first temporal attention encoding. GRU is used to encode attribute feature encoding and context encoding again to highlight temporal feature information and obtain spatial feature encoding. The spatial feature encoding is input into the temporal attention module to obtain the second and third temporal attention encodings about temporal features. After obtaining the three temporal attention encodings, they are concatenated and fused and input into the modal attention module to obtain the attention encodings weighted by each modality feature. The Dense layer is input to obtain the probability of the target pedestrian crossing action, thereby determining the pedestrian's crossing intention.

[0021] A pedestrian crossing intention recognition system is characterized by comprising a video loading module, a pedestrian traffic feature extraction module, a feature encoding module, and multiple intention recognition model loading modules, which implement any of the above methods by executing the program of the system modules.

[0022] A storage medium storing a program, characterized in that the program, when executed by a processor, implements any of the methods described above.

[0023] Beneficial effects

[0024] This invention establishes a pedestrian traffic feature extraction model and completes feature extraction. A multi-feature encoding network is constructed to re-encode shallow traffic feature information related to pedestrian crossing behavior into deep feature encoding information. A backbone network is built based on VGG19, and spatial attention is introduced after the pooling layer. A spatially separable convolutional attention mechanism module is obtained based on a local-global attention alternation mechanism, establishing a static feature encoding network to improve the network's attention to high-weight information in static features. By introducing dilated convolution and spatiotemporally separable convolution to optimize I3D, basic, improved Inception, and spatiotemporally separable convolutional blocks are obtained, establishing a temporal feature encoding network TFENet. BN and ReLU functions are introduced to enhance the network's sensitivity to temporal feature information. Based on the GRU's mechanism for preserving and updating long-term temporal information, a bidirectional GRU dynamic feature encoding network is established to bidirectionally encode pedestrian-vehicle information and determine the pedestrian-vehicle interaction relationship. Based on the correlation between traffic feature encoding information and pedestrian crossing intention, a tiered hierarchical feature fusion network architecture based on an attention mechanism is determined, establishing a pedestrian crossing intention recognition network model. The binary cross-entropy loss function is optimized based on the positive and negative sample weights of the data, and the Adam-optimized stochastic gradient descent method is used to optimize the network, ensuring the generalization and robustness of the intent recognition network. Comparative experimental results show that the pedestrian intent recognition model achieves an intent recognition accuracy of approximately 96% on the JAAD and PIE datasets, outperforming similar intent recognition models PCIP and PCPA. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the pedestrian crossing intent recognition method based on multi-source information fusion according to the present invention.

[0026] Figure 2 Network diagram for extracting time features.

[0027] Figure 3 This is a schematic diagram of the structure of the improved Inception convolutional block.

[0028] Figure 4 This is a schematic diagram of the structure for separating convolutional blocks.

[0029] Figure 5 This is a schematic diagram of a static feature encoding network structure.

[0030] Figure 6 This is a bidirectional GRU network diagram.

[0031] Figure 7 This is a diagram of a hierarchical, tiered hybrid and converged architecture. Detailed Implementation

[0032] The present invention will be further described below with reference to specific embodiments and accompanying drawings.

[0033] Based on the shortcomings of existing technologies, this application takes pedestrian behavioral intentions in urban environments as the research object. By analyzing the characteristics of pedestrian behavior in urban road scenarios, a pedestrian traffic feature extraction model is established. The extracted traffic features are recoded based on neural networks, and an attention network based on multi-feature fusion is built to identify pedestrian intentions.

[0034] This application proposes a pedestrian crossing intention recognition method based on multi-source information fusion. To identify pedestrian crossing intentions, a pedestrian traffic feature extraction model is established to extract traffic feature information from image sequences. Feature encoding networks are built for different traffic features to encode information related to pedestrian crossing actions. Based on the correlation between the obtained encoded information and pedestrian crossing intention recognition, these are fused together at different levels to obtain temporal attention encoding. This fused encoding is then input into a modal attention module to obtain a weighted attention encoding for each modal feature. This weighted encoding is then input into a fully connected layer to obtain the probability of the target pedestrian crossing action, thereby determining the pedestrian crossing intention. Finally, an algorithm model for pedestrian crossing intention recognition based on multi-feature fusion is obtained using experimental data. Experimental results demonstrate that the algorithm can effectively identify pedestrian crossing intentions.

[0035] Based on the pedestrian-vehicle-road classification, features that significantly influence pedestrian intent identification were selected, including road environment features, traffic features, and pedestrian behavior data features. A pedestrian traffic feature extraction model was constructed, and the probability of pedestrian i's crossing action was estimated based on the extracted pedestrian traffic feature information. The formula for determining a pedestrian's intention to cross the street based on the probability of their crossing action is as follows:

[0036]

[0037] Where m is the number of observed frames, t is the time before the last frame was observed before the event occurred, and n is the total time from time t until the pedestrian crosses the street / does not cross the street event occurs.

[0038] Due to the unpredictability of most urban traffic scenarios and pedestrians, longer prediction timeframes are unreliable. Therefore, during feature extraction, the observation data for each pedestrian is sampled so that the time t of the last observation frame is between 30-60 frames (1-2 seconds) before the event occurs, which refers to the pedestrian making a decision to take further action. Each target pedestrian i has a given continuous image sequence of fixed length m. In this embodiment, the length of the observation image sequence is determined to be 16 frames.

[0039] The overall steps are as follows:

[0040] Step 1: Feature Extraction: Traffic feature information is extracted from the urban road environment using the established pedestrian traffic feature extraction model. Specifically, this includes:

[0041] The vehicle-mounted camera captures real-time images of the road and surrounding environment ahead of the autonomous vehicle, obtaining video data. The acquired video data stream is then preprocessed. Based on pedestrian-vehicle-road traffic feature extraction, pedestrian bounding boxes L are extracted according to pedestrian features. i Pedestrian skeleton posture P i Pedestrian appearance representation R Li Extract global context C based on road environment features. g The vehicle speed S is extracted based on vehicle characteristics.

[0042] 1) The pedestrian bounding box uses the coordinates of the target pedestrian bounding box to describe the 2D position trajectory changes of the target pedestrian, as shown in the formula:

[0043]

[0044] The coordinates of the bounding box of the target pedestrian in the tm frame image are shown in the following formula:

[0045]

[0046] in This represents the coordinates of the top-left corner of the bounding box. The bounding box represents the coordinates of the bottom right corner. The pedestrian bounding box provides information about the pedestrian's positional changes in the image.

[0047] 2) Pedestrian skeleton pose is obtained by estimating the pose of pedestrians in the image sequence, as shown in the equation:

[0048]

[0049] The vector of two-dimensional coordinates of the pedestrian in the tm frame image, which includes 25 pose joints, is shown in the following formula:

[0050]

[0051] Pedestrian skeleton pose provides detailed information about pedestrian movement to predict future pedestrian actions.

[0052] 3) Pedestrian appearance representation is based on pedestrian bounding box L i The position coordinates of the cropped local image are magnified by a fixed factor, as shown in the following formula:

[0053]

[0054] Pedestrian appearance representation provides pedestrian attribute information to determine pedestrian movement tendencies.

[0055] 4) The global context is obtained by semantic segmentation of the image sequence frame by frame, as shown in the equation:

[0056]

[0057] The global context provides semantic information about all elements in the image, as well as the interaction relationships between pedestrians and all objects on the road.

[0058] 5) The vehicle speed is represented by the vehicle's motion label x. v,t In the expression {ego_dep, ego_vel}, ego_vel is as shown in the formula:

[0059] S={s t-m ,s t-m+1 ,...,s t}

[0060] The vehicle speed is closely related to the pedestrian's position change and the pedestrian bounding box L. i The fusion process can reveal the human-vehicle interaction relationship.

[0061] Step 2: Feature Encoding: Pedestrian traffic features are shallow information, which is difficult to use directly for intent recognition. Further processing is needed to obtain deeper information from the traffic features.

[0062] Therefore, feature encoding networks were established for different traffic characteristics to encode information related to pedestrian crossing actions. Optical flow was introduced to supplement the temporal information of the scene, and a temporal feature extraction network, TFENet, was established by optimizing the I3D network. A static feature encoding network was established based on the VGG19 network with a spatial attention mechanism, and a bidirectional GRU encoding network was established based on GRU units to encode context and pedestrian-vehicle information respectively, obtaining corresponding deep feature information. The extracted traffic feature information was re-encoded by building static, dynamic, and temporal feature encoding networks in three parts to obtain deep information related to pedestrian crossing actions.

[0063] The specific steps are as follows:

[0064] 2.1) Introducing optical flow O to supplement the temporal information of the scene, the optical flow extracted based on the image sequence is shown in the following equation:

[0065]

[0066] Where i is the target pedestrian, m is the number of observed frames, t is the time before the event occurs when the last frame is observed, and n is the total time from time t until the pedestrian crosses the street / does not cross the street event occurs.

[0067] A lightweight temporal feature extraction network, TFENet, is built by optimizing the I3D (Dilated 3D Convolutional Network) network, as shown in the attached figure. Figure 2The lightweight temporal feature extraction network consists of twelve convolutional layers, four max-pooling layers, and one global average pooling layer, divided into five modules. The post-processing part includes flattening layers and fully connected layers. Modules 1 and 2 use basic convolutional and max-pooling layers, modules 3 and 4 use improved Inception convolutional and max-pooling layers, and module 5 uses separable convolutional and global average pooling layers. The Max Pooling layers in modules 3 and 4 can simultaneously downsample temporal and spatial dimensional information, fusing spatiotemporal information for backpropagation. After processing by modules 3 and 4, the input features yield a feature map of size 2×7×7×832.

[0068] There are three types of convolutional blocks (Conv_Block): basic convolutional blocks, improved Inception convolutional blocks, and separating convolutional blocks. All convolutional layers undergo batch normalization (BN) to batch normalize the data, ensuring it still conforms to a standardized distribution. The ReLU non-linear activation function is added to increase the network's non-linearity, allowing it to express more complex features, improving its discriminative ability, and avoiding the vanishing and exploding gradient problems.

[0069] To reduce network parameters and improve network lightweighting while maintaining network depth, separable convolution is introduced. The original 3D convolutional kernels (d×d×d) in some convolutional layers are split into a temporal dimension (d×1×1) and a spatial dimension (1×d×d). This step-by-step convolution of the input feature map effectively reduces network parameters. To increase the receptive field of the convolutional kernels, enabling them to perceive features at larger scales, dilated convolution is introduced. Dilated convolution does not change the feature map size while reducing the feature loss rate caused by downsampling. It can be combined with regular convolutional kernels to extract features at different scales.

[0070] Therefore, the improved Inception convolutional blocks all use 2D convolutional kernels and introduce dilated convolutions to increase the receptive field, ensuring that the number of parameters remains basically unchanged, and acquiring a wider range of feature information. Branch 1 propagates the temporal dimension information backward, with a 3×1×1 convolutional kernel. Branches 2 and 3 first use 1×1×1 convolutions to achieve feature dimensionality reduction and parameter reduction, and then use 1×3×3 convolutions to ensure that the temporal dimension information propagates backward while continuing to convolve the spatial dimension information. The pooling and convolutional layers in branch 4 ensure that the input and output sizes are consistent, and the number of network channels remains matched. The convolutional block structure is shown in the attached figure. Figure 3 As shown.

[0071] The separable convolutional block is based on the improved Inception convolutional block, introducing both dilated convolution and depthwise separable convolution. Branch 1 of the improved Inception convolutional block remains unchanged, while branches 2 and 3 each have a 1D convolutional layer added after their 3D convolutional layers. A 1×3×3 2D convolutional layer samples spatial dimension information, and a 3×1×1 1D convolutional layer samples temporal dimension information. The Max Pooling layer in branch 4 has its pooling kernel size modified to 3×3×3, downsampling the temporal dimension while improving its ability to perceive time information with higher weights. All four branches can perform convolutions on the temporal dimension, improving the efficiency of obtaining high-level temporal semantic features. The concatenate layer connects the feature maps from the four branches, using the number of channels as the concatenation axis, to obtain the output features of the convolutional block. The convolutional block structure is shown in the attached figure. Figure 4 As shown.

[0072] A spatiotemporal feature map of size 2×7×7×1024 is obtained by continuous convolution of the spatiotemporal features in the optical flow using spatiotemporal convolutional blocks. Then, the dimensionality is reduced and matched with the feature dimension, and a Dense layer is used to perform feature space transformation to obtain the temporal feature information encoding T of size [None, 256]. O As shown in the following formula:

[0073] T O =TFENet(input[O])=sigmoid(f conv (input[O]))

[0074] 2.2) A static feature encoding network was established based on the VGG19 network by introducing a spatial attention mechanism, as shown in the attached figure. Figure 5 The diagram illustrates a network structure divided into four modules, comprising twelve convolutional layers and four max-pooling layers. The first three modules each contain four convolutional layers using 3×3 kernels with a stride of 1. The number of kernels is 64 and 128 in module one, 256 in module two, and 512 in module three. The pooling layers use 2×2 kernels with a stride of 2, with the number of kernels being 64, 128, 256, and 512 respectively. The fourth module is a spatially separable convolutional attention module with six layers, outputting a 16×512 feature map to obtain attribute feature encoding and context feature encoding.

[0075] A static feature coding network based on the VGG19 network is designed to characterize pedestrian appearance R. Li and global context C gThe spatial features in the image are encoded. The input size of the image sequence is 16×224×224×3. The backbone network consists of convolutional module 1 to pooling module 8 of the VGG19 network. After continuous convolution and pooling, an image feature map F with a size of 14×14×512 is obtained. A spatial attention module is added after the backbone network to increase the network's attention weight to important features in the image. The static feature encoding network structure is shown in the attached figure. Figure 5 As shown, the attribute feature encoding F is obtained after spatial attention operation. RLi and context encoding F cg As shown in the following formula:

[0076]

[0077]

[0078] Where f conv For convolution operations in the backbone network, f SSCBAM For spatial attention operations, R Li For pedestrian appearance representation, C g For the global context, F is the input feature map, and M is the global context. C Here, σ represents the channel attention weights, σ ​​is the sigmoid activation function, W1 and W0 are shared parameters of the perceptron, and M... SS The attention weights are spatially separable, and the number of sub-windows is m×n. F represents the size of the child window. C For the output of the channel attention module, This is the output after average pooling. This is the output after max pooling. f is the symbol for pointwise multiplication. 2×2 It is a 2x2 convolutional layer.

[0079] 2.3) Previous studies on pedestrian intent recognition, when using RNNs to encode feature information of the current frame, only considered the impact of past pedestrian positions and actions on the current moment, neglecting information about future pedestrian motivations in constantly changing traffic scenarios. Therefore, this application's solution establishes a bidirectional GRU encoding network module based on GRUs, considering the past, present, and future states of pedestrians to fully utilize contextual information. The bidirectional GRU encodes the input pedestrian state information and vehicle information in sequential and reverse order in the time dimension, capturing information about state changes caused by human-vehicle interactions, simulating human-vehicle interaction, and improving the accuracy of pedestrian state information encoding. A bidirectional GRU encoding network is established based on GRU units to encode pedestrian skeleton pose, pedestrian bounding box, and vehicle speed, obtaining corresponding deep feature information. The structure of the bidirectional GRU encoding network module is shown in the attached figure. Figure 6 As shown.

[0080] The input x of the bidirectional GRU module j By pedestrian bounding box L i Pedestrian skeleton posture P i Composed of vehicle information S, x j The definition is shown in the following formula:

[0081]

[0082] Where j is the j-th GRU unit used in the bidirectional GRU module, and in this embodiment, the number of GRU units in the network is set to 256; i is the i-th target pedestrian in the observation sequence; m is the length of the observation sequence; t is the last frame of the observation sequence; l ij t The input is the pedestrian bounding box coordinates of the target pedestrian i in frame t for the j-th GRU cell, where the coordinates of the pedestrian bounding box are... in This represents the coordinates of the top-left corner of the bounding box. This indicates the coordinates of the bottom right corner of the bounding box.

[0083] In this embodiment, the bidirectional GRU uses a stack of two layers: a forward layer f and a reverse layer b. The output y of the j-th unit of the bidirectional GRU module is... j The definition is as follows:

[0084] in and These are the hidden layer state information output by the j-th unit in the forward and backward layers, respectively. The hidden layer state information output by the last GRU unit is used as the pedestrian bounding box L. i Pedestrian skeleton posture P i The feature encoding result of vehicle information S is shown in the following formula:

[0085]

[0086] Among them, G f and G b These are forward and backward GRU units, respectively. To encode pedestrian trajectories obtained from pedestrian bounding box data, For pedestrian motion encoding obtained from pedestrian skeleton pose, The vehicle speed code is obtained from the vehicle information.

[0087] Step 3: Multi-feature fusion: Through steps 1 and 2 above, the entire intent recognition model acquires image sequences from the input video stream. In the feature extraction module, traffic feature information is extracted from the image sequences using the established pedestrian and traffic feature extraction model, resulting in optical flow O and pedestrian bounding boxes L.i Pedestrian skeleton posture P i Vehicle speed S, pedestrian appearance representation R Li and global context C g Then, the traffic features are further encoded through the established coding network to obtain deeper information, resulting in the time feature code T. O Attribute feature encoding F RLi Context encoding F Cg Pedestrian trajectory coding Pedestrian motion coding Vehicle speed coding In step 3, based on the correlation between the encoded information obtained above and the pedestrian crossing intention recognition, the temporal attention encoding is fused together at different levels. This fused encoding is then input into the modal attention module to obtain a weighted attention encoding for each modal feature. This weighted attention encoding is then input into the fully connected layer to obtain the probability of the target pedestrian crossing action, thereby determining the pedestrian's crossing intention. The hierarchical, tiered hybrid fusion architecture used is shown in the attached figure. Figure 7 The meaning is as shown. Specifically:

[0088] Based on pedestrian trajectory coding Pedestrian motion coding Vehicle speed coding The characteristic attributes and their correlation with pedestrian intent recognition are fused together according to different levels. This is because pedestrian action encoding... and time feature encoding T O All of these contain human motion information. They are first concatenated and fused together, then input into the temporal attention module to obtain the attention vector a. f1 Next, the attention vector a f1 Pedestrian trajectory coding The splicing and fusion are fed into the time attention module to obtain a f2 Then, combined with vehicle speed coding The temporal attention code a is obtained by splicing and fusion. t1 .

[0089] Pedestrian appearance representation R Li and global context C g After processing by a static feature coding network, the attribute feature code F is obtained. RLi Context encoding F Cg To unify the dimension of feature encoding for easier concatenation and fusion, GRU is used to re-encode the attribute features F. RLi Context encoding F Cg Encoding is performed to highlight temporal feature information to obtain spatial feature code S. Rli and S Cg Spatial feature encoding is input into the temporal attention module to obtain temporal attention encoding a for temporal features. t2 and at3 .

[0090] After obtaining the three temporal attention codes, they are concatenated and fused, and then input into the modal attention module to obtain the attention code a weighted by each modal feature. m The probability of a pedestrian crossing the street is obtained by inputting the Dense layer. This is used to determine the pedestrian's intention to cross the street. The result of the action prediction probability is shown in the following formula:

[0091]

[0092] Where sigmoid is the activation function, α m This is the attention module.

[0093] Finally, by comparing the model accuracy, the model hyperparameters were adjusted to obtain the optimal multi-feature fusion attention network model. The accuracy was calculated as follows:

[0094] Accuracy=(TP+TN) / (TP+FN+FP+TN)

[0095] In this system, TP stands for true class, FN for false inverse class, FP for false positive class, and TN for true inverse class.

[0096] During network training, to eliminate the influence of other factors, both the hardware platform and software parameters were kept consistent. The proposed model was compared with those from PCPA and PCIP. The model was reproduced using the benchmark code released by PCIP, employing the training parameters from the original literature, and the datasets JAAD and PIE.

[0097] Five-fold cross-validation was used to evaluate the model's training performance. Validation data was sampled from the training data but was not used in network training. The original data was divided into five groups, with each subset used as a validation set, and the remaining subsets used as the training set. The optimizer used was the Adam optimizer with momentum, a batch size of 16, and a learning rate of 4×10⁻⁶. -5 The training and validation results are shown in the table below:

[0098] Table 1. Accuracy of different models on the dataset.

[0099]

[0100] The effectiveness of the pedestrian crossing intention recognition method based on multi-source information fusion of the present invention was verified.

[0101] Approximately 200 valid pedestrian samples were collected. All intent recognition models were used to identify the intent of these 200 pedestrian samples, and the recognition results were statistically analyzed. The statistical results are shown in the table below:

[0102] Table 2 Recognition results of different models

[0103]

[0104] As shown in Table 2, the multi-feature fusion attention network model achieves a 96% accuracy rate in recognizing pedestrian crossing intentions, outperforming other models. Comparison of recognition results on the test and validation sets verifies the model's stability, indicating that the proposed model meets the accuracy requirements.

[0105] In summary, the pedestrian crossing intention recognition method based on multi-source information fusion of this invention determines the fusion principles and methods of feature encoding based on the correlation between traffic feature coding information and pedestrian crossing intention. It introduces time and modal attention mechanisms, establishes a hierarchical feature fusion network architecture based on the attention mechanism, and builds a pedestrian crossing intention recognition network model. Comparative experimental results show that the pedestrian crossing intention recognition accuracy is high and it can effectively identify pedestrian crossing intentions.

[0106] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention, all of which fall within the scope of the claims. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for recognizing pedestrian crossing intentions based on multi-source information fusion, characterized in that... Includes the following steps: Step 1: Pedestrian and Traffic Feature Extraction: Based on the classification of people, vehicles, and roads, select features that have a significant impact on identifying pedestrian intentions, including road environment features, traffic features, and pedestrian behavior data features, and construct a pedestrian and traffic feature extraction model; Step 2: Encode all the feature information extracted in Step 1: Construct static, dynamic and temporal feature encoding networks to re-encode the extracted feature information and obtain deeper information related to pedestrian crossing actions; Step 3: Identify pedestrian crossing intentions through multi-feature fusion: Based on the correlation between feature encoding and pedestrian intention recognition, determine a hierarchical and step-by-step feature fusion architecture, introduce an attention mechanism, establish a pedestrian crossing intention recognition network model, calculate the probability of the target pedestrian's crossing action, and thus determine the pedestrian's crossing intention; In step two, feature encoding networks are established for different traffic characteristics to encode information related to pedestrian crossing actions. Through temporal analysis of traffic scenes, optical flow is used to supplement the temporal information of the scene. A 3D convolutional I3D network with dilation and separable convolution is introduced to optimize dilation and build a lightweight temporal feature extraction network to encode optical flow. Based on the VGG19 network, a spatially separable convolutional attention module is introduced to build a static feature encoding network to encode context and pedestrian appearance attribute information. A bidirectional GRU dynamic feature encoding network is built based on gated recurrent units, which encodes the context, pedestrian state, action information, and vehicle information bidirectionally to capture human-vehicle interaction relationships and obtain corresponding deep feature information. The layered and step-by-step feature fusion architecture in step three is as follows: First, pedestrian action encoding and temporal feature encoding are concatenated and fused and input into the temporal attention module to obtain the first attention vector. Next, the first attention vector is concatenated and fused with pedestrian trajectory encoding and input into the temporal attention module to obtain the second attention vector. Then, it is concatenated and fused with vehicle speed encoding to obtain the first temporal attention encoding. GRU is used to encode attribute feature encoding and context encoding again to highlight temporal feature information and obtain spatial feature encoding. The spatial feature encoding is input into the temporal attention module to obtain the second and third temporal attention encodings about temporal features. After obtaining the three temporal attention encodings, they are concatenated and fused and input into the modal attention module to obtain the attention encodings weighted about each modality feature. The Dense layer is input to obtain the probability of the target pedestrian crossing action, thereby determining the pedestrian's intention to cross the street.

2. The pedestrian crossing intent recognition method based on multi-source information fusion as described in claim 1, characterized in that: In step one, by analyzing the characteristics of pedestrian behavior in urban road scenarios and the situational awareness of autonomous vehicles making dynamic decisions in urban roads, the road environment is structurally represented according to the role of functional traffic elements, a hierarchical framework model of the traffic environment is built, and traffic features in urban scenarios are analyzed layer by layer based on the hierarchical framework model to determine the traffic features that need to be extracted, a corresponding feature extraction module is built, a pedestrian traffic feature extraction model is established, and feature extraction is completed.

3. The pedestrian crossing intent recognition method based on multi-source information fusion as described in claim 2, characterized in that: The pedestrian traffic feature extraction model includes pedestrian bounding box feature data obtained by object detection, pedestrian skeleton pose feature data obtained by pose estimation, pedestrian appearance representation feature data obtained by image cropping, global context environmental feature data obtained by image semantic segmentation, and vehicle information feature data obtained from vehicle sensors.

4. The pedestrian crossing intent recognition method based on multi-source information fusion as described in claim 1, characterized in that: The lightweight temporal feature extraction network consists of twelve convolutional layers, four max pooling layers, and one global average pooling layer. It is divided into five modules according to the pooling process. The post-processing part includes flattening layers and fully connected layers. Modules 1 and 2 use basic convolutional and max pooling layers, modules 3 and 4 use improved Inception convolutional and max pooling layers, and module 5 uses separable convolutional and global average pooling layers.

5. The pedestrian crossing intent recognition method based on multi-source information fusion as described in claim 4, characterized in that: The improved Inception convolution uses a 2D convolution kernel and introduces dilated convolution, including four branches. The first branch propagates the temporal dimension information backward separately, with the convolution kernel set to 3×1×1. The second and third branches first use 1×1×1 convolution to achieve feature dimensionality reduction and parameter reduction, and then use 1×3×3 convolution to ensure that the temporal dimension information continues to propagate backward while continuing to convolve the spatial dimension information. The pooling and convolution in the fourth branch ensure that the input and output sizes are consistent, and the number of channels in the network remains matched.

6. The pedestrian crossing intent recognition method based on multi-source information fusion as described in claim 5, characterized in that: The separable convolution is based on the improved Inception convolution, while introducing dilated convolution and depthwise separable convolution. The first branch of the improved Inception convolution remains unchanged. After the 3D convolutional layers of the second and third branches, a 1D convolutional layer is added. The 2D convolutional layer with a kernel size of 1×3×3 samples spatial dimension information, and the 1D convolutional layer with a kernel size of 3×1×1 samples temporal dimension information. The max pooling layer of the fourth branch has a pooling kernel size of 3×3×3, which downsamples the temporal dimension while improving the ability to perceive temporal information with larger weights.

7. A pedestrian crossing intention recognition system, characterized in that, It includes a video loading module, a pedestrian and traffic feature extraction module, a feature encoding module, and multiple intent recognition model loading modules. By executing the program of the system module, it implements the method as described in any one of claims 1 to 6.

8. A storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.

Images