Pedestrian crossing intention prediction method based on progressive multi-modal feature fusion

Abstract

The invention belongs to the technical field of intention prediction, and particularly relates to a pedestrian crossing intention prediction method based on progressive multi-modal feature fusion, which comprises the following steps: acquiring a pedestrian anchor frame, a self-vehicle speed and pedestrian joints in front of a vehicle under a self-vehicle view angle; constructing a multi-expert feature embedding module to obtain anchor frame features, speed features and joint features; constructing a specific modal feature extraction module comprising a self-attention module and a multi-time scale map convolution module to obtain enhanced anchor frame features, speed features and joint features; constructing a token generation module based on a sliding window mechanism to obtain aggregated tokens; constructing a progressive feature fusion module to obtain a fused token; constructing a token fusion module based on an attention mechanism to obtain fusion features; and predicting the pedestrian crossing probability based on the aggregated tokens and the fusion features. According to the method, the street crossing intention of the pedestrian can be predicted in an automatic driving environment, and the automatic driving automobile can have enough response time to make a correct decision.

Description

Technical Field

[0001] The present invention belongs to the technical field of intention prediction, and in particular relates to a method for predicting pedestrian crossing intention based on progressive multimodal feature fusion. Background Art

[0002] With the rapid development of artificial intelligence technology, autonomous vehicles have begun to appear in people's daily lives. Predicting pedestrians' intentions to cross the street in advance is one of the important functions of autonomous vehicles. Pedestrians are one of the important traffic participants on the road. Predicting pedestrians' behavior can help autonomous driving systems respond in time, avoid traffic accidents, and improve driving safety. In addition, accurate pedestrian intention prediction helps improve traffic flow, reduce unnecessary stops or decelerations, and thus optimize overall traffic efficiency. However, there are many challenges in accurately predicting pedestrians' intentions to cross the street. First, pedestrians' behavior has strong individual differences and is affected by many factors, including weather, traffic conditions, and mood. Second, pedestrians' intentions are often not easy to observe or quantify directly. They usually need to be inferred by analyzing their gait, position, and line of sight, which places high demands on the accuracy of sensors and the complexity of algorithms. In addition, pedestrians may show sudden changes in behavior in some cases, such as sudden stops or rapid crossings, which also places high demands on the real-time response capabilities of autonomous driving systems.

[0003] Current pedestrian crossing intention recognition methods mainly enhance the diversity and comprehensiveness of features by fusing multiple different modal context information of traffic scenes. These modalities include pedestrian anchor frames, vehicle speed, pedestrian posture, traffic lights, local context images, and global context images. However, existing methods still face many challenges. On the one hand, with the diversification of input modalities (such as anchor frame data, speed data, and posture data), the model architecture becomes more complex, resulting in redundant information in the modal fusion process, which affects the generalization ability of the model. Inappropriate architecture design may also lead to information loss and reduce the performance of the method. On the other hand, although the introduction of multimodal data provides the system with a richer information dimension, it also significantly increases the complexity of the fusion architecture. Summary of the invention

[0004] In view of the above deficiencies in the prior art, the purpose of the present invention is to provide a pedestrian crossing intention prediction method based on progressive multimodal feature fusion, which solves the problem of feature redundancy caused by too many input modalities and information loss caused by inappropriate model design in the existing algorithm, and improves the accuracy and robustness of pedestrian crossing intention recognition.

[0005] To achieve the above objectives, the present invention provides a method for predicting pedestrian crossing intention based on progressive multimodal feature fusion, comprising the following steps: S1, obtain the pedestrian anchor frame B, the vehicle speed V and the pedestrian joint P from the perspective of the vehicle in front of the vehicle; S2. Construct a multi-expert feature embedding module, input the pedestrian anchor frame B, vehicle speed V and pedestrian joint P into the multi-expert feature embedding module to map to high-dimensional space to extract features and obtain anchor frame features , speed characteristics and joint characteristics ; S3. Construct a modality-specific feature extraction module including a self-attention module and a multi-timescale graph convolution module, and embed the multi-expert feature into the anchor box feature output by the module. , speed characteristics Input to the self-attention module to obtain the enhanced anchor box features and speed characteristics , and the joint features Input to the multi-time scale graph convolution module to obtain enhanced joint features ; S4. Build a token generation module based on the sliding window mechanism, and extract the anchor box features output by the specific modality feature extraction module. , speed characteristics and joint characteristics Input into the token generation module to obtain the aggregated token , , ; S5. Build a progressive feature fusion module to generate the output of the token generation module. , , Input to the progressive feature fusion module to obtain the fused token ; S6. Construct a token fusion module based on the attention mechanism to combine the output of the progressive feature fusion module Input the token fusion module to enhance the token information and obtain the fusion feature U; S7. Input the fused feature U into the fully connected layer to predict the probability Pred of pedestrians crossing the street.

[0006] As a preferred embodiment of the present invention, in the S1, the pedestrian anchor frame B is a coordinate sequence of the upper left corner and lower right corner of the two-dimensional rectangular frame surrounding the pedestrian, the vehicle speed V is the continuous vehicle speed data obtained by the sensor, and the pedestrian joints P include the three-dimensional coordinates of 19 skeletal joints including the head, neck, shoulders, elbows, wrists, hips, knees, ankles, toes, heels and the center of the torso.

[0007] As a preferred solution of the present invention, in S2, the multi-expert feature embedding module includes an input layer, an average pooling layer, a score unit, a score adjustment unit, an expert network and an output layer, wherein the score unit includes a fully connected layer and a Softmax activation function, and the expert network includes expert 1, expert 2, ..., expert n; based on the multi-expert feature embedding module, the anchor frame feature is obtained , speed characteristics and joint characteristics The process is: S2.1. Input the pedestrian anchor frame B into the expert network to obtain the corresponding output: ; In the formula, Conv1D represents 1D convolution; represents the convolution kernel weight of the i-th expert, i=1, 2, ..., n, where n is the total number of experts; represents the output of the i-th expert; S2.2. Use the average pooling layer to average pool the pedestrian anchor box B in the time dimension, input the fully connected layer and the Softmax activation function, calculate the score of each expert, and convert the output of each expert Multiply by the corresponding score to adjust the importance of the expert; S2.3. Add the weighted outputs of all experts to get the anchor box tensor ; S2.4. Add position encoding to introduce time series relationship into features and obtain anchor frame features ; S2.5. Based on S2.1-S2.4, similarly process the vehicle speed V and pedestrian joint P to obtain the speed feature and joint characteristics (The corresponding tensors are the velocity tensor and the joint tensor ).

[0008] As a preferred solution of the present invention, in S3, the enhanced anchor frame feature is obtained. and speed characteristics The process is: S3.1. Anchor box features , the keys, values, and queries of self-attention are all , based on the self-attention module, the attention weight matrix is ​​calculated to achieve feature enhancement, and the enhanced anchor frame features are obtained after completing the self-attention operation ; Similarly, the enhanced speed feature is obtained ; Get joint features The method is: S3.2. The multi-timescale graph convolution module includes an input layer, a linear transformation layer, a multi-scale convolution layer, a ReLU activation function, a fully connected layer, a global average pooling layer, a multi-layer perceptron, a Sigmoid activation function, and an output layer. The multi-scale convolution layer includes three convolution kernels of 3×1, 5×1, and 7×1. The multi-timescale graph convolution module is used to process joint features. , the process is: S3.2.1. Establish a learnable adjacency matrix that expresses the connection relationship between pedestrian skeleton joints , and normalize it to , Initialize to the identity matrix; S3.2.2. Using matrix multiplication, using the adjacency matrix Pedestrian joint features Perform aggregation to obtain the aggregation matrix ; S3.2.3. Use multi-scale convolutional layers to extract the motion timing correlation of the same joint at different time scales: ; In the formula, ReLU is the activation function; Conv2D represents two-dimensional convolution, Indicates convolution kernels, , that is , , , respectively represent the convolution kernels of 3×1, 5×1, and 7×1; Indicates The output of the convolution kernel; S3.2.4, connect the outputs of the three convolution kernels in the channel dimension to obtain feature M; S3.2.5, re-weight the channel dimension of feature M; S3.2.6. Compress the channel dimension through the fully connected layer and perform average pooling in the joint dimension to obtain joint features. .

[0009] As a preferred embodiment of the present invention, in S3.2.5, the method for re-weighting is: S3.2.5.1. Perform average pooling on the feature M in the time dimension and joint dimension to generate a global representation y. S3.2.5.2. Compress and restore the channel dimension features through two fully connected layers to obtain the weight of each channel : ; In the formula, represents the sigmoid activation function; FC represents the fully connected layer; S3.2.5.3. The weight obtained Applied to the feature M of the original input to adjust the feature M.

[0010] As a preferred embodiment of the present invention, in S4, the token generation module includes an input layer, a sequence segmentation layer, an attention mechanism, a token splicing layer, a self-attention layer and an output layer, and the method for obtaining the aggregated token based on the token generation module is: S4.1. The token generation module divides the entire sequence into multiple subsequences and introduces a learnable token for each subsequence. Assume that the subsequence length is , then each subsequence is represented as: ; In the formula, , Represents the number of tokens; , That is , , ; express The jth subsequence is divided; the number of subsequences is , T is the total length of the sequence; S4.2. For each subsequence Introducing Tokens , the token As a query, the corresponding subsequence As keys and values, we apply an attention mechanism to aggregate useful features of subsequences by tokens and obtain ; S4.3. Tokens of all subsequences Splice along a new dimension to get the splicing token ; S4.4. Use the self-attention mechanism to model the relationship between different tokens and obtain global information , that is , , .

[0011] As a preferred solution of the present invention, in the S5, the progressive feature fusion module includes a symmetric channel decentralized attention mechanism SCSA and a fusion layer, wherein the SCSA includes a fully connected layer, a Softmax activation function and a layer normalization, and the fused token is obtained. The process is: S5.1. Use a fully connected layer to transform the input tokens The channel dimension is expanded to twice the original one, and then the expanded features are divided into two parts, namely and , , , represents the jth token of the corresponding branch; S5.2, using the fully connected layer Mapping and weighting through Softmax activation function ; S5.3. Apply the weights obtained Features Adjustment is performed, expressed as: ; In the formula, represents element-wise multiplication, , Respectively represent the adjusted and ; S5.4. Using learnable parameters Tokens enhanced by fusing the left and right branches : ; Where LN represents the layer normalization operation; S5.5, use the feature combination of anchor frame-velocity bv, anchor frame-joint bp and velocity-joint vp to perform bimodal fusion respectively, and compare the result of bimodal fusion with Connect, then apply SCSA to enhance the features, realize the fusion of three-modal features, and obtain the fused token .

[0012] As a preferred solution of the present invention, in the above S5.5, when dual-modality fusion is performed, the features of the two modalities are first connected, and then the SCSA module is applied to enhance the features, and the features interact with each other, which is expressed as: ; ; ; Where, SCSA is the symmetric channel distributed attention mechanism SCSA; Concat represents the connection operation; , , They are the fused anchor frame-velocity bv, anchor frame-joint bp and velocity-joint vp feature combinations respectively; The trimodal feature fusion is expressed as: ; ; ; ; In the formula, The features representing the final fusion are the fused tokens; , , They are the feature combinations of the fused anchor frame-velocity bv and joint p, anchor frame-joint bp and velocity v, velocity-joint vp and anchor frame b respectively; , , are learnable parameters used to adaptively control the importance of the three modes.

[0013] As a preferred solution of the present invention, in S6, the token fusion module includes an input layer, a fully connected layer, a Softmax activation function, a global average pooling layer, a global maximum pooling layer, a normalization layer, a Dropout layer and an output layer, and the process of obtaining the fusion feature U is: S6.1. Use two fully connected layers and then apply the Softmax activation function to generate token weights : ; In the formula, FC represents the fully connected layer; S6.2. Weight Apply to And perform global average pooling and maximum pooling on the weighted features in the time dimension; S6.3. Introducing learnable parameters , the two pooling results are fused to form a more comprehensive feature representation, and then the normalization operation LN and regularization Dropout are applied to obtain the fused feature U: ; In the formula, GAP represents global average pooling; GMP represents global maximum pooling; Represents element-wise multiplication.

[0014] As a preferred solution of the present invention, after each module is constructed, the entire network composed of each module is trained using the cross entropy loss function. In order to promote single-modal feature learning, the tokens of each modality are Aggregate and input the probability of pedestrian crossing predicted by the fully connected layer as , and combined with Pred, set the cross entropy loss function Loss to: ; In the formula, Label represents the true label of whether the pedestrian is crossing the street; CE represents cross entropy, which is used to measure the difference between the predicted probability and the true intention; is a learnable parameter used to balance the contribution of different losses.

[0015] The algorithm involved in the present invention can be executed by an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the above algorithm calculation is implemented by executing the software by the processor.

[0016] The beneficial effects of the present invention are: The present invention can predict the intention of pedestrians to cross the street in an autonomous driving environment, so that the autonomous driving car has sufficient reaction time to make correct decisions, better coordinate with pedestrians and other traffic participants (such as cyclists and other vehicles), avoid misjudgments and conflicts, and improve the decision-making ability and safety of the autonomous driving system.

[0017] In order to solve the problem of limited representation capability for different input data due to the heterogeneity of input data, the present invention proposes a multi-expert feature embedding method to map heterogeneous features into a high-dimensional space to achieve feature alignment of different modalities. In order to solve the problem of feature redundancy and modality degradation that may result from the increase of input modalities, the present invention proposes a token generation module based on a sliding window mechanism to interact with the information of all time steps in the time series and gradually learn the key features of the sequence. In order to solve the problem of temporal modeling of pedestrians' intention to cross the street, the present invention uses a graph convolution operation based on multiple time scales to extract features from pedestrian joints. In addition to modeling the spatial correlation between different joints, the motion features of pedestrians are extracted at different time scales to capture their short-term rapid changes and long-term trend changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 is a mechanism flow chart of a multi-expert feature embedding module in an embodiment of the present invention; Figure 3 is a mechanism flow chart of a multi-time-scale graph convolution module in an embodiment of the present invention; Figure 4 is a fusion flow chart in a progressive feature fusion module in an embodiment of the present invention; Figure 5 is a mechanism flow chart of the symmetric channel distraction mechanism SCSA in an embodiment of the present invention; Figure 6 is a comparison chart of the results during the verification process of the present invention, Figure 6 (a), (b), and (c) are three groups of comparisons under different scenarios. DETAILED DESCRIPTION

[0019] The embodiments of the present invention are further described below in conjunction with the accompanying drawings: like Figure 1 As shown, a pedestrian crossing intention prediction method based on progressive multimodal feature fusion includes the following steps: S1. Obtain the pedestrian anchor frame B, ego vehicle speed V and pedestrian joint P from the perspective of the ego vehicle in front of the vehicle (the subsequent anchor frame b, speed v and joint p are abbreviated forms of them); S2. Construct a multi-expert feature embedding module, input the pedestrian anchor frame B, the vehicle speed V and the pedestrian joint P into the multi-expert feature embedding module to map them to the high-dimensional space to extract features (rich features) and obtain the anchor frame features , speed characteristics and joint characteristics ; S3. Construct a modality-specific feature extraction module including a self-attention module and a multi-timescale graph convolution module, and embed the multi-expert feature into the anchor box feature output by the module. , speed characteristics Input to the self-attention module to obtain the enhanced anchor box features and speed characteristics , and the joint features Input to the multi-time scale graph convolution module to obtain enhanced joint features ; S4. Build a token generation module based on the sliding window mechanism, and extract the anchor box features output by the specific modality feature extraction module. , speed characteristics and joint characteristics Input into the token generation module to obtain the aggregated token , , ; S5. Build a progressive feature fusion module to generate the output of the token generation module. , , Input to the progressive feature fusion module to obtain the fused token ; S6. Construct a token fusion module based on the attention mechanism to combine the output of the progressive feature fusion module Input the token fusion module, enhance the token information, and obtain the fusion feature U; enhance the information of important tokens to make the feature more discriminative; S7. Input the fused feature U into the fully connected layer to predict the probability Pred of pedestrians crossing the street.

[0020] In S7, the probability can be obtained based on a well-known technology. For example, the output of the fully connected layer can be converted into a probability value between 0 and 1 based on a Sigmoid function.

[0021] In S1, the pedestrian anchor frame B is the coordinate sequence of the upper left corner and lower right corner of the two-dimensional rectangular frame surrounding the pedestrian, the vehicle speed V is the continuous vehicle speed data obtained by the sensor, and the pedestrian joints P include the three-dimensional coordinates of 19 skeleton joints including the head, neck, shoulders, elbows, wrists, hips, knees, ankles, toes, heels and torso center.

[0022] In S2, the multi-expert feature embedding module includes an input layer, an average pooling layer, a score unit, a score adjustment unit, an expert network, and an output layer, wherein the score unit includes a fully connected layer and a Softmax activation function, and the expert network includes expert 1, expert 2, ..., expert n; anchor frame features are obtained based on the multi-expert feature embedding module. , speed characteristics and joint characteristics The process is: S2.1. Input the pedestrian anchor frame B into the expert network to obtain the corresponding output: ; In the formula, Conv1D represents 1D convolution; represents the convolution kernel weight of the i-th expert, i=1, 2, ..., n, where n is the total number of experts; represents the output of the i-th expert; S2.2. Use the average pooling layer to average pool the pedestrian anchor box B in the time dimension, input the fully connected layer and the Softmax activation function, calculate the score of each expert, and convert the output of each expert Multiply by the corresponding score to adjust the importance of the expert; S2.3. Add the weighted outputs of all experts to get the anchor box tensor ; Contains rich features; S2.4. Add position encoding to introduce time series relationship into features and obtain anchor frame features ; S2.5. Based on S2.1-S2.4, similarly process the vehicle speed V and pedestrian joint P to obtain the speed feature and joint characteristics (The corresponding tensors are the velocity tensor and the joint tensor ).

[0023] Position encoding is a method of providing the model with information about the position of elements in a sequence when processing sequence data. The mechanism flow chart of the multi-expert feature embedding module is as follows: Figure 2 shown.

[0024] In S3, the enhanced anchor box features are obtained and speed characteristics The process is: S3.1. Anchor box features , the keys, values, and queries of self-attention are all , based on the self-attention module, the attention weight matrix is ​​calculated to capture the correlation between different frames and realize feature enhancement. After completing the self-attention operation, the enhanced anchor frame features are obtained. ; Similarly, the enhanced speed feature is obtained ; Get joint features The method is: S3.2. The multi-timescale graph convolution module includes an input layer, a linear transformation layer, a multi-scale convolution layer, a ReLU activation function, a fully connected layer, a global average pooling layer, a multi-layer perceptron, a Sigmoid activation function, and an output layer. The multi-scale convolution layer includes three convolution kernels of 3×1, 5×1, and 7×1. The multi-timescale graph convolution module is used to process joint features. , the process is: S3.2.1. Establish a learnable adjacency matrix that expresses the connection relationship between pedestrian skeleton joints (based on the prior knowledge of human bone structure), and normalize it to , Initialize to the identity matrix; S3.2.2. Using matrix multiplication, using the adjacency matrix Pedestrian joint features Perform aggregation to obtain the aggregation matrix ; S3.2.3. Use multi-scale convolutional layers to extract the motion timing correlation of the same joint at different time scales: ; In the formula, ReLU is the activation function; Conv2D represents two-dimensional convolution, Indicates convolution kernels, , that is , , , respectively represent the convolution kernels of 3×1, 5×1, and 7×1; Indicates The output of the convolution kernel; S3.2.4, connect the outputs of the three convolution kernels in the channel dimension to obtain feature M; S3.2.5, in order to highlight important features and suppress redundant features, the channel dimension of feature M is re-weighted; S3.2.6. To align with the shapes of other modalities, the channel dimension is compressed through the fully connected layer, and average pooling is performed in the joint dimension to obtain joint features. .

[0025] In S3.2.5, the method for reweighting is: S3.2.5.1. Perform average pooling on the feature M in the time dimension and joint dimension to generate a global representation y. S3.2.5.2. Compress and restore the channel dimension features through two fully connected layers to obtain the weight of each channel : ; In the formula, represents the sigmoid activation function; FC represents the fully connected layer; S3.2.5.3. The weight obtained Applied to the feature M of the original input to adjust the feature M.

[0026] The mechanism flow chart of the multi-time scale graph convolution module is as follows Figure 3 shown.

[0027] In S4, the token generation module includes an input layer, a sequence segmentation layer, an attention mechanism, a token concatenation layer, a self-attention layer, and an output layer. The method for obtaining the aggregated token based on the token generation module is: S4.1. The token generation module divides the entire sequence into multiple subsequences and introduces a learnable token for each subsequence. Assume that the subsequence length is , then each subsequence is represented as: ; In the formula, , Represents the number of tokens; , That is , , (The same goes for the other superscript m); express The jth subsequence is divided; the number of subsequences (equivalent to the number of tokens) is , T is the total length of the sequence; S4.2. For each subsequence Introducing Tokens , the token As a query, the corresponding subsequence As keys and values, the attention mechanism is applied to aggregate the useful features of the subsequences (key information, including temporal dynamic features, spatial relationship features, modal interaction features, key event features, contextual features, etc.) through tokens to obtain ; S4.3. Tokens of all subsequences Splice along a new dimension to get the splicing token (The concatenation operation enables features from different modalities to be considered simultaneously in subsequent steps); S4.4. Use the self-attention mechanism to model the relationship between different tokens and obtain higher-level global information , that is , , .

[0028] In S5, the progressive feature fusion module includes the symmetric channel distributed attention mechanism SCSA and the fusion layer. SCSA includes the fully connected layer, the Softmax activation function and the layer normalization to obtain the fused token The process is: S5.1. Use a fully connected layer to transform the input tokens The channel dimension is expanded to twice the original one, and then the expanded features are divided into two parts, namely and , , , represents the jth token of the corresponding branch; S5.2, using the fully connected layer Mapping and weighting through Softmax activation function ; S5.3. Apply the weights obtained Features Adjustment is performed, expressed as: ; In the formula, represents element-wise multiplication, , Respectively represent the adjusted and ; S5.4. Using learnable parameters Tokens enhanced by fusing the left and right branches : ; Where LN represents the layer normalization operation; S5.5, use the feature combination of anchor frame-velocity bv, anchor frame-joint bp and velocity-joint vp to perform bimodal fusion respectively, and compare the result of bimodal fusion with Connect, and then apply SCSA to enhance the features (steps S5.1-S5.4), realize the fusion of trimodal features, and obtain the fused token .

[0029] In S5.5, when fusing two modalities, the features of the two modalities are first connected, and then the SCSA module is applied to enhance the features and interact between the features, which is expressed as: ; ; ; Where, SCSA is the symmetric channel distributed attention mechanism SCSA; Concat represents the connection operation; , , They are the fused anchor frame-velocity bv, anchor frame-joint bp and velocity-joint vp feature combinations respectively; The trimodal feature fusion is expressed as: ; ; ; ; In the formula, The features representing the final fusion are the fused tokens; , , They are the feature combinations of the fused anchor frame-velocity bv and joint p, anchor frame-joint bp and velocity v, velocity-joint vp and anchor frame b respectively; , , are learnable parameters used to adaptively control the importance of the three modes.

[0030] Symmetric channel split attention mechanism SCSA is a new type of attention mechanism that aims to improve the performance of visual tasks by combining spatial attention and channel attention. The fusion flow chart in the progressive feature fusion module is shown in Figure 4 As shown in the figure, the mechanism flow chart of SCSA is as follows Figure 5 shown.

[0031] In S6, the token fusion module includes an input layer, a fully connected layer, a Softmax activation function, a global average pooling layer, a global maximum pooling layer, a normalization layer, a Dropout layer, and an output layer. The process of obtaining the fused feature U is: S6.1. Use two fully connected layers and then apply the Softmax activation function to generate token weights : ; In the formula, FC represents the fully connected layer; S6.2. Weight Apply to And perform global average pooling and maximum pooling on the weighted features in the time dimension; S6.3. Introducing learnable parameters , the two pooling results are fused to form a more comprehensive feature representation. In order to improve the stability and generalization ability of the model, the normalization operation LN and regularization Dropout are applied to obtain the fused feature U: ; In the formula, GAP represents global average pooling; GMP represents global maximum pooling; Represents element-wise multiplication.

[0032] After constructing each module, the entire network composed of each module is trained using the cross entropy loss function. In order to promote single-modal feature learning, the tokens of each modality are Aggregate and input the probability of pedestrian crossing predicted by the fully connected layer as , and combined with Pred, set the cross entropy loss function Loss to: ; In the formula, Label represents the true label of whether the pedestrian is crossing the street; CE represents cross entropy, which is used to measure the difference between the predicted probability and the true intention; is a learnable parameter used to balance the contribution of different losses.

[0033] Applying the joint cross entropy loss function to train the entire network can reduce the difference between the model's predicted probability and the true label, while enhancing the discriminative ability of unimodal features.

[0034] During testing, the pedestrian anchor frame B, the vehicle speed V, and the pedestrian joint P are input into the trained prediction model (the entire network), and the probability Pred of the pedestrian's intention to cross the street from the perspective of the vehicle is output.

[0035] This embodiment is further described by a specific example: The network composed of various modules constructed by this embodiment is a pedestrian crossing intention prediction / recognition model, and is trained and tested on the PIE dataset (a dataset for pedestrian recognition and pedestrian re-identification research). For a pedestrian sample, the input history time length is set to 16 frames, and it is predicted whether the pedestrian has the intention to cross the street. In the experiment, it is carried out in an environment with an Intel Xeon E5-2678 CPU and an Nvidia GTX 3090 GPU. The batch size is set to 8. The number of attention heads is set to 8. The dropout of all modules is set to 0.1. The learning rate is 4e -4 The training is performed for a maximum of 400 epochs. The AdamW optimizer is used, and EarlyStop is used to prevent overfitting. If the validation set loss does not decrease after more than 15 epochs, the training is stopped. Regarding the model architecture, the feature dimension is set to 128.

[0036] Table 1 Performance comparison between different pedestrian intention recognition models

[0037] Table 1 is a performance comparison between different pedestrian intention recognition models, among which FFSTP, PedGraph+, CIPF, and TrEP are all recognition models / methods in existing research. ACC reflects the percentage of correctly predicted positive and negative samples in the total number, AUC reflects the area under the ROC curve, P1 reflects the ratio between the number of correctly classified positive samples and the total number of samples classified as positive, R reflects the ratio of actual positive samples predicted as positive, and F1 reflects the harmonic mean of precision P1 and recall R. Compared with other models, the model proposed in this embodiment achieves better recognition performance.

[0038] Figure 6 The following is a comparison chart of the experimental results of each model. Among them, C means crossing the street, NC means not crossing the street, and GroundTruth means the true intention.

[0039] Specifically, Figure 6 (a) and Figure 6 The changes in pedestrian behavior shown in (b) in the figure have a long delay, so the inherent characteristics of the model determine that it is not good at handling such scenarios. Although the model in this embodiment uses the same graph convolution structure as Pedestrian Graph+, considering the characteristics of this scenario, the concept of multi-scale is introduced to capture short-term, medium-term and long-term features respectively. Therefore, the model in this embodiment can predict this type of scenario more confidently. Figure 6(c) in the figure shows a pedestrian crossing the road with multiple pedestrians at an intersection. Due to the interference of noise in visual information and insufficient modeling of temporal information, Pedestrian Graph+, FFSTP, and TrEP all make incorrect predictions. Although PedCMT makes a correct prediction, its confidence is 5% lower than that of the model in this embodiment. It can be seen that the model in this embodiment always maintains accurate predictions in a series of scenarios, showing excellent robustness and adaptability.

Claims

1. A method for predicting pedestrian crossing intention based on progressive multimodal feature fusion, characterized in that The following steps are involved: S1, obtain the pedestrian anchor frame B, the vehicle speed V and the pedestrian joint P from the perspective of the vehicle in front of the vehicle; S2. Construct a multi-expert feature embedding module, input the pedestrian anchor frame B, vehicle speed V and pedestrian joint P into the multi-expert feature embedding module to map to high-dimensional space to extract features and obtain anchor frame features , speed characteristics and joint characteristics ; S3. Construct a modality-specific feature extraction module including a self-attention module and a multi-timescale graph convolution module, and embed the multi-expert feature into the anchor box feature output by the module. , speed characteristics Input to the self-attention module to obtain the enhanced anchor box features and speed characteristics , and the joint features Input to the multi-time scale graph convolution module to obtain enhanced joint features ; S4. Build a token generation module based on the sliding window mechanism, and extract the anchor box features output by the specific modality feature extraction module. , speed characteristics and joint characteristics Input into the token generation module to obtain the aggregated token , , ; S5. Build a progressive feature fusion module to generate the output of the token generation module. , , Input into the progressive feature fusion module to obtain the fused token ; S6. Construct a token fusion module based on the attention mechanism to combine the output of the progressive feature fusion module Input the token fusion module to enhance the token information and obtain the fusion feature U; S7. Input the fused feature U into the fully connected layer to predict the probability Pred of pedestrians crossing the street.

2. The method for predicting pedestrian crossing intention based on progressive multimodal feature fusion according to claim 1, characterized in that: In the above S1, the pedestrian anchor frame B is a coordinate sequence of the upper left corner and lower right corner of the two-dimensional rectangular frame surrounding the pedestrian, the vehicle speed V is the continuous vehicle speed data obtained by the sensor, and the pedestrian joints P include the three-dimensional coordinates of 19 skeleton joints including the head, neck, shoulders, elbows, wrists, hips, knees, ankles, toes, heels and the center of the torso.

3. The method for predicting pedestrian crossing intention based on progressive multimodal feature fusion according to claim 1, characterized in that: In the S2, the multi-expert feature embedding module includes an input layer, an average pooling layer, a score unit, a score adjustment unit, an expert network and an output layer, wherein the score unit includes a fully connected layer and a Softmax activation function, and the expert network includes expert 1, expert 2, ..., expert n; the anchor frame feature is obtained based on the multi-expert feature embedding module , speed characteristics and joint characteristics The process is: S2.

1. Input the pedestrian anchor frame B into the expert network to obtain the corresponding output: ； In the formula, Conv1D represents 1D convolution; represents the convolution kernel weight of the i-th expert, i=1, 2, ..., n, where n is the total number of experts; represents the output of the i-th expert; S2.

2. Use the average pooling layer to average pool the pedestrian anchor box B in the time dimension, input the fully connected layer and the Softmax activation function, calculate the score of each expert, and convert the output of each expert Multiply by the corresponding score to adjust the importance of the expert; S2.

3. Add the weighted outputs of all experts to get the anchor box tensor ; S2.

4. Add position encoding to introduce time series relationship into features and obtain anchor frame features ; S2.

5. Based on S2.1-S2.4, similarly process the vehicle speed V and pedestrian joint P to obtain the speed feature and joint characteristics (The corresponding tensors are the velocity tensor and the joint tensor ).

4. The method for predicting pedestrian crossing intention based on progressive multimodal feature fusion according to claim 1, characterized in that: In S3, the enhanced anchor box features are obtained and speed characteristics The process is: S3.

1. Anchor box features , the keys, values, and queries of self-attention are all , based on the self-attention module, the attention weight matrix is ​​calculated to achieve feature enhancement, and the enhanced anchor frame features are obtained after completing the self-attention operation ; Similarly, the enhanced speed feature is obtained ; Get joint features The method is: S3.

2. The multi-timescale graph convolution module includes an input layer, a linear transformation layer, a multi-scale convolution layer, a ReLU activation function, a fully connected layer, a global average pooling layer, a multi-layer perceptron, a Sigmoid activation function, and an output layer. The multi-scale convolution layer includes three convolution kernels of 3×1, 5×1, and 7×1. The multi-timescale graph convolution module is used to process joint features. , the process is: S3.2.

1. Establish a learnable adjacency matrix that expresses the connection relationship between pedestrian skeleton joints , and normalize it to , Initialize to the identity matrix; S3.2.

2. Using matrix multiplication, using the adjacency matrix Pedestrian joint features Perform aggregation to obtain the aggregation matrix ; S3.2.

3. Use multi-scale convolutional layers to extract the motion timing correlation of the same joint at different time scales: ； In the formula, ReLU is the activation function; Conv2D represents two-dimensional convolution, Indicates convolution kernels, , that is , , , respectively represent the convolution kernels of 3×1, 5×1, and 7×1; Indicates The output of the convolution kernel; S3.2.4, connect the outputs of the three convolution kernels in the channel dimension to obtain feature M; S3.2.5, re-weight the channel dimension of feature M; S3.2.

6. Compress the channel dimension through the fully connected layer and perform average pooling in the joint dimension to obtain joint features. .

5. The method for predicting pedestrian crossing intention based on progressive multimodal feature fusion according to claim 4 is characterized in that: In S3.2.5, the method for re-weighting is: S3.2.5.

1. Perform average pooling on the feature M in the time dimension and joint dimension to generate a global representation y. S3.2.5.

2. Compress and restore the channel dimension features through two fully connected layers to obtain the weight of each channel : ； In the formula, represents the sigmoid activation function; FC represents the fully connected layer; S3.2.5.

3. The weight obtained Applied to the feature M of the original input to adjust the feature M.

6. The method for predicting pedestrian crossing intention based on progressive multimodal feature fusion according to claim 1, characterized in that: In the above S4, the token generation module includes an input layer, a sequence segmentation layer, an attention mechanism, a token splicing layer, a self-attention layer and an output layer. The method for obtaining the aggregated token based on the token generation module is: S4.

1. The token generation module divides the entire sequence into multiple subsequences and introduces a learnable token for each subsequence. Assume that the subsequence length is , then each subsequence is represented as: ； In the formula, , Represents the number of tokens; , That is , , ; express The jth subsequence is divided; the number of subsequences is , T is the total length of the sequence; S4.

2. For each subsequence Introducing Tokens , the token As a query, the corresponding subsequence As keys and values, we apply an attention mechanism to aggregate useful features of subsequences by tokens and obtain ; S4.

3. Tokens of all subsequences Splice along a new dimension to get the splicing token ; S4.

4. Use the self-attention mechanism to model the relationship between different tokens and obtain global information , that is , , .

7. The method for predicting pedestrian crossing intention based on progressive multimodal feature fusion according to claim 6, characterized in that: In the S5, the progressive feature fusion module includes a symmetric channel distributed attention mechanism SCSA and a fusion layer, wherein the SCSA includes a fully connected layer, a Softmax activation function and a layer normalization to obtain a fused token The process is: S5.

1. Use a fully connected layer to transform the input tokens The channel dimension is expanded to twice the original one, and then the expanded features are divided into two parts, namely and , , , represents the jth token of the corresponding branch; S5.2, using the fully connected layer Mapping and weighting through Softmax activation function ; S5.

3. Apply the weights obtained Features Adjustment is performed, expressed as: ； In the formula, represents element-wise multiplication, , Respectively represent the adjusted and ; S5.

4. Using learnable parameters Tokens enhanced by fusing the left and right branches : ； Where LN represents the layer normalization operation; S5.5, use the feature combination of anchor frame-velocity bv, anchor frame-joint bp and velocity-joint vp to perform bimodal fusion respectively, and compare the result of bimodal fusion with Connect, then apply SCSA to enhance the features, realize the fusion of three-modal features, and obtain the fused token .

8. The method for predicting pedestrian crossing intention based on progressive multimodal feature fusion according to claim 7, characterized in that: In S5.5, when dual-modality fusion is performed, the features of the two modalities are first connected, and then the SCSA module is applied to enhance the features and interact between the features, which is expressed as: ； ； ； Where, SCSA is the symmetric channel distributed attention mechanism SCSA; Concat represents the connection operation; , , They are the fused anchor frame-velocity bv, anchor frame-joint bp and velocity-joint vp feature combinations respectively; The trimodal feature fusion is expressed as: ； ； ； ； In the formula, The features representing the final fusion are the fused tokens; , , They are the feature combinations of the fused anchor frame-velocity bv and joint p, anchor frame-joint bp and velocity v, velocity-joint vp and anchor frame b respectively; , , are learnable parameters used to adaptively control the importance of the three modes.

9. The method for predicting pedestrian crossing intention based on progressive multimodal feature fusion according to claim 1, characterized in that: In the S6, the token fusion module includes an input layer, a fully connected layer, a Softmax activation function, a global average pooling layer, a global maximum pooling layer, a normalization layer, a Dropout layer and an output layer. The process of obtaining the fusion feature U is: S6.

1. Use two fully connected layers and then apply the Softmax activation function to generate token weights : ； In the formula, FC represents the fully connected layer; S6.

2. Weight Apply to And perform global average pooling and maximum pooling on the weighted features in the time dimension; S6.

3. Introducing learnable parameters , the two pooling results are fused to form a more comprehensive feature representation, and then the normalization operation LN and regularization Dropout are applied to obtain the fused feature U: ； In the formula, GAP represents global average pooling; GMP represents global maximum pooling; Represents element-wise multiplication.

10. The method for predicting pedestrian crossing intention based on progressive multimodal feature fusion according to claim 6, characterized in that: After constructing each module, the entire network composed of each module is trained using the cross entropy loss function. In order to promote single-modal feature learning, the tokens of each modality are Aggregate and input the probability of pedestrian crossing predicted by the fully connected layer as , and combined with Pred, set the cross entropy loss function Loss to: ； In the formula, Label represents the true label of whether the pedestrian is crossing the street; CE represents cross entropy, which is used to measure the difference between the predicted probability and the true intention; is a learnable parameter used to balance the contribution of different losses.

Images