A pedestrian intention prediction method based on a Transformer multi-modal fusion strategy

Abstract

The application discloses a pedestrian intention prediction method based on a multi-modal fusion strategy of a Transformer structure, at present, the pedestrian intention prediction based on an RNN structure can achieve good performance, but when processing pedestrian intention prediction sequence data, gradient disappearance exists and most models are not deep enough for the fusion between multi-modal characteristics, in order to solve these problems, the application proposes a Transformer model of a multi-modal fusion strategy, fully utilizes the advantages of the Transformer, and introduces a pre-normalization technology for stabilizing a training process and improving the generalization ability of the model, and additionally adds an activation function in the structure for capturing more complex characteristics and patterns, improving gradient flow, effectively processing the complexity of pedestrian movement, and capturing time dependence and multi-modal interaction, better processing the complexity of pedestrian movement.

Description

Technical Field

[0001] This invention relates to the field of pedestrian crossing intention prediction based on Transformer, and solves recommendation-related problems by using Transformer and multimodal fusion technology. Background Technology

[0002] In daily life, cars are an indispensable means of transportation. With the continuous development of science and technology, autonomous driving technology and autonomous vehicles have gradually appeared in the public eye. Ensuring the safety of road users in traffic scenarios is a prerequisite for the widespread application of autonomous vehicles.

[0003] Accurately predicting pedestrian behavior can not only prevent potential collisions and ensure safety, but also optimize route planning and vehicle speed adjustments, alleviating traffic congestion. When pedestrians intend to cross the road while a car is in motion, they exhibit a range of behavioral characteristics. For example, they may pause, turn around to observe oncoming vehicles, and wait for them to move away or for a safe crossing opportunity to emerge. After confirming it is safe, pedestrians may begin walking, or even accelerate their crossing.

[0004] Common pedestrian trajectory models are based on recurrent neural networks (RNNs). These models typically use the pedestrian's historical location information as input, then learn the pedestrian's movement patterns through the RNN network to predict the pedestrian's future location. In pedestrian intent recognition technology for autonomous vehicles, much research focuses on models such as LSTM and GRU. Examples include PCPA, SF-GRU, IPVO-LSTM, and Social LSTM. PCPA uses an LSTM encoder-decoder architecture to predict future pedestrian behavior by analyzing current and historical bounding box data; SF-GRU uses stacked GRUs to process different modalities to predict pedestrian crossings.

[0005] While traditional RNN-based models excel at processing sequence data, they suffer from vanishing or exploding gradients and struggle with parallel data processing, limiting computational efficiency. LSTM models also present challenges in pedestrian trajectory prediction. For example, LSTM models can be sensitive to small changes in the input data, potentially leading to performance degradation in challenging environments. Furthermore, LSTM models require significant computational resources and time for training and inference, which may limit their use in some real-time applications.

[0006] Since the Transformer model was introduced in 2017, researchers have developed numerous derivative studies based on it. Compared to Recurrent Neural Networks (RNNs), the Transformer model offers several advantages when processing sequential data: powerful parallel computing capabilities, employing a self-attention mechanism that allows for parallel processing of inputs regardless of sequence order; better global information capture, as the Transformer's self-attention mechanism can capture long-range dependencies in the input sequence, thus better capturing global information; and superior modeling capabilities and easier optimization and parameter tuning. Applying the Transformer model to pedestrian crossing intention recognition improves attention computation across multiple modalities, making it closer to human drivers' judgments of pedestrian intentions. Therefore, research has begun to explore Transformer-based pedestrian crossing intention prediction methods, such as PIT and Intformer. Summary of the Invention

[0007] This invention addresses the limitations and challenges of existing solutions by proposing a pedestrian intent prediction method based on a Transformer multimodal fusion strategy. It improves the Transformer model by introducing pre-normalization techniques and an additional activation function after the attention layer, processing and fusing different data modalities to provide a comprehensive and robust approach to pedestrian behavior understanding. The pre-normalization activation encoding module preprocesses and encodes the input information for each modality, while the modal pre-normalization activation decoding module further processes the modal information and learns the correlations between modalities. A pre-normalization method, different from traditional Transformer models, is employed, which helps stabilize the training process and improve the model's generalization ability in pedestrian intent prediction tasks. An additional activation function is added before the Feed Forward Network (FFN) layer to capture more complex features and patterns, improving gradient flow.

[0008] The present invention provides a pedestrian intent prediction method based on a Transformer multimodal fusion strategy, comprising the following steps:

[0009] S1: Pre-normalization activation encoding module

[0010] S2: Modal pre-normalization activation decoding module

[0011] S3: Pedestrian crossing intention prediction module

[0012] S4: Model Training

[0013] Furthermore, the specific steps of S1 are as follows:

[0014] S11. The Transformer model takes a one-dimensional vector as input, so it projects data from different modalities into a one-dimensional vector space to reduce dimensionality and extract features. Then, it combines positional embedding to preserve time-series information.

[0015] X m,p =X m +p

[0016] in, m represents different modalities, such as pedestrian pose or pedestrian bounding box, and p represents the position embedding;

[0017] S12. The different modal vectors obtained from S11, after position embedding, are first normalized by layer. The output of the neurons is normalized in the channel dimension to ensure that the activation values ​​in the same layer have the same distribution. After residual connection, an attention mechanism is used to capture the dependencies between sequences, focusing on specific parts of the input data and dynamically focusing on different parts of the input, thereby enhancing the correlation between the modalities themselves.

[0018] z′ m,e =Attention(LayerNorm(x) m,p )+x m,p )

[0019] Attentention() is the attention mechanism, and LayerNorm() is the layer normalization.

[0020] S13. After the attention mechanism is completed, the data passes through a LayerNorm layer again to adjust and stabilize the input to subsequent network layers. Then, the data enters an Activation & Add module, which may involve non-linear activation functions to introduce non-linear characteristics into the model. The output is then added back to the previous layer's output through residual connections to enhance the model's learning ability.

[0021] z″ m,e =LayerNorm(GELU(LayerNorm(z′ m,e ))+z′ m,e )

[0022] Where GELU() is a nonlinear activation function and Gaussian error linear unit activation function;

[0023] S14. Finally, the data flows through a Feed Forward Network (FFN) and another LayerNorm layer. The Feed Forward Network contains multiple fully connected layers for further feature extraction and learning of complex function mappings. Ultimately, the data after layer normalization completes one cycle of this process and can proceed to the next cycle or stage of processing.

[0024] z m,e =LayerNorm(FFN(z″) m,e )+z″ m,e )

[0025] FFN stands for feedforward network.

[0026] Furthermore, the specific steps of S2 are as follows:

[0027] S21. After each mode is processed by the pre-normalization encoding module, the z corresponding to each mode is obtained. m,e The modes are combined in pairs;

[0028] Specifically, firstly, the two modes z m,e The input streams are simultaneously passed through a LayerNorm layer, which helps control numerical stability during training and allows each layer to learn an appropriate hidden representation. Then, the two normalized input streams pass through a CrossAttention module. In the CrossAttention mechanism, one input stream... The elements of one input stream are used as queries, while the elements of another input stream are used as queries. As keys and values, this mechanism allows the model to focus on relevant information from another input stream while processing one input stream.

[0029]

[0030] CrossAttention() is the cross-attention mechanism;

[0031] S22. After completing the cross-attention processing, the output is passed through a LayerNorm layer to further stabilize the network's activation values. Then, the data enters an Activation & Add module, which may contain a non-linear activation function to provide the network with non-linear transformation capabilities. The residual connection (Add) adds the current layer's output to its input, which helps gradient flow and reduces the vanishing problem during training.

[0032] z″ m,d=LayerNorm(GELU(LayerNorm(z′ m,d ))+z′ m,d )

[0033] S23. Finally, the data is processed through a feedforward network (FFN) and another layer normalization layer; the feedforward network typically consists of dense layers used to extract higher-level feature representations.

[0034] z m,d =LayerNorm(FFN(z″) m,d )+z″ m,d ).

[0035] Furthermore, the specific steps of S3 are as follows:

[0036] S31. The z-axis of each mode is obtained by activating the decoding module through modal pre-normalization. m,d , all modal z m,d After concatenating them, we get Z. d :

[0037]

[0038] Where Concate[...] is the vector concatenation operation;

[0039] Then, each mode is projected into a one-dimensional vector X. m Concatenate is also performed, and the result is fed into the pre-normalization activation decoding module to obtain Z. q :

[0040] Z′ q,e =Attention(LayerNorm(X) m )+X m )

[0041] Z″ q,e =LayerNorm(GELU(LayerNorm(Z′ q,e ))+Z′ q,e )

[0042] Z q =LayerNorm(FFN(Z″) q,e )+Z″ q,e )

[0043] S32, obtain Z q and Z dThe input is then fed into the modality pre-normalization activation decoding module to obtain Z. c :

[0044] Z′ c =CrossAttention(LayerNorm(Z) q )+Z q LayerNorm(Z) d )+Z d )

[0045] Z″ c =LayerNorm(GELU(LayerNorm(Z′ c ))+Z′ c )

[0046] Z c =LayerNorm(FFN(Z′) c )+Z′ c )

[0047] Then Z c The input is fed into the prediction module for processing; in the prediction module, Z... c The input is fed into a multilayer perceptron and activated by the sigmoid function:

[0048] Z c =MPNAD(Z) q Z d )

[0049]

[0050] In this context, MPNAD stands for Modality Pre-Normalized Activation Decoding Module, and Sigmoid() is the activation function. This is the predicted result.

[0051] Furthermore, the specific steps of S4 are as follows:

[0052] S41. In the task of predicting pedestrian crossing intentions, we face a typical binary classification problem, namely, determining whether a pedestrian intends to cross. The cross-entropy loss function is used to quantify the difference between the model's prediction and the actual label. Specifically, for each category, the model predicts a probability of p. t The probability of not traversing is 1-p. t The formula for calculating cross-entropy loss is as follows:

[0053]

[0054] Where Lce For cross-entropy loss, y t For actual labels, y represents the actual situation when pedestrians cross. t =1, otherwise y t =0;

[0055] To further improve the accuracy of predictions, an auxiliary loss function L was used. r The aim is to reduce the output z of the model in the premodal normalization activation module. d [t] and the final predicted output z d The difference between [T] is expressed as:

[0056]

[0057] This auxiliary loss helps ensure the consistency and stability of the model's predictions across a series of outputs before the final decision; ultimately, we get:

[0058] L = L ce +L r

[0059] Among them, L r Let L be the auxiliary function, and L be the total loss.

[0060] This loss strategy encourages the model to benefit from the decoder's final representation at an early stage of prediction, thereby allowing the model to observe the contextual information of the entire sequence before pedestrian actions occur, enhancing the model's overall understanding and predictive ability of pedestrian crossing intentions. Attached Figure Description

[0061] Figure 1 This is a flowchart of the method.

[0062] Figure 2 Flowchart for Activating the Pre-normalization Encoding Module

[0063] Figure 3 Flowchart for Modal Pre-normalization Activation Decoding Module Detailed Implementation

[0064] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to examples and accompanying drawings.

[0065] As attached Figure 1 As shown, a pedestrian intent prediction method based on the Transformer multimodal fusion strategy includes the following steps:

[0066] Step 1: Pre-normalization activation encoding stage:

[0067] The Transformer model takes a one-dimensional vector as input, so it projects data from different modalities into a one-dimensional vector space to reduce dimensionality and extract features. Then, it combines positional embedding to preserve time-series information.

[0068] X m,p =X m +p

[0069] in, m represents different modalities, such as pedestrian pose or pedestrian bounding box, and p represents the position embedding;

[0070] The obtained modality vectors with different positions embedded are first normalized by layer, and the output of the neurons is normalized in the channel dimension to make the activation values ​​in the same layer have the same distribution. After residual connection, the attention mechanism is used to capture the dependencies between sequences, focusing on specific parts of the input data and dynamically focusing on different parts of the input, thereby enhancing the correlation between the modalities themselves.

[0071] z′ m,e =Attention(LayerNorm(x) m,p )+x m,p )

[0072] Attentention() is the attention mechanism, and LayerNorm() is the layer normalization.

[0073] After the attention mechanism is completed, the data passes through a LayerNorm layer to adjust and stabilize the input to subsequent network layers. Then, the data enters an Activation & Add module, which may involve non-linear activation functions to introduce non-linear characteristics into the model. The output is then added back to the previous layer's output via residual connections to enhance the model's learning ability.

[0074] z″ m,e =LayerNorm(GELU(LayerNorm(z′ m,e ))+z′ m,e )

[0075] Where GELU() is a nonlinear activation function and Gaussian error linear unit activation function;

[0076] Finally, the data flows through a feed-forward network (FFN) and another layer normalization layer. The feed-forward network contains multiple fully connected layers for further feature extraction and learning of complex function mappings. Ultimately, the data after layer normalization completes one cycle of the process and can proceed to the next cycle or stage of processing.

[0077] z m,e =LayerNorm(FFN(z″) m,e )+z″ m,e )

[0078] FFN stands for feedforward network.

[0079] Step 2: Modal pre-normalization activation decoding stage:

[0080] After each mode is processed by the pre-normalization encoding module, the z-value corresponding to each mode is obtained. m,e The modes are then combined in pairs; specifically, first, two modes z... m,e The input streams are simultaneously passed through a LayerNorm layer, which helps control numerical stability during training and allows each layer to learn an appropriate hidden representation. Then, the two normalized input streams pass through a CrossAttention module. In the CrossAttention mechanism, one input stream... The elements of one input stream are used as queries, while the elements of another input stream are used as queries. As keys and values, this mechanism allows the model to focus on relevant information from another input stream while processing one input stream.

[0081]

[0082] CrossAttention() is the cross-attention mechanism;

[0083] After cross-attention processing, the output is passed through a LayerNorm layer to further stabilize the network's activation values. Then, the data enters an Activation & Add module, which may contain a non-linear activation function to provide the network with non-linear transformation capabilities. The residual connection (Add) adds the current layer's output to its input, which helps gradient flow and reduces the vanishing gradient problem during training.

[0084] z″ m,d =LayerNorm(GELU(LayerNorm(z′ m,d))+z′ m,d )

[0085] Finally, the data passes through a feedforward network (FFN) and another layer normalization layer; the feedforward network typically consists of dense layers used to extract higher-level feature representations.

[0086] z m,d =LayerNorm(FFN(z″) m,d )+z″ m,d ).

[0087] Step 3: Pedestrian Intent Prediction Stage

[0088] The z-values ​​of each mode are obtained by the modality pre-normalization activation decoding module. m,d , all modal z m,d After concatenating them, we get Z. d :

[0089]

[0090] Where Concate[...] is the vector concatenation operation;

[0091] Then, each mode is projected into a one-dimensional vector X. m Concatenate is also performed, and the result is fed into the pre-normalization activation decoding module to obtain Z. q :

[0092] Z′ q,e =Attention(LayerNorm(X) m )+X m )

[0093] Z″ q,e =LayerNorm(GELU(LayerNorm(Z′ q,e ))+Z′ q,e )

[0094] Z q =LayerNorm(FFN(Z″) q,e )+Z″ q,e )

[0095] Get Z q and Z d The input is then fed into the modality pre-normalization activation decoding module to obtain Z. c :

[0096] Z′ c=CrossAttention(LayerNorm(Z) q )+Z q LayerNorm(Z) d )+Z d )

[0097] Z″ c =LayerNorm(GELU(LayerNorm(Z′ c ))+Z′ c )

[0098] Z c =LayerNorm(FFN(Z′) c )+Z′ c )

[0099] Then Z c The input is fed into the prediction module for processing; in the prediction module, Z... c The input is fed into a multilayer perceptron and activated by the sigmoid function;

[0100] Z c =MPNAD(Z) q Z d )

[0101]

[0102] In this context, MPNAD stands for Modality Pre-Normalized Activation Decoding Module, and Sigmoid() is the activation function. This is the predicted result.

[0103] Step 4: Model Training Phase

[0104] In the task of predicting pedestrian crossing intentions, a typical binary classification problem is encountered, namely, determining whether a pedestrian intends to cross. The Cross-Entropy Loss function is used to quantify the difference between the model's prediction and the actual label. Specifically, for each category, the model predicts a probability of p. t The probability of not traversing is 1-p. t The formula for calculating cross-entropy loss is as follows:

[0105]

[0106] Where L ce For cross-entropy loss, y t For actual labels, y represents the actual situation when pedestrians cross. t =1, otherwise y t =0;

[0107] To further improve the accuracy of predictions, an auxiliary loss function L was used. r The aim is to reduce the output z of the model in the premodal normalization activation module. d [t] and the final predicted output z d The difference between [T] is expressed as:

[0108]

[0109] This auxiliary loss helps ensure the consistency and stability of the model's predictions across a series of outputs before the final decision; ultimately, we get:

[0110] L = L ce +L r

[0111] Among them, L r Let L be the auxiliary function, and L be the total loss.

[0112] This loss strategy encourages the model to benefit from the decoder's final representation at an early stage of prediction, thereby allowing the model to observe the contextual information of the entire sequence before pedestrian actions occur, enhancing the model's overall understanding and predictive ability of pedestrian crossing intentions.

[0113] As attached Figure 1 As shown, the pre-normalization activation encoding module includes the following steps:

[0114] Step 1: Location Embedding Stage

[0115] The Transformer model takes a one-dimensional vector as input, so it projects data from different modalities into a one-dimensional vector space to reduce dimensionality and extract features. Then, it combines positional embedding to preserve time-series information.

[0116] X m,p =X m +p

[0117] in, m represents different modalities, such as pedestrian pose or pedestrian bounding box, and p represents the position embedding;

[0118] Step 2: Attention Mechanism Stage

[0119] The obtained modality vectors with different positions embedded are first normalized by layer, and the output of the neurons is normalized in the channel dimension to make the activation values ​​in the same layer have the same distribution. After residual connection, the attention mechanism is used to capture the dependencies between sequences, focusing on specific parts of the input data and dynamically focusing on different parts of the input, thereby enhancing the correlation between the modalities themselves.

[0120] z′ m,e =Attention(LayerNorm(x) m,p )+x m,p );

[0121] Step 3: Activation Function Activation Phase:

[0122] After the attention mechanism is completed, the data passes through a LayerNorm layer to adjust and stabilize the input to subsequent network layers. Then, the data enters an Activation & Add module, which may involve non-linear activation functions to introduce non-linear characteristics into the model. The output is then added back to the previous layer's output via residual connections to enhance the model's learning ability.

[0123] z″ m,e =LayerNorm(GELU(LayerNorm(z′ m,e ))+z′ m,e );

[0124] Step 4: Feature Extraction Stage

[0125] Finally, the data is processed through a feedforward network (FFN) and another layer normalization layer; the feedforward network typically consists of dense layers used to extract higher-level feature representations.

[0126] z m,d =LayerNorm(FFN(z″) m,d )+z″ m,d );

[0127] As attached Figure 3 As shown, the modality pre-normalization activation decoding module includes the following steps:

[0128] Step 1: Cross-attention mechanism stage:

[0129] After each mode is processed by the pre-normalization encoding module, the z-value corresponding to each mode is obtained. m,e The modes are then combined in pairs; specifically, first, two modes z... m,e The input streams are simultaneously passed through a LayerNorm layer, which helps control numerical stability during training and allows each layer to learn an appropriate hidden representation. Then, the two normalized input streams pass through a CrossAttention module. In the CrossAttention mechanism, one input stream... The elements of one input stream are used as queries, while the elements of another input stream are used as queries. As keys and values, this mechanism allows the model to focus on relevant information from another input stream while processing one input stream.

[0130]

[0131] Step 2: Activation Function Activation Phase:

[0132] After cross-attention processing, the output is passed through a LayerNorm layer to further stabilize the network's activation values. Then, the data enters an Activation & Add module, which may contain a non-linear activation function to provide the network with non-linear transformation capabilities. The residual connection (Add) adds the current layer's output to its input, which helps gradient flow and reduces the vanishing gradient problem during training.

[0133] z″ m,d =LayerNorm(GELU(LayerNorm(z′ m,d ))+z′ m,d );

[0134] Step 3: Feature Extraction Stage

[0135] Finally, the data is processed through a feedforward network (FFN) and another layer normalization layer; the feedforward network typically consists of dense layers used to extract higher-level feature representations.

[0136] z m,d =LayerNorm(FFN(z″) m,d )+z″ m,d ).

Claims

1. A pedestrian intent prediction method based on a Transformer multimodal fusion strategy, characterized in that... Includes the following steps: S1: Pre-normalization activation encoding module; S2: Modal pre-normalization activates the decoding module; The specific steps are as follows: S21. After each mode is processed by the pre-normalization encoding module, the z corresponding to each mode is obtained. m,e The modalities are combined in pairs; Specifically, firstly, the two modes z m,e The normalization layer, which is passed through the training layer at the same time, helps to control the numerical stability during training and allows each layer to learn a suitable hidden representation. Next, the two normalized input streams pass through a cross-attention module; In the cross-attention mechanism, an input stream One element is used as the query, while the other element is used as the input stream. As keys and values, this mechanism allows the model to focus on relevant information from another input stream while processing one input stream: CrossAttention() is the cross-attention mechanism, and LayerNorm() is the layer normalization mechanism. S22. After completing the cross-attention processing, the output is passed through a normalization layer to further stabilize the activation values ​​of the network. Then, the data enters an activation and addition module, which contains a non-linear activation function to provide the network with the ability to undergo non-linear transformations; the residual connection adds the output of the current layer to its input. z″ m,d =LayerNorm(GELU(LayerNorm(z′ m,d ))+z′ m,d ); S23. Finally, the data is passed through a feedforward network and another layer normalization layer; the feedforward network consists of dense layers used to extract higher-level feature representations: z m,d =LayerNorm(FFN(z″ m,d )+z″ m,d ); S3: Pedestrian crossing intention prediction module; S4: Model training.

2. The pedestrian intent prediction method based on the Transformer multimodal fusion strategy according to claim 1, characterized in that... In step S1: The specific steps of pre-normalized activation encoding are as follows: S11. The Transformer model takes a one-dimensional vector as input. Therefore, it projects data from different modalities into a one-dimensional vector space to reduce dimensionality and extract features. Then, it combines positional embeddings to preserve time-series information. X m,p =X m +p in, m represents different modes, and p represents position embedding; S12. The different modal vectors obtained from S11, after position embedding, are first normalized by layer. This normalization process normalizes the neuron outputs along the channel dimension, ensuring that the activation values ​​in the same layer have the same distribution. After residual connections, an attention mechanism is used to capture the dependencies between sequences, focusing on specific parts of the input data and dynamically paying attention to different parts of the input, thereby enhancing the correlation between the modalities themselves. z′ m,e =Attention(LayerNorm(x m,p )+x m,p ) Attentention() is the attention mechanism; S13. After the attention mechanism is completed, the data passes through a normalization layer to adjust and stabilize the input to subsequent network layers. Then, the data enters an activation and addition module, which involves a non-linear activation function to introduce non-linear characteristics into the model. The output is then added back to the previous layer's output via residual connections to enhance the model's learning ability. z′ m ′ ,e =LayerNorm(GELU(LayerNorm(z′ m,e ))+z′ m,e ) Where GELU() is a nonlinear activation function and Gaussian error linear unit activation function; S14. Finally, the data flows through a feedforward network and another layer normalization layer; the feedforward network contains multiple fully connected layers for further feature extraction and learning of complex function mappings; Finally, the data after layer normalization completes one full cycle of the process and proceeds to the next cycle or stage of processing: z m,e =LayerNorm(FFN(z′ m ′ ,e )+z′ m ′ ,e ) FFN stands for feedforward network.

3. The pedestrian intent prediction method based on the Transformer multimodal fusion strategy according to claim 1, characterized in that... In step S3: The specific steps of the pedestrian crossing intention prediction module are as follows: S31. The z-axis of each mode is obtained by activating the decoding module through modal pre-normalization. m,d , all modal z m,d After piecing them together, we get Z. d : Where Concate[...] is the vector concatenation operation; Then, each mode is projected into a one-dimensional vector X. m The data is also concatenated and input into the pre-normalization activation decoding module to obtain Z. q : Z q ′ ,e =Attention(LayerNorm(X m )+X m ) Z q ″ ,e =LayerNorm(GELU(LayerNorm(Z q ′ ,e ))+Z q ′ ,e ) Z q =LayerNorm(FFN(Z q ″ ,e )+Z q ″ ,e ) S32, obtain Z q and Z d The input is then fed into the modality pre-normalization activation decoding module to obtain Z. c : WITH c ′=CrossAttention(LayerNorm(Z q )+Z q ,LayerNorm(Z d )+Z d ) Z c ″=LayerNorm(GELU(LayerNorm(Z c ′))+Z c ′) Z c =LayerNorm(FFN(Z c ′)+Z c ′) Then Z c The input is fed into the prediction module for processing; in the prediction module, Z... c The input is fed into a multilayer perceptron and activated by the sigmoid function: WITH c =MPNAD(Z q ,WITH d ) In this context, MPNAD stands for Modality Pre-Normalized Activation Decoding Module, and Sigmoid() is the activation function. For the predicted results; MLP is a fully connected neural network that compresses the output into probability values ​​through weighted summation and activation transformation, making it suitable for binary classification. Then, the output of MLP is converted into probability values ​​through the sigmoid activation function, which is very useful in binary classification problems where the output is interpreted as the probability of belonging to a certain class.

4. The pedestrian intent prediction method based on the Transformer multimodal fusion strategy according to claim 1, characterized in that... In step S4: The specific steps for model training are as follows: S41. In the task of predicting pedestrian crossing intentions, we face a typical binary classification problem, namely, determining whether a pedestrian intends to cross; the cross-entropy loss function is used to quantify the difference between the model prediction and the actual label. Specifically, for each category, the model predicts a probability of p. t The probability of not traversing is 1-p. t The formula for calculating cross-entropy loss is as follows: Where L ce For cross-entropy loss, y t For actual labels, y represents the actual situation when pedestrians cross. t =1, otherwise y t =0; To further improve the accuracy of predictions, an auxiliary loss function L was used. r The aim is to reduce the output z of the model in the premodal normalization activation module. d [t] and the final predicted output z d The difference between [T] is expressed as: This auxiliary loss helps ensure that the model's predictions remain consistent and stable across a series of outputs before the final decision; The final result is: L=L ce +L r Among them, L r Let L be the auxiliary function, and L be the total loss. This loss strategy encourages the model to benefit from the decoder's final representation at an early stage of prediction, thereby allowing the model to observe the contextual information of the entire sequence before pedestrian actions occur, enhancing the model's overall understanding and predictive ability of pedestrian crossing intentions.

Images