Pedestrian passing intention prediction method based on cross-modal attention interaction

Abstract

The invention provides a pedestrian passing intention prediction method based on cross-modal attention interaction, and the method comprises the steps: obtaining the speed information of a vehicle and the coordinate information of a human body bounding box, and obtaining the speed modal features of the vehicle and the coordinate modal features of the human body bounding box through a cross-modal attention interaction module; fusing the self-vehicle speed modal feature and the human body bounding box coordinate modal feature through a feature fusion module to obtain a fused feature; processing the fusion feature by using a Transform model to obtain a pedestrian passing intention score; and predicting the pedestrian passing intention based on at least the pedestrian passing intention score. According to the method, a cross-modal attention interaction module and a feature fusion module are adopted, complementarity and difference between different modal features are fully utilized to be fused into fusion features, after the fusion features are coded and decoded based on a Transform model, whether pedestrians pass through a predicted time node or not is judged through an output pedestrian pass-through intention score, and the pedestrian pass-through intention prediction accuracy is improved. And the prediction performance is improved.

Description

Technical Field

[0001] This invention belongs to the field of traffic perception and control technology, specifically relating to a method for predicting pedestrian crossing intentions based on cross-modal attention interaction. Background Technology

[0002] With the development of autonomous driving technology, traffic control faces new challenges and opportunities. Future traffic control will place greater emphasis on scientific, precise, and intelligent approaches. The rapid development of autonomous driving brings both new opportunities and challenges to the automotive industry. Road traffic safety issues arising from intelligent vehicles are gradually becoming more prominent, causing significant inconvenience to urban traffic management and control systems and people's travel, leading to continuous public questioning of the safety of autonomous driving.

[0003] Pedestrians are among the most vulnerable and exposed road users. According to statistics from a World Health Organization report, more than half of all traffic accident fatalities worldwide each year are among vulnerable road users. Therefore, protecting the safety of pedestrians and other vulnerable road users is of paramount importance. One of the effective measures to protect pedestrian safety in the past has been to avoid conflicts between pedestrians and vehicles.

[0004] With the rapid development of advanced driver assistance and autonomous driving technologies, autonomous vehicles can use onboard sensors such as vision, lidar, and millimeter-wave radar to perceive the surrounding road environment, vehicles, and pedestrians. This allows them to predict pedestrian behavior, enabling them to plan and make decisions in advance and avoid pedestrian-vehicle conflicts. In the context of traffic management, pedestrian-vehicle conflicts frequently occur at crosswalks. Accurately predicting whether pedestrians will cross in front of vehicles can reduce pedestrian delays and traffic congestion, improve road efficiency, and thus help autonomous vehicles find a balance between safety and efficiency.

[0005] In recent years, the accuracy of pedestrian crossing prediction has been gradually improved from various aspects. For example, to enhance prediction accuracy, single-modal data such as image frame sequences, single-frame static images, and human poses are used for pedestrian crossing prediction. Alternatively, the effectiveness of pedestrian crossing prediction can be improved by adding traffic scene data of different modalities, such as pedestrian detection boxes, grid images, and semantic segmentation information. Another approach is to select finely designed network structures (various attention mechanisms, scene relationship modeling, and visual information extraction) for pedestrian crossing prediction.

[0006] It can be seen that although existing technologies have solved many problems and significantly improved the prediction effect in predicting pedestrian crossing tasks, the following problems still exist: (1) Using single-modal data to predict pedestrian crossing does not take into account the influence of other information such as traffic scenes on pedestrian crossing intentions; (2) Some studies have added multimodal data to improve the prediction effect, but have not taken into account the differences and complementarities between modalities, nor have they had an effective and efficient feature fusion method; (3) The input features of pedestrian crossing prediction are mainly in the form of time series. Recurrent neural networks are directly used to extract information of various modal data in the time dimension, which makes a large number of linear layers prone to overfitting and resulting in poor prediction effect. Furthermore, no consideration has been given to how to select a suitable encoder and decoder. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of feature fusion between different modal data in existing technologies, and to provide a pedestrian crossing intention prediction method based on cross-modal attention interaction. This method employs a cross-modal attention interaction module and a feature fusion module, which fully utilizes the complementarity and differences between features of different modalities to fuse them into a fused feature. After encoding and decoding the fused feature based on the Transformer model, the pedestrian crossing intention score is output to determine whether the pedestrian will cross at the predicted time point, thereby improving prediction performance.

[0008] To achieve the above objectives, the technical solution provided by this invention is:

[0009] This invention provides a method for predicting pedestrian crossing intentions based on cross-modal attention interaction, comprising:

[0010] Step 1: Obtain vehicle speed information and human bounding box coordinate information, and obtain vehicle speed modal features and human bounding box coordinate modal features through the cross-modal attention interaction module;

[0011] Step 2: The vehicle velocity mode features and the human bounding box coordinate mode features obtained in Step 1 are fused together using a feature fusion module to obtain fused features;

[0012] Step 3: Process the fused features obtained in Step 2 using the Transformer model to obtain pedestrian crossing intention scores;

[0013] Step 4: Predict pedestrian crossing intentions based on the pedestrian crossing intention scores.

[0014] As a further limitation of the present invention, step one includes:

[0015] Acquire vehicle speed information and pedestrian bounding box coordinates from the vehicle's perspective;

[0016] The vehicle speed information and the human bounding box coordinate information are processed into vehicle speed data and human bounding box data;

[0017] The cross-modal attention interaction module performs cross-modal attention interaction on the vehicle speed data and the human bounding box data to obtain vehicle speed modal features and human bounding box coordinate modal features.

[0018] As a further limitation of the present invention, in step one, the vehicle speed information is obtained based on the vehicle speed, and the expression is:

[0019] Formula (1)

[0020] In formula (1), This indicates the vehicle's speed information. Indicates pedestrian, Represents the set of real numbers. The number of frames in a series of time frames. Feature dimensions representing vehicle speed;

[0021] In step one, the human body bounding box coordinate information is obtained based on the position of the pedestrian's human body bounding box in the vehicle's view, and the expression is:

[0022] Formula (2)

[0023] In formula (2), This represents the coordinate information of the human body bounding box. pedestrian The bounding box coordinates, The feature dimension representing the bounding box coordinates; where, pedestrian exist Bounding box coordinates at time Represented as: , In the formula, This represents the x-coordinate of the top-left corner of the bounding box. The y-coordinate represents the top-left corner of the bounding box. This represents the x-coordinate of the bottom right corner of the bounding box. This represents the ordinate of the bottom right corner of the bounding box.

[0024] As a further limitation of the present invention, the process of processing the vehicle speed information and the human body bounding box coordinate information into vehicle speed data and human body bounding box data in step one includes:

[0025] Vehicle speed information Human bounding box coordinate information The feature dimensions are mapped to the feature dimensions of the fused features through the processing unit. Obtain vehicle speed data and human body bounding box data ;

[0026] The cross-modal attention interaction module in step one is represented as follows:

[0027] Formula (3)

[0028] In formula (3), Attention mechanisms representing multiple modal values, Indicates based on the first Query matrix for various modalities Indicates the first Key data for each modality Indicates the first Value data for each modality; express Attention and The chain operation of attention heads, This represents the output weight matrix. Indicates the first Pay attention to the head. This represents the attention mechanism. Represents the linear transformation matrix of the query matrix. The linear transformation matrix representing the key data. The linear transformation matrix representing the value data. Represents the query matrix. This represents the transpose of the key matrix. Represents a value matrix, Indicates the scaling factor;

[0029] After processing with formula (3), the vehicle speed modal characteristics are obtained. Human bounding box coordinate modal features .

[0030] As a further limitation of the present invention, step two includes:

[0031] The vehicle speed modal features and the human bounding box coordinate modal features obtained in step one are multiplied by their respective weights and summed using the feature fusion module to complete feature fusion, outputting the fused feature of the pedestrian's crossing intention. Feature fusion refers to the use of feature fusion methods. Fusion of vehicle speed modal features Human bounding box coordinate modal features The process is as follows:

[0032] Formula (4)

[0033] In formula (4), Represents the set of real numbers. The number of frames in a series of time frames. The feature dimension representing the fused features;

[0034] Fusion features after feature fusion , represented as:

[0035]

[0036] In the formula, Indicates pedestrian, Represents the set of real numbers. The number of frames in a series of time frames. The feature dimension representing the fused features.

[0037] As a further limitation of the present invention, step three includes:

[0038] The encoder of the Transformer model is used to output global features for the fused features;

[0039] The global features are processed by the Transformer model and then decoded by its decoder to output the decoding result, which is the predicted target feature.

[0040] The predicted target features are input into a classifier to obtain a pedestrian crossing intention score.

[0041] As a further limitation of the present invention, the encoder of the Transformer model is used to output global features for the fused features, including:

[0042] Fusion features of pedestrian crossing intentions after fusion using the Transformer model Obtain coding features ; Encoded features after encoding Represented as: In the formula, This represents the feature dimension after encoding. , Represents the set of real numbers. Represents the number of frames in a series of time frames, selecting the encoded features. The average value is used as a global feature. ;

[0043] The global features are processed by the Transformer model, decoded by its decoder, and the decoded result is output. The decoded result is the predicted target features, including:

[0044] Using the Transformer model, global features are analyzed through formula (5). Decode it as follows:

[0045] Formula (5)

[0046] In formula (5), Represents the query matrix. Represents the query matrix in which the first... A query vector, Represents the set of real numbers. The number of frames in a series of time frames. Represents a set of dimensions that can be learned and queried;

[0047] The propagation process of each Transformer module in the decoder is as follows:

[0048] Formula (6)

[0049] In formula (6), Indicates the first The layer self-attention module updates the query matrix after residual connections and layer normalization. Indicates the normalization layer. This indicates a multi-head self-attention mechanism for the mask. This indicates a multi-head self-attention mechanism. Indicates the first The query matrix of the layer, Indicates the first Multi-head attention module usage and global features The calculated output, This indicates the decoding result. This represents a feedforward neural network;

[0050] The predicted target features are input into a classifier to obtain a pedestrian crossing intention score, including: feeding the predicted target features corresponding to the decoding result into the classifier to obtain the pedestrian crossing intention score at the target time, expressed as:

[0051] Formula (7)

[0052] In formula (7), The score represents the pedestrian's crossing intention at the target time. Represents a classifier. This indicates the decoding result. Indicates the flattening operation. This represents the weight matrix of the fully connected layer. This represents the bias vector.

[0053] As a further limitation of the present invention, step four includes:

[0054] Based on the predicted data, including the pedestrian crossing intention score, it is determined whether a pedestrian will cross at the predicted time point, and the judgment result is obtained.

[0055] Predict the travel intentions of pedestrians based on the judgment results.

[0056] As a further limitation of the present invention, the aforementioned prediction data based on at least the pedestrian crossing intention score includes:

[0057] The pedestrian crossing intention score is based on the output of the trained Transformer model.

[0058] or,

[0059] The predicted data is composed of the pedestrian crossing intention score output by the trained Transformer model and the driving data of the surrounding vehicles at the current moment obtained by the vehicle.

[0060] The advantages of this invention are:

[0061] This invention employs a cross-modal attention interaction module and a feature fusion module, fully utilizing the complementarity and differences between features of different modalities to fuse them into a fused feature. After encoding and decoding the fused feature based on the Transformer model, the output pedestrian crossing intention score determines whether the pedestrian is crossing at the predicted time node. This solves the problems of existing prediction methods not considering the efficiency of feature fusion of various modalities and relying on a single convolutional neural network or recurrent neural network for encoding, thereby improving prediction performance.

[0062] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0063] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0064] Figure 1 The present invention provides a flowchart of a method for predicting pedestrian crossing intentions based on cross-modal attention interaction;

[0065] Figure 2 The framework diagram of the pedestrian crossing intention prediction network provided by this invention;

[0066] Figure 3 : Schematic diagram of the feature fusion module provided by this invention. Detailed Implementation

[0067] The embodiments of the present invention are described in detail below. These embodiments are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0068] To improve prediction performance, most existing pedestrian intention prediction methods focus on the model architecture and loss function selection of convolutional neural networks and recurrent neural networks for visual or non-visual inputs. However, these existing methods do not take into account the efficiency of feature fusion of different modalities in pedestrian intention prediction and rely more on a single convolutional neural network or recurrent neural network for encoding, thus failing to capture the correlation between long-term sequences well.

[0069] The invention patent publication with publication number 118781575A discloses a pedestrian intention prediction method based on a multimodal fusion strategy using the Transformer architecture. This method addresses the issues of gradient vanishing and insufficient fusion of multimodal features in most models when processing pedestrian intention prediction sequence data. It fully utilizes the advantages of the Transformer model, introduces pre-normalization technology for stable model training and improves the model's generalization ability, and adds an additional activation function to the structure to capture more complex features and patterns, improve gradient flow, effectively handle the complexity of pedestrian motion, and capture time dependence and multimodal interaction, thus better handling the complexity of pedestrian motion.

[0070] When predicting pedestrian intentions, insufficient understanding of dynamic relationships can be problematic. Combining vehicle speed and pedestrian historical bounding box coordinates can help understand the relative motion between pedestrians and vehicles. However, if this fusion is inadequate, the model may fail to effectively capture the impact of vehicle speed on pedestrian behavior, such as the effect of vehicle acceleration or deceleration on pedestrian decisions (e.g., whether to cross the road). Therefore, the model cannot accurately understand the dynamic behavior of pedestrians relative to vehicles, leading to reduced accuracy in behavior prediction. Furthermore, separating spatial and temporal information—pedestrian bounding box coordinates reflecting the pedestrian's spatial trajectory, while vehicle speed reflects the motion state of vehicles in the surrounding environment—can cause the model to process these two pieces of information separately if modal fusion is insufficient. This lack of effective temporal correlation prevents the model from fully understanding the interaction between pedestrians and vehicles. This can result in the model's poor performance in judging pedestrian intentions in complex traffic environments. Additionally, failing to accurately capture interaction timing is another issue. Vehicle speed changes (e.g., deceleration or sudden acceleration) can be a crucial factor in a pedestrian's decision to cross, while the pedestrian's historical motion (reflected in the bounding box coordinates) also determines their current state. If these two modal information cannot be effectively integrated, the model may not be able to accurately grasp the timing of interactions between pedestrians and vehicles, resulting in an inability to accurately determine when pedestrians are likely to change their behavior when predicting pedestrian intentions.

[0071] To fully understand observed content and capture the dependency between current observation and future actions, this invention utilizes cross-modal attention interaction and efficient modal feature fusion. It leverages the complementarity and differences between different modalities, designing an encoder-decoder based on the Transformer Encoder-Decoder (Transformer model). Based on the Transformer model's powerful long-term series modeling capabilities, the fused modal features are encoded and decoded, and the output pedestrian crossing intention score determines whether a pedestrian crossed at the predicted time point. This addresses the problems of existing prediction methods that do not consider the efficiency of modal feature fusion and rely on a single convolutional neural network or recurrent neural network for encoding, thus improving prediction performance.

[0072] Please see Figure 1 This invention provides a method for predicting pedestrian crossing intentions based on cross-modal attention interaction. It involves inputting a series of non-visual information, converting this information into features adapted to the encoder input, and then using cross-modal attention interaction to fuse these features. Specifically, it includes:

[0073] Step 1: Obtain vehicle speed information and human bounding box coordinate information, and obtain vehicle speed modal features and human bounding box coordinate modal features through the cross-modal attention interaction module.

[0074] Specifically, step one of the embodiments of the present invention includes:

[0075] Step 11: Obtain the vehicle's speed information and the human bounding box coordinates of the pedestrian in the vehicle's view.

[0076] More specifically, the acquisition of vehicle speed information and pedestrian bounding box coordinates from the vehicle's perspective, as described in the embodiments of the present invention, includes:

[0077] The vehicle speed information is obtained based on the vehicle's own speed, including:

[0078] Formula (1)

[0079] In formula (1), This indicates the vehicle's speed information. Indicates pedestrian, Represents the set of real numbers; The number of frames in a series of time frames. The feature dimension representing the vehicle's speed.

[0080] More specifically, in the embodiments of the present invention, the human body bounding box coordinate information is obtained based on the position of the pedestrian's human body bounding box in the perspective of the vehicle, wherein:

[0081] Formula (2)

[0082] In formula (2), This represents the coordinate information of the human body bounding box. Indicates pedestrian, pedestrian The bounding box coordinates, The feature dimension representing the bounding box coordinates; where: pedestrian exist Bounding box coordinates at time Represented as: , In the formula, This represents the x-coordinate of the top-left corner of the bounding box. The y-coordinate represents the top-left corner of the bounding box. This represents the x-coordinate of the bottom right corner of the bounding box. This represents the y-coordinate of the bottom right corner of the bounding box. Represents the set of real numbers; The number of frames in a series of time frames. The feature dimension representing the bounding box coordinates.

[0083] In practical applications, embodiments of the present invention utilize pedestrians The original frame sequence The information contained in various modalities is encoded into features suitable for encoder input, where... The information in the various modalities of this invention represents the attributes of the currently observed traffic scene, particularly the pedestrians and the vehicle itself, including pedestrians. Human body bounding box feature information and pedestrians Vehicle speed characteristics from the user's own perspective within the given scene.

[0084] Step 12: The processing unit processes the vehicle speed information and human bounding box coordinate information into vehicle speed data and human bounding box data.

[0085] More specifically, the acquisition of vehicle speed information and human body bounding box coordinate information from the vehicle's perspective, as described in the embodiments of the present invention, includes: the vehicle speed information and human body bounding box coordinate information being processed by a processing unit into vehicle speed data and human body bounding box data, wherein:

[0086] Vehicle speed information and bounding box location information The feature dimensions are mapped to the feature dimensions of the fused features through the processing unit. Obtain vehicle speed data and human body bounding box data ;in:

[0087] , ;

[0088] The mapping process includes: , ;

[0089] Mapped vehicle speed data , represented as Human body bounding box data , represented as In time The vehicle speed provides information about the vehicle's movement at different points in time, which is crucial for predicting pedestrian behavior and decision-making.

[0090] Vehicle speed information from the user's perspective is one of the main factors influencing pedestrian crossing decisions. The vehicle speed can be directly read from the vehicle system. Since the dataset contains labeled information about the vehicle speed, this embodiment of the invention directly uses the vehicle speed read from the vehicle system for encoding.

[0091] Step 13: The cross-modal attention interaction module performs cross-modal attention interaction on the vehicle speed data and the human bounding box data to obtain the vehicle speed modal features and the human bounding box coordinate modal features.

[0092] More specifically, in this embodiment of the invention, the vehicle speed data and human bounding box data undergo cross-modal attention interaction through two attention units of the cross-modal attention interaction module, enabling each modality to dynamically pay attention to relevant information in the other modality while maintaining its own feature structure. The specific interaction mechanism is as follows:

[0093] When the vehicle speed modality is used as a query in the attention mechanism, its query vector is similar to the key vector of the human bounding box modality to locate spatial position change patterns in the bounding box sequence related to the current vehicle motion state. When the human bounding box modality is used as a query, its query vector interacts with the key vector of the vehicle speed modality to capture the potential impact of vehicle motion on pedestrian behavior. The expression for the cross-modal attention interaction module is:

[0094] Formula (3)

[0095] In formula (3), Attention mechanisms representing multiple modal values, Indicates based on the first Query matrix for various modalities Indicates the first Key data for each modality Indicates the first Value data for various modalities, among which, , ,and ; and This indicates two heads of attention. This indicates a serial operation. express Attention and The chain operation of attention heads, This represents the output weight matrix. Indicates the first Pay attention to the head. This represents the attention mechanism. Represents the linear transformation matrix of the query matrix. The linear transformation matrix representing the key data. The linear transformation matrix representing the value data. Represents the query matrix. This represents the transpose of the key matrix. Represents a value matrix, This represents a scaling factor used to avoid excessively large dot product values ​​that could lead to vanishing or exploding gradients. After the aforementioned cross-modal attention interaction processing, each modality, when updating its own feature representation, fuses relevant information from the other modality. The two modal features output after cross-modal interaction remain consistent in the channel dimension as before the interaction, but semantically they have mutually enhanced each other. (Vehicle speed modal features) Represented as Human bounding box coordinate modal features Represented as .

[0096] Step 2: The vehicle velocity modal features and the human bounding box coordinate modal features are fused together using the feature fusion module to obtain fused features.

[0097] To effectively integrate the two modal features mentioned above, please refer to [link / reference]. Figure 3 This invention uses a "learnable weighted sum" method to integrate different modal information through a feature fusion module. This method can better combine information from different modalities, improving the model's ability to predict future actions. The feature fusion principle of this method lies in: adaptively learning the contribution weights of different modal features to the final prediction task, achieving dynamic weighted integration of multimodal information, rather than simple concatenation or average fusion. Specifically, step two of this embodiment includes:

[0098] The vehicle speed modal features and the human bounding box coordinate modal features are multiplied by their respective weights and summed by the feature fusion module to complete feature fusion. The feature fusion module outputs the fused features.

[0099] More specifically, in the embodiments of the present invention, the feature fusion module is used to fuse the vehicle speed modal features and the human bounding box coordinate modal features to output the fused features of the pedestrian's crossing intention. ,include:

[0100] Using weighted sum method Fusion of vehicle speed modal features Human bounding box coordinate modal features The process is as follows:

[0101] Formula (4)

[0102] In formula (4), Represents the set of real numbers. The number of frames in a series of time frames. The feature dimension representing the fused features.

[0103] The specific operation involves using the learnable first weight parameter. Second weight parameter Weighted summation of modal features after cross-modal interaction:

[0104]

[0105] Among them, the first weight parameter Second weight parameter These are parameters learned during training, used to adjust the weights of different modal features, resulting in the fused features. , represented as: , The feature dimension represents the fused feature. This weighting mechanism allows the fused feature to retain the modal semantics enhanced by cross-modal interaction, while also achieving a dynamic balance of modal contributions through weight optimization.

[0106] Step 3: Process the fused features using the Transformer model to obtain the pedestrian crossing intention score.

[0107] Since the Transformer model is insensitive to sequences, and the fused features are ordered, this embodiment of the invention adds positional encoding to the feature sequence. To handle variable-length input sequences, this embodiment employs two-dimensional fixed-positional encoding, as shown below:

[0108]

[0109]

[0110] In the formula, Indicates the sequence position. This represents the dimensional position of the positional encoding vector. The process of adding positional encoding is shown in the following equation:

[0111]

[0112] After adding two-dimensional fixed-position encoding, the fused sequence features Incoming Each Transformer block consists of a multi-head self-attention mechanism, residual connections with layer normalization, and a feed-forward neural network. The multi-head self-attention mechanism introduces learning parameters, allowing the Transformer block to continuously optimize during training, thus enabling different attention heads to learn different patterns. The propagation process for each Transformer block is as follows:

[0113]

[0114]

[0115] The Transformer block does not change the dimensions of the input feature sequence, meaning the output and input sequences maintain the same shape, as shown below: .

[0116] Encoding features , , This represents the feature dimension after encoding by the Transformer output. To obtain the aggregated features of the encoded feature sequence, the average value of the Transformer output is taken as the global feature. .

[0117] Specifically, step three of this embodiment of the invention includes:

[0118] Step 31: Use the encoder of the Transformer model to output global features for the fused features.

[0119] More specifically, the encoder outputting global features using the Transformer model for the fused features in the embodiments of the present invention includes:

[0120] Features of pedestrian crossing intentions encoded using the Transformer model The formula is ; Encoded features after encoding ,in, , Represents the feature dimension after encoding; selects the encoded features after encoding. The average value is used as a global feature. .

[0121] The principle behind using average pooling to obtain global features is that by aggregating the information from the entire time series, local temporal features are transformed into representations with global semantics. This design mitigates the information loss that may result from focusing only on specific time points, ensuring that the model can comprehensively consider the dynamic changes in pedestrian intentions throughout the entire observation sequence.

[0122] Step 32: After the global features are processed by the Transformer model, they are decoded by its decoder and the decoding result is output. The decoding result is the predicted target feature.

[0123] In the decoder stage based on the Transformer model, a set of learnable queries is set to decode the decoded results obtained from the encoder output. These query vectors gradually learn during training how to extract key information related to pedestrian crossing intentions from global features. (Query matrix) Represented as:

[0124] Formula (5)

[0125] In formula (5), Represents the query matrix in which the first... A query vector, Represents the set of real numbers. The number of frames in a series of time frames. This represents a set of learnable queries.

[0126] More specifically, in this embodiment of the invention, the global features described above are processed by the Transformer model, decoded by its decoder, and the decoding result is output as the predicted target features, including:

[0127] Using the Transformer model, global features are analyzed through formula (5). Decode it.

[0128] Each Transformer module in the decoder includes a masked multi-head self-attention mechanism, a multi-head self-attention mechanism, a residual with a normalization layer, and a feedforward neural network. The propagation process of each Transformer module in the decoder is represented as follows:

[0129] Formula (6)

[0130] In formula (6), Indicates the first The layer self-attention module updates the query matrix after residual connections and layer normalization. Indicates the normalization layer. This indicates a multi-head self-attention mechanism for the mask. This indicates a multi-head self-attention mechanism. Indicates the first The query matrix of the layer, Indicates the first Multi-head attention module usage and global features The calculated output, This indicates the decoding result. This represents a feedforward neural network.

[0131] The Transformer decoder employs a multi-stage attention mechanism. First, masked self-attention enables each query vector to focus on itself and its previous query states, ensuring causality in the decoding process. Then, an attention mechanism allows the query vector to focus on the semantic information most relevant to the pedestrian's crossing intention within the global features. This design principle allows the decoder to dynamically extract task-specific information from the encoded features while maintaining temporal dependencies.

[0132] In the multi-head self-attention mechanism of the Transformer decoder, the query learned by the self-attention mechanism serves as the query for the multi-head self-attention mechanism, and the global features output by the Transformer encoder are used as the query. This serves as both key and value data in the multi-head self-attention mechanism of the decoder. The decoded result after decoding. Consistent with the dimensions of the query, represented as .

[0133] Step 33: Input the predicted target features into the classifier to obtain the pedestrian crossing intention score.

[0134] In this embodiment of the invention, the decoded result is fed into the classifier (a fully connected layer with a sigmoid activation function) to obtain the pedestrian crossing intention score at the target time, predicting whether the pedestrian will cross at the predicted time point. More specifically, in this embodiment of the invention, the above-mentioned inputting the predicted target features into the classifier to obtain the pedestrian crossing intention score includes: feeding the predicted target features corresponding to the decoded result into the classifier to obtain the pedestrian crossing intention score at the target time, expressed as:

[0135] Formula (7)

[0136] In formula (7), The score represents the pedestrian's crossing intention at the target time. Represents a classifier. This indicates the decoding result. This represents the weight matrix of the fully connected layer. This represents the bias vector. This represents the Sigmoid activation function.

[0137] This step employs an encoder-decoder architecture. The encoder extracts rich temporal semantic information from the multimodal fusion features and obtains a global representation through average pooling. The decoder dynamically extracts the most relevant information for predicting pedestrian crossing intentions from the global features using a learnable query vector. This design achieves a complete information processing flow from local temporal features to global semantic understanding and then to task-specific decoding. It retains the advantages of the Transformer model in processing temporal data while ensuring effective prediction through the designed decoding mechanism.

[0138] Step 4: Predict pedestrian crossing intentions based on pedestrian crossing intention scores.

[0139] Specifically, step four includes: judging whether a pedestrian will cross at the predicted time point based on prediction data, including pedestrian crossing intention scores, and obtaining a judgment result; predicting the pedestrian's crossing intention based on the judgment result. Preferably, when the Transformer model is a Transformer model that has completed training, the pedestrian crossing intention score output by the trained Transformer model is used to judge whether a pedestrian will cross at the predicted time point, and a judgment result is obtained, and then the pedestrian's crossing intention is predicted based on the judgment result. When the Transformer model is a Transformer model that has already been trained, the pedestrian crossing intention score output by the trained Transformer model is used, along with prediction data composed of the current driving data of the surrounding vehicles obtained by the vehicle, to judge whether a pedestrian will cross at the predicted time point, and a judgment result is obtained, and then the pedestrian's crossing intention is predicted based on the judgment result.

[0140] This invention first constructs and trains a pedestrian crossing intention prediction network; then, it inputs a series of vehicle speed information and the human bounding box coordinates of pedestrians from the vehicle's perspective; finally, it uses the pedestrian crossing intention prediction network to predict pedestrian crossing intentions. Please refer to... Figure 2 The pedestrian crossing intention prediction network of this invention predicts pedestrian crossing intention through the following process: (1) Input the human body bounding box coordinate information and vehicle speed information of the pedestrian; (2) Input the human body bounding box coordinate information and vehicle speed information after position encoding into the multi-head self-attention module (MH Self-Attention) to perform cross-modal attention interaction and capture the correlation between features of different modalities; (3) The features after cross-modal attention interaction are fused through the feature fusion module; (4) The fused features are input into the encoder of the Transformer model; (5) Take the average value of the output features of the Transformer model as the global feature. (6) Set up a set of learnable queries to decode the global features output by the encoder; (7) Input the decoded result O into the classifier to obtain the pedestrian crossing intention score at the target time to determine whether the pedestrian will cross.

[0141] In practical applications, the above decoding results The data is fed into a classifier, whose role is to convert the decoder's output into a pedestrian crossing intention score at the target time. The classifier typically consists of one or more fully connected layers used to map high-dimensional features to the final pedestrian crossing intention. Preferably:

[0142] The present invention predicts the travel intentions of pedestrians, and the specific operation is as follows:

[0143] First, flatten the decoded result O into a one-dimensional vector using the Flatten function. The shape is After being flattened, it becomes ( The flattened vector is fed into a fully connected layer, which maps the high-dimensional features to a smaller dimension, namely the classification score output dimension, which is 1. The output of the fully connected layer is then activated by a sigmoid function, converting it into a probability representing the pedestrian's intention to cross. A threshold for the pedestrian's intention to cross is set; for example, a preferred threshold is 0.5. If the pedestrian's intention to cross score is greater than 0.5, the judgment is that the pedestrian intends to cross; if the score is less than 0.5, the judgment is that the pedestrian does not intend to cross.

[0144] This invention maps the velocity features of the vehicle and the bounding box features of the pedestrian in a visual scene to the same channel dimension, and then uses a multi-head attention unit to perform cross-modal attention interaction on these two modal features. The feature fusion module of this invention efficiently fuses the interacting modal features to obtain spatial information between different modalities. Then, it encodes the spatial information from the feature dimension, and reassembles the vehicle velocity features and pedestrian bounding box features after cross-modal attention interaction through the feature fusion module to fully utilize the complementarity between different modalities. The encoder module of this invention utilizes the powerful temporal modeling capabilities of the Transformer as an encoder to fully understand the observed content and capture the dependency between the current observation and future actions. The decoder module of this invention sets a set of learnable queries, uses the Transformer to use the aggregated output features of the encoder as keys and values, decodes using the learnable queries, and uses the decoder output as the predicted target feature. This result is input into a classifier to obtain a pedestrian crossing intention score to predict whether a pedestrian will cross at the predicted time point.

[0145] This invention proposes cross-modal interaction and feature fusion between modal information for multimodal feature fusion. Cross-modal interaction enables attention interaction between vehicle speed information and pedestrian bounding box coordinate information. The vehicle speed modal features and human bounding box coordinate modal features obtained after attention interaction are multiplied by their respective weights and summed through a feature fusion module to complete feature fusion, fully utilizing the complementarity and differences of different modal features and strengthening the interaction between modalities. In the encoding and decoding process of the fused features, this invention proposes an action prediction framework based on the Transformer model. In the encoder stage, the Transformer model is used to encode the fused modal features. In the decoder stage, a set of learnable queries is set. The global features of the encoder output are used as keys and values ​​by the Transformer, and the learnable queries are used for decoding. The decoder output is used as the prediction target feature. The prediction target feature is input into a classifier to obtain a pedestrian crossing intention score. Based on the pedestrian crossing intention score or the pedestrian crossing intention score and the driving data of the surrounding vehicles at the current time, it is determined whether the pedestrian will cross at the predicted time node.

[0146] This invention utilizes cross-modal interaction to achieve attentional interaction between various modal information. Through a feature fusion module, it fully leverages the differences between different modal information and effectively fuses these differences, which is highly beneficial for subsequent model learning. Furthermore, it helps the encoder better understand and utilize the correlation between temporal and spatial contexts, improving the model's ability to model pedestrian crossing intention modal features. This invention constructs a Transformer-based encoder-decoder branch, capturing long-term dependencies between different modalities through a self-attention mechanism and generating fused feature representations; it can effectively capture global contextual information and enhance the understanding of categorical features.

[0147] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the scope of the technology disclosed in the present invention, and such modifications or substitutions should all be covered within the scope of protection of the present invention.

Claims

1. A method for predicting pedestrian crossing intentions based on cross-modal attention interaction, characterized in that, include: Step 1: Obtain vehicle speed information and human bounding box coordinate information, and obtain vehicle speed modal features and human bounding box coordinate modal features through the cross-modal attention interaction module; Step 2: The vehicle velocity mode features and the human bounding box coordinate mode features obtained in Step 1 are fused together using a feature fusion module to obtain fused features; Step 3: Process the fused features obtained in Step 2 using the Transformer model to obtain pedestrian crossing intention scores; Step 4: Predict pedestrian crossing intentions based on the pedestrian crossing intention scores.

2. The method for predicting pedestrian crossing intentions based on cross-modal attention interaction according to claim 1, characterized in that, Step one includes: Acquire vehicle speed information and pedestrian bounding box coordinates from the vehicle's perspective; The vehicle speed information and the human bounding box coordinate information are processed into vehicle speed data and human bounding box data; The cross-modal attention interaction module performs cross-modal attention interaction on the vehicle speed data and the human bounding box data to obtain vehicle speed modal features and human bounding box coordinate modal features.

3. The method for predicting pedestrian crossing intentions based on cross-modal attention interaction according to claim 2, characterized in that, In step one, the vehicle speed information is obtained based on the vehicle speed, and the expression is: Official (1) In formula (1), This indicates the vehicle's speed information. Indicates pedestrian, Represents the set of real numbers. The number of frames in a series of time frames. Feature dimensions representing vehicle speed; In step one, the human body bounding box coordinate information is obtained based on the position of the pedestrian's human body bounding box in the vehicle's view, and the expression is: Official (2) In formula (2), This represents the coordinate information of the human body bounding box. pedestrian The bounding box coordinates, The feature dimension representing the bounding box coordinates; where, pedestrian exist Bounding box coordinates at time Represented as: , In the formula, This represents the x-coordinate of the top-left corner of the bounding box. The y-coordinate represents the top-left corner of the bounding box. This represents the x-coordinate of the bottom right corner of the bounding box. This represents the ordinate of the bottom right corner of the bounding box.

4. The method for predicting pedestrian crossing intentions based on cross-modal attention interaction according to claim 3, characterized in that, The process of processing the vehicle speed information and the human bounding box coordinate information into vehicle speed data and human bounding box data in step one includes: Vehicle speed information Human bounding box coordinate information Feature dimensions mapped to fused feature dimensions Obtain vehicle speed data and human body bounding box data ; The cross-modal attention interaction module in step one is represented as follows: Official (3) In formula (3), Attention mechanisms representing multiple modal values, Indicates based on the first Query matrix for various modalities Indicates the first Key data for each modality Indicates the first Value data for each modality; express Attention and The chain operation of attention heads, This represents the output weight matrix. Indicates the first Pay attention to the head. This represents the attention mechanism. Represents the linear transformation matrix of the query matrix. The linear transformation matrix representing the key data. The linear transformation matrix representing the value data. Represents the query matrix. This represents the transpose of the key matrix. Represents a value matrix, Indicates the scaling factor; After processing with formula (3), the vehicle speed modal characteristics are obtained. Human bounding box coordinate modal features .

5. The method for predicting pedestrian crossing intentions based on cross-modal attention interaction according to claim 1, characterized in that, Step two includes: The vehicle speed modal features and the human bounding box coordinate modal features obtained in step one are multiplied by their respective weights and summed using the feature fusion module to complete feature fusion, outputting the fused feature of the pedestrian's crossing intention. Feature fusion includes: using feature fusion methods. Fusion of vehicle speed modal features Human bounding box coordinate modal features The process is as follows: Official (4) In formula (4), Represents the set of real numbers. The number of frames in a series of time frames. The feature dimension representing the fused features; Fusion features after feature fusion , represented as: In the formula, Indicates pedestrian, Represents the set of real numbers. The number of frames in a series of time frames. The feature dimension representing the fused features.

6. The method for predicting pedestrian crossing intentions based on cross-modal attention interaction according to claim 1, characterized in that, Step three includes: The encoder of the Transformer model is used to output global features for the fused features; The global features are processed by the Transformer model and then decoded by its decoder to output the decoding result, which is the predicted target feature. The predicted target features are input into a classifier to obtain a pedestrian crossing intention score.

7. The method for predicting pedestrian crossing intentions based on cross-modal attention interaction according to claim 6, characterized in that, The encoder of the Transformer model outputs global features for the fused features, including: Fusion features of pedestrian crossing intentions after fusion using the Transformer model Obtain coding features ; Encoded features after encoding Represented as: In the formula, This represents the feature dimension after encoding. , Represents the set of real numbers. Represents the number of frames in a series of time frames, selecting the encoded features. The average value is used as a global feature. ; The global features are processed by the Transformer model, decoded by its decoder, and the decoded result is output. The decoded result is the predicted target features, including: Using the Transformer model, global features are analyzed through formula (5). Decode it as follows: Official (5) In formula (5), Represents the query matrix. Represents the query matrix in which the first... A query vector, Represents the set of real numbers. The number of frames in a series of time frames. Represents a set of dimensions that can be learned and queried; The propagation process of each Transformer module in the decoder is as follows: Official (6) In formula (6), Indicates the first The layer self-attention module updates the query matrix after residual connections and layer normalization. Indicates the normalization layer. This indicates a multi-head self-attention mechanism for the mask. This indicates a multi-head self-attention mechanism. Indicates the first The query matrix of the layer, Indicates the first Multi-head attention module usage and global features The calculated output, This indicates the decoding result. This represents a feedforward neural network; The predicted target features are input into a classifier to obtain a pedestrian crossing intention score, including: feeding the predicted target features corresponding to the decoding result into the classifier to obtain the pedestrian crossing intention score at the target time, expressed as: Official (7) In formula (7), The score represents the pedestrian's crossing intention at the target time. Represents a classifier. This indicates the decoding result. Indicates the flattening operation. This represents the weight matrix of the fully connected layer. This represents the bias vector.

8. The method for predicting pedestrian crossing intentions based on cross-modal attention interaction according to claim 1, characterized in that, Step four includes: Based on the predicted data, including the pedestrian crossing intention score, it is determined whether a pedestrian will cross at the predicted time point, and the judgment result is obtained. Predict the travel intentions of pedestrians based on the judgment results.

9. A method for predicting pedestrian crossing intentions based on cross-modal attention interaction according to claim 1 or 8, characterized in that, The aforementioned prediction data based on at least the pedestrian crossing intention scores includes: The pedestrian crossing intention score is based on the output of the trained Transformer model. or, The predicted data is composed of the pedestrian crossing intention score output by the trained Transformer model and the driving data of the surrounding vehicles at the current moment obtained by the vehicle.

Images