Pedestrian crossing prediction method and system based on multi-modal feature fusion

Abstract

The invention relates to the technical field of behavior prediction, and discloses a pedestrian crossing prediction method and system based on multi-modal feature fusion, and the method comprises the steps: obtaining multi-modal input data in a current observation time window; obtaining a currently observed main modal time sequence feature based on the multi-modal input data, and constructing a condition vector; extracting future observation features of the main mode; inputting the condition vector and the future observation features of the main mode into a teacher network to obtain first potential distribution; inputting the condition vector and the prior potential vector into a student network, enabling second potential distribution output by the student network to approach the first potential distribution through KL divergence alignment loss, and refining the prior potential vector to obtain a refined potential vector; inputting the condition vector and the refined potential vector into a decoder, and outputting a pedestrian crossing prediction probability and a corresponding uncertainty measure; according to the invention, the problems of low prediction precision and poor robustness of an existing pedestrian crossing prediction mode are solved.

Description

Technical Field

[0001] This invention relates to the field of behavior prediction technology, and in particular to a pedestrian crossing prediction method and system based on multimodal feature fusion. Background Technology

[0002] Pedestrian crossing prediction is a key technology in Autonomous Driving Systems (ADS) and Advanced Driver Assistance Systems (ADAS). Its goal is to predict in advance whether a pedestrian is about to cross the lane based on historical observation information collected by vehicle and environmental sensors, enabling the vehicle to make safe and smooth decisions. Existing methods typically input raw RGB video, semantic segmentation results, detection box sequences, pose keypoints, and vehicle speed into deep networks such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Graph Neural Networks (GNN), or Transformers, outputting a binary classification result of "crossing / not crossing". Some works introduce multi-head attention and cross-modal attention to model the relationships between different modalities and improve prediction performance through structures such as cross-modal Transformers, scene graphs, and spatiotemporal graph attention networks. Most existing methods use a fusion approach of "independent encoding of each modality + feature concatenation + fully connected classifier," sometimes supplemented by simple attention weighting. These methods can utilize multimodal features to some extent, but they are mostly fixed structures, making it difficult to adaptively adjust the contribution of each modality to the final prediction according to the specific scene. Pedestrian behavior itself is highly uncertain, and simple discriminative networks are prone to making overconfident and erroneous predictions in ambiguous scenarios. Some works introduce methods such as Evidential Deep Learning and Bayesian learning to characterize prediction uncertainty from the output layer; others use Conditional Variational Autoencoders (CVAEs) to model multi-possibility trajectories in trajectory prediction tasks.

[0003] However, in the specific problem of "pedestrian crossing prediction," existing works often only use CVAE for pedestrian trajectory generation instead of directly for multimodal feature fusion; or they only introduce uncertainty indicators at the prediction result level without fully utilizing uncertainty in the latent feature space, lacking a mechanism to "use future observation information to guide the current multimodal fusion method." Furthermore, to achieve higher accuracy, many methods rely on complete RGB video frames and frame-by-frame semantic segmentation maps as input, employing complex visual backbone networks, resulting in large model parameter scale, high computational overhead, and difficulty in real-time deployment on in-vehicle terminals. Recent research shows that using only "lightweight modalities" such as detection box sequences, pose keypoints, and vehicle speed can achieve performance close to methods using images and segmentation maps. However, existing lightweight methods still suffer from high misclassification rates and poor prediction stability in high-uncertainty scenarios such as stationary pedestrians, complex intersections, and high-speed driving.

[0004] In summary, existing technologies mainly suffer from the following problems: 1) Multimodal feature fusion structures are mostly static and fixed, failing to utilize "future information" to constrain the current fusion method and lacking adaptive fusion mechanisms for different scenarios; 2) Although some future observation information can be obtained during the training phase, it is mostly used only as labels or simple features, without constructing a systematic guidance of future information for current feature fusion at the "latent distribution level," resulting in significant differences in distribution between the training and inference phases; 3) Existing uncertainty modeling mostly stays at the output confidence level, without feeding uncertainty back into the design of the latent spatial structure and feature fusion strategy, and is still not robust enough in handling high-uncertainty scenarios; 4) Many high-performance methods rely on high-resolution images, semantic segmentation, and massive network structures, making it difficult to balance prediction accuracy, model complexity, and vehicle real-time performance. Summary of the Invention

[0005] This invention provides a pedestrian crossing prediction method and system based on multimodal feature fusion to solve the problems of low prediction accuracy and poor robustness in existing pedestrian crossing prediction methods.

[0006] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, the present invention provides a pedestrian crossing prediction method based on multimodal feature fusion, comprising: S1. Obtain multimodal input data within the current observation time window, wherein the multimodal input data includes pedestrian detection box sequence, posture key point sequence, vehicle speed information and static environment information; S2. Preprocess and extract features from the multimodal input data to obtain the temporal features of the current observation; S3. Based on the temporal features of the current observation, the posture features obtained by encoding the posture key point sequence, the static environment features obtained by encoding the static environment information, and the dynamic intent features estimated based on the temporal features of the current observation and the posture features, a unified conditional vector is constructed. S4. During the training phase, acquire future observation multimodal data for the corresponding time window and extract future observation features; S5. Construct an autoencoder model that includes a teacher network, a student network, and a decoder. Input the conditional vector and the future observation features into the teacher network to obtain the first latent distribution defined by the teacher network. S6. Construct a learnable Gaussian mixture model prior in the latent space, and sample from the learnable Gaussian mixture model prior to obtain the prior latent vector; S7. Input the conditional vector and the prior latent vector into the student network, minimize the distribution alignment loss to make the second latent distribution output by the student network approximate the first latent distribution, and sample the refined latent vector from the second latent distribution; S8. Input the conditional vector and the refined latent vector into the decoder, and output the predicted probability of pedestrian crossing and the corresponding uncertainty measure. The teacher network, the student network, and the decoder together constitute a conditional variational autoencoder structure. By guiding the second latent distribution through the first latent distribution, the future information in the latent space adjusts the current multimodal feature fusion strategy.

[0007] Optionally, the feature extraction described in step S2 is achieved by enhancing the Transformer backbone network; The enhanced Transformer backbone network introduces an adaptive position coding module, an expert hybrid layer, and an information bottleneck compression network on the basis of the standard Transformer encoder. The adaptive position coding module generates an adaptive position code that can assign different weights to different time positions by combining sinusoidal position coding, learnable position coding and attention gating mechanism. The expert hybrid layer comprises multiple parallel expert subnetworks and a routing network, wherein the routing network dynamically selects and activates the top-k expert subnetworks based on input features. The information bottleneck compression network is used to compress the output features of the Transformer into a low-dimensional space, and to ensure that key discriminative information is not lost by reconstructing the loss constraints.

[0008] Optionally, the adaptive position encoding satisfies the following relationship: ; In the formula, This represents a two-layer network containing LayerNorm, GELU activation function, and dropout. This represents position attention gating implemented using a two-layer feedforward network. This represents the scaled sine wave position code. This indicates a learnable positional embedding. This indicates adaptive positional encoding.

[0009] Optionally, constructing the condition vector in step S3 specifically includes: Extract the current observation time series features of the dominant mode as the dominant mode time features; Pose features are obtained by using a lightweight multilayer perceptron and residual block encoding to encode the pose key point sequence in the auxiliary mode; Static environment features are obtained by encoding static environment information and initial detection box positions in the auxiliary mode using a multilayer perceptron. By using the dynamic intent estimation branch, a continuous intent score that changes over time is output based on the main modality time features and the posture features, which serves as the dynamic intent feature; The conditional vector is obtained by concatenating the main modality time features, posture features, static environment features, and dynamic intent features.

[0010] Optionally, the Gaussian mixture model prior satisfies the following relationship: ; In the formula, Indicates the number of Gaussian components. Indicates mixed weights, This represents the learnable parameters.

[0011] Optionally, the training process of the method employs a multi-task joint loss function, and the total loss satisfies the following relationship: ; In the formula, Indicates the main predicted loss. Represents the distribution alignment loss. Indicates the prior regularization loss. Indicates dynamic intention loss. This represents the reconstruction loss.

[0012] Optionally, the training of the method employs a staged strategy: In the first stage, based on the current observation data and future observation data of the main modality, a first feature extraction network for extracting current time-series features and a second feature extraction network for extracting future observation features are independently pre-trained. In the second stage, the parameters of the first feature extraction network and the second feature extraction network are frozen, and the conditional variational autoencoder structure, the multimodal feature encoder and the dynamic intent estimation branch are jointly trained.

[0013] Optionally, a potential hybrid strategy can be employed in the second phase of training: The refined latent vectors used by the decoder are replaced with samples sampled from the first latent distribution of the teacher network with a preset probability.

[0014] In a second aspect, embodiments of this application provide a pedestrian crossing prediction system based on multimodal feature fusion, used to implement the method described in any one of the first aspects, the system comprising: The data acquisition and preprocessing module is used to acquire and preprocess multimodal input data within the current observation time window; The feature extraction module, which includes an enhanced Transformer backbone network, is used to extract the temporal features of the current observation and the features of future observations from the preprocessed data. The multimodal coding and condition construction module is used to construct a condition vector based on the temporal features, attitude features, static environment features, and dynamic intention features of the current observation; The future-oriented integrated generation module includes a conditional variational autoencoder with a teacher-student structure; The teacher network is used to receive the conditional vector and future observation features, and output the first latent distribution; The student network is used to receive the conditional vector and a prior latent vector sampled from the Gaussian mixture model prior, output a second latent distribution aligned with the first latent distribution, and generate a refined latent vector. The decoding and output module is used to receive the condition vector and the refined latent vector, decode and output the predicted probability and uncertainty measure of pedestrian crossing.

[0015] Optionally, the enhanced Transformer backbone network includes: An adaptive position coding unit is used to generate adaptive position codes for focusing on key time segments; The expert hybrid unit comprises multiple expert subnetworks dynamically selected by the routing network to achieve diverse feature representations; The information bottleneck compression unit is used to reduce the dimensionality of features while retaining key information.

[0016] Beneficial effects: This invention provides a pedestrian crossing prediction method based on multimodal feature fusion. Through teacher-student CVAE, GMM prior, and KL distribution alignment, this invention transforms available future observation information from the training phase into guiding targets in the latent space, rather than simply concatenating features. This allows the multimodal fusion strategy to adaptively adjust according to the prediction target. Experiments show that on standard datasets, this invention achieves approximately 93% (PIE) and 92% (JAAD) accuracy in short-term prediction tasks using only lightweight modalities, and a balanced accuracy of approximately 87% (PIE) and 91% (JAAD) in long-term prediction tasks. In typical high-uncertainty scenarios such as stationary pedestrians, high-speed vehicles, and complex environments, this invention significantly outperforms traditional methods through future guiding distribution alignment and uncertainty modeling. For example, it improves mAP by approximately 3.8% in the "standing pedestrian" scenario and by approximately 5.4% in the high-speed vehicle scenario, effectively reducing missed detections and false positives in critical hazardous scenarios. Relying solely on lightweight modalities such as bounding box sequences, pose keypoints, vehicle speed, and static environment, it eliminates the need for frame-by-frame image convolution and semantic segmentation, significantly reducing data acquisition and computational costs. A single inference time of approximately 8.99 ms meets real-time operational requirements on in-vehicle platforms. and The joint constraints of this invention mean that the student network relies primarily on prior sampling and current conditions for refinement during the training phase, with only a small amount of teacher latent vectors introduced, thus mitigating the distribution shift of "training with future information – inference without future information." In long-term prediction tasks, this invention significantly outperforms the comparison methods in terms of confidence decay index (confΔ), with the maximum / average confidence decrease controlled at approximately 0.05 / 0.02, indicating that the prediction changes more smoothly and reliably over time. The enhanced Transformer backbone, multimodal encoder, future guidance generation module, and uncertainty output module are decoupled, facilitating on-demand tailoring and expansion under different hardware resources and application scenarios. Furthermore, this framework can be naturally extended to other temporal tasks that require "using future information to guide current decisions," such as pedestrian trajectory prediction, vehicle lane-changing intent recognition, and multi-pedestrian interaction risk assessment. Attached Figure Description

[0017] Figure 1 This is a flowchart of a pedestrian crossing prediction method based on multimodal feature fusion, which is a preferred embodiment of the present invention. Detailed Implementation

[0018] The technical solution of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms "an" or "a" and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms "connected" or "linked" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up," "down," "left," "right," etc., are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship also changes accordingly.

[0020] Please see Figure 1 This application provides a pedestrian crossing prediction method based on multimodal feature fusion, including: S1. Obtain multimodal input data within the current observation time window, wherein the multimodal input data includes pedestrian detection box sequence, posture key point sequence, vehicle speed information and static environment information; S2. Preprocess and extract features from the multimodal input data to obtain the temporal features of the current observation; S3. Based on the temporal features of the current observation, the posture features obtained by encoding the posture key point sequence, the static environment features obtained by encoding the static environment information, and the dynamic intent features estimated based on the temporal features of the current observation and the posture features, a unified conditional vector is constructed. S4. During the training phase, acquire future observation multimodal data for the corresponding time window and extract future observation features; S5. Construct an autoencoder model that includes a teacher network, a student network, and a decoder. Input the conditional vector and the future observation features into the teacher network to obtain the first latent distribution defined by the teacher network. S6. Construct a learnable Gaussian mixture model prior in the latent space, and sample from the learnable Gaussian mixture model prior to obtain the prior latent vector; S7. Input the conditional vector and the prior latent vector into the student network, minimize the distribution alignment loss to make the second latent distribution output by the student network approximate the first latent distribution, and sample the refined latent vector from the second latent distribution; S8. Input the conditional vector and the refined latent vector into the decoder, and output the predicted probability of pedestrian crossing and the corresponding uncertainty measure. The teacher network, the student network, and the decoder together constitute a conditional variational autoencoder structure. By guiding the second latent distribution through the first latent distribution, the future information in the latent space adjusts the current multimodal feature fusion strategy.

[0021] Optionally, the feature extraction described in step S2 is achieved by enhancing the Transformer backbone network; The enhanced Transformer backbone network introduces an adaptive position coding module, an expert hybrid layer, and an information bottleneck compression network on the basis of the standard Transformer encoder. The adaptive position coding module generates an adaptive position code that can assign different weights to different time positions by combining sinusoidal position coding, learnable position coding and attention gating mechanism. The expert hybrid layer comprises multiple parallel expert subnetworks and a routing network, wherein the routing network dynamically selects and activates the top-k expert subnetworks based on input features. The information bottleneck compression network is used to compress the output features of the Transformer into a low-dimensional space, and to ensure that key discriminative information is not lost by reconstructing the loss constraints.

[0022] Optionally, the adaptive position encoding satisfies the following relationship: ; In the formula, This represents a two-layer network containing LayerNorm, GELU activation function, and dropout. This represents position attention gating implemented using a two-layer feedforward network. This represents the scaled sine wave position code. This indicates a learnable positional embedding. This indicates adaptive positional encoding.

[0023] Optionally, constructing the condition vector in step S3 specifically includes: Extract the current observation time series features of the dominant mode as the dominant mode time features; Pose features are obtained by using a lightweight multilayer perceptron and residual block encoding to encode the pose key point sequence in the auxiliary mode; Static environment features are obtained by encoding static environment information and initial detection box positions in the auxiliary mode using a multilayer perceptron. By using the dynamic intent estimation branch, a continuous intent score that changes over time is output based on the main modality time features and the posture features, which serves as the dynamic intent feature; The conditional vector is obtained by concatenating the main modality time features, posture features, static environment features, and dynamic intent features.

[0024] Optionally, the Gaussian mixture model prior satisfies the following relationship: ; In the formula, Indicates the number of Gaussian components. Indicates mixed weights, This represents the learnable parameters.

[0025] Optionally, the training process of the method employs a multi-task joint loss function, and the total loss satisfies the following relationship: ; In the formula, Indicates the main predicted loss. Represents the distribution alignment loss. Indicates the prior regularization loss. Indicates dynamic intention loss. This represents the reconstruction loss.

[0026] Optionally, the training of the method employs a staged strategy: In the first stage, based on the current observation data and future observation data of the main modality, a first feature extraction network for extracting current time-series features and a second feature extraction network for extracting future observation features are independently pre-trained. In the second stage, the parameters of the first feature extraction network and the second feature extraction network are frozen, and the conditional variational autoencoder structure, the multimodal feature encoder and the dynamic intent estimation branch are jointly trained.

[0027] Optionally, a potential hybrid strategy can be employed in the second phase of training: The refined latent vectors used by the decoder are replaced with samples sampled from the first latent distribution of the teacher network with a preset probability.

[0028] Example 1 This embodiment presents a training process for a pedestrian crossing prediction model based on multimodal feature fusion, which mainly includes the following steps: S1) Multimodal data acquisition and preprocessing Video footage and related information involving pedestrians are collected via forward-facing cameras, vehicle speed sensors, and roadside infrastructure. A mature pedestrian detection and 2D pose estimation network is used to obtain the bounding box sequence and pose keypoint sequence for each frame, while simultaneously recording vehicle speed and road environment information. The data is preprocessed, including coordinate normalization, differential transformation, missing value imputation, and environmental semantic encoding, ultimately yielding a multimodal time series suitable for deep network input.

[0029] S2) Enhanced Transformer Backbone (ETA) pre-training (Phase 0 / 0.5) Use the current observation window respectively and future observation window Under the constraint of a supervisory signal (e.g., short-term street crossing labels), t_model and tn_model are pre-trained to enable them to extract stable and discriminative temporal features from lightweight modalities, and the expressive power of compressed features is guaranteed by information bottleneck reconstruction loss.

[0030] S3) Multimodal feature encoding and conditional vector construction In Phase 1+, the parameters of t_model and tn_model are fixed. For each sample: The t_model is used to extract the temporal features of the pedestrian detection box difference sequence, vehicle speed, and pedestrian detection box center coordinates to obtain the main temporal features. ; The 2D pose keypoint sequence is encoded using a lightweight MLP + residual block network; MLP encoding is performed on static environment information and the initial position of pedestrian detection boxes; Branch pairs estimated by dynamic intent It provides assisted supervision based on pose features and outputs a dynamic intent score; Concatenate all the above features into a unified condition vector. .

[0031] S4) Future-guided fusion generation module training For each sample, perform the following steps: (1) Use tn_model to extract time features from future observations to obtain future vectors. , and condition vector Input the teacher network together, obtain the posterior distribution parameters and define them. ; (2) From GMM prior Mid-sampled latent vector ,and Input the student refine encoder together to obtain the refine distribution. ; (3) Based on the set potential mixing probability, part of The samples are replaced with the teacher's posterior distribution to obtain the final latent vector for the decoder; (4) Decoder reception Combined with latent vectors, it outputs street crossing predictions and uncertainty measures; (5) According to , , , and Construct the total loss and backpropagate and update the parameters of Future-GuidedCVAE, dynamic intent branches, and related modules.

[0032] Through the above training process, the model obtained in this embodiment can learn multimodal fusion behavior that is close to "future perception" under the premise of only accessing the current observation and prior distribution.

[0033] Example 2 This embodiment illustrates the specific application process of the method of the present invention in an autonomous vehicle, mainly including the following steps: T1) Online data acquisition and preprocessing During operation, the autonomous vehicle's forward-facing camera and speed sensor collect real-time video streams including pedestrians and vehicle status information, while roadside equipment provides environmental information such as traffic lights and intersection types. Online pedestrian detection and 2D pose estimation algorithms are used to obtain the bounding box and pose keypoint coordinates for each frame. Within a sliding time window, the data undergoes the same preprocessing and normalization operations as during the training phase.

[0034] T2) Real-time feature extraction and conditional vector generation The dominant modality sequence within the current time window is input into the frozen t_model to extract the dominant modality temporal features; then, a multimodal encoder is used to obtain pose features, static environment features, pedestrian detection box initial position features, and dynamic intent scores; finally, these are concatenated to obtain the conditional vector for the current time step. .

[0035] T3) Sample – Refined Reasoning and Uncertainty Output Sample latent vectors from GMM priors , and condition vector The student refined encoder is input together to obtain the aligned latent distribution and sampled. ;Will and Input the data into the decoder to obtain the predicted probability of whether a pedestrian will cross the street within a certain period of time in the future, as well as the corresponding uncertainty measure.

[0036] T4) Decision and Control Interface Based on the predicted probability and uncertainty indicators, the vehicle decision-making module can adopt different strategies, such as: When the predicted probability of crossing the street is high and the uncertainty is low, slow down or stop to yield in advance; When the predicted probability of "not crossing the street" is high but the uncertainty is large, a conservative strategy should be adopted, such as appropriately slowing down and strengthening the monitoring of pedestrian areas. When uncertainty is extremely high, the highest level of security control strategy is triggered.

[0037] Through the above steps, the present invention can provide autonomous vehicles with high-precision and high-stability pedestrian crossing prediction capabilities without adding additional high-cost sensors and complex vision processing modules.

[0038] This application also provides a pedestrian crossing prediction system based on multimodal feature fusion, used to implement the method described in any one of the first aspects, the system comprising: The data acquisition and preprocessing module is used to acquire and preprocess multimodal input data within the current observation time window; The feature extraction module, which includes an enhanced Transformer backbone network, is used to extract the temporal features of the current observation and the features of future observations from the preprocessed data. The multimodal coding and condition construction module is used to construct a condition vector based on the temporal features, attitude features, static environment features, and dynamic intention features of the current observation; The future-oriented integrated generation module includes a conditional variational autoencoder with a teacher-student structure; The teacher network is used to receive the conditional vector and future observation features, and output the first latent distribution; The student network is used to receive the conditional vector and a prior latent vector sampled from the Gaussian mixture model prior, output a second latent distribution aligned with the first latent distribution, and generate a refined latent vector. The decoding and output module is used to receive the condition vector and the refined latent vector, decode and output the predicted probability and uncertainty measure of pedestrian crossing.

[0039] Optionally, the enhanced Transformer backbone network includes: An adaptive position coding unit is used to generate adaptive position codes for focusing on key time segments; The expert hybrid unit comprises multiple expert subnetworks dynamically selected by the routing network to achieve diverse feature representations; The information bottleneck compression unit is used to reduce the dimensionality of features while retaining key information.

[0040] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A pedestrian crossing prediction method based on multimodal feature fusion, characterized in that, include: S1. Obtain multimodal input data within the current observation time window, wherein the multimodal input data includes pedestrian detection box sequence, posture key point sequence, vehicle speed information and static environment information; S2. Preprocess and extract features from the multimodal input data to obtain the temporal features of the current observation; S3. Based on the temporal features of the current observation, the posture features obtained by encoding the posture key point sequence, the static environment features obtained by encoding the static environment information, and the dynamic intent features estimated based on the temporal features of the current observation and the posture features, a unified conditional vector is constructed. S4. During the training phase, acquire future observation multimodal data for the corresponding time window and extract future observation features; S5. Construct an autoencoder model that includes a teacher network, a student network, and a decoder. Input the conditional vector and the future observation features into the teacher network to obtain the first latent distribution defined by the teacher network. S6. Construct a learnable Gaussian mixture model prior in the latent space, and sample from the learnable Gaussian mixture model prior to obtain the prior latent vector; S7. Input the conditional vector and the prior latent vector into the student network, minimize the distribution alignment loss to make the second latent distribution output by the student network approximate the first latent distribution, and sample the refined latent vector from the second latent distribution; S8. Input the conditional vector and the refined latent vector into the decoder, and output the predicted probability of pedestrian crossing and the corresponding uncertainty measure. The teacher network, the student network, and the decoder together constitute a conditional variational autoencoder structure. By guiding the second latent distribution through the first latent distribution, the future information in the latent space adjusts the current multimodal feature fusion strategy.

2. The pedestrian crossing prediction method based on multimodal feature fusion according to claim 1, characterized in that, The feature extraction described in step S2 is achieved by enhancing the Transformer backbone network; The enhanced Transformer backbone network introduces an adaptive position coding module, an expert hybrid layer, and an information bottleneck compression network on the basis of the standard Transformer encoder. The adaptive position coding module generates an adaptive position code that can assign different weights to different time positions by combining sinusoidal position coding, learnable position coding and attention gating mechanism. The expert hybrid layer comprises multiple parallel expert subnetworks and a routing network, wherein the routing network dynamically selects and activates the top-k expert subnetworks based on input features. The information bottleneck compression network is used to compress the output features of the Transformer into a low-dimensional space, and to ensure that key discriminative information is not lost by reconstructing the loss constraints.

3. The pedestrian crossing prediction method based on multimodal feature fusion according to claim 2, characterized in that, The adaptive position encoding satisfies the following relationship: ； In the formula, This represents a two-layer network containing LayerNorm, GELU activation function, and dropout. This represents position attention gating implemented using a two-layer feedforward network. This represents the scaled sine wave position code. This indicates a learnable positional embedding. This indicates adaptive positional encoding.

4. The pedestrian crossing prediction method based on multimodal feature fusion according to claim 1, characterized in that, The construction of the condition vector in step S3 specifically includes: Extract the current observation time series features of the dominant mode as the dominant mode time features; Pose features are obtained by using a lightweight multilayer perceptron and residual block encoding to encode the pose key point sequence in the auxiliary mode; Static environment features are obtained by encoding static environment information and initial detection box positions in the auxiliary mode using a multilayer perceptron. By using the dynamic intent estimation branch, a continuous intent score that changes over time is output based on the main modality time features and the posture features, which serves as the dynamic intent feature; The conditional vector is obtained by concatenating the main modality time features, posture features, static environment features, and dynamic intent features.

5. The pedestrian crossing prediction method based on multimodal feature fusion according to claim 1, characterized in that, The priors of the Gaussian mixture model satisfy the following relationship: ； In the formula, Indicates the number of Gaussian components. Indicates mixed weights, This represents the learnable parameters.

6. The pedestrian crossing prediction method based on multimodal feature fusion according to claim 1, characterized in that, The training process of the method employs a multi-task joint loss function, and the total loss satisfies the following relationship: ； In the formula, Indicates the main predicted loss. Represents the distribution alignment loss. Indicates the prior regularization loss. Indicates dynamic intention loss. This represents the reconstruction loss.

7. The pedestrian crossing prediction method based on multimodal feature fusion according to claim 1, characterized in that, The training of the method employs a phased strategy: In the first stage, based on the current observation data and future observation data of the main modality, a first feature extraction network for extracting current time-series features and a second feature extraction network for extracting future observation features are independently pre-trained. In the second stage, the parameters of the first feature extraction network and the second feature extraction network are frozen, and the conditional variational autoencoder structure, the multimodal feature encoder and the dynamic intent estimation branch are jointly trained.

8. The pedestrian crossing prediction method based on multimodal feature fusion according to claim 7, characterized in that, In the second phase of training, a potential hybrid strategy is employed: The refined latent vectors used by the decoder are replaced with samples sampled from the first latent distribution of the teacher network with a preset probability.

9. A pedestrian crossing prediction system based on multimodal feature fusion, characterized in that, The system for implementing the method of any one of claims 1 to 9 comprises: The data acquisition and preprocessing module is used to acquire and preprocess multimodal input data within the current observation time window; The feature extraction module, which includes an enhanced Transformer backbone network, is used to extract the temporal features of the current observation and the features of future observations from the preprocessed data. The multimodal coding and condition construction module is used to construct a condition vector based on the temporal features, attitude features, static environment features, and dynamic intention features of the current observation; The future-oriented integrated generation module includes a conditional variational autoencoder with a teacher-student structure; The teacher network is used to receive the conditional vector and future observation features, and output the first latent distribution; The student network is used to receive the conditional vector and a prior latent vector sampled from the Gaussian mixture model prior, output a second latent distribution aligned with the first latent distribution, and generate a refined latent vector. The decoding and output module is used to receive the condition vector and the refined latent vector, decode and output the predicted probability and uncertainty measure of pedestrian crossing.

10. The pedestrian crossing prediction system based on multimodal feature fusion according to claim 9, characterized in that, The enhanced Transformer backbone network includes: An adaptive position coding unit is used to generate adaptive position codes for focusing on key time segments; The expert hybrid unit comprises multiple expert subnetworks dynamically selected by the routing network to achieve diverse feature representations; The information bottleneck compression unit is used to reduce the dimensionality of features while retaining key information.

Images