Pedestrian situation awareness joint prediction method based on multi-modal time sequence feature fusion

Abstract

The application relates to a pedestrian situation awareness joint prediction method based on multi-modal time sequence feature fusion. The method extracts head posture features, upper limb behavior features, gait dynamics features and motion trajectory features of pedestrians through multi-dimensional feature extraction on preprocessed continuous vehicle-mounted video frame images. The extracted features are subjected to space-time attention fusion to generate fused space-time feature representation. The space-time feature representation is input into a joint prediction model to output situation awareness classification probability, risk level score and behavior prediction probability, thereby comprehensively improving the accuracy and robustness of pedestrian state prediction. The method provides key technical support for active safety protection of an intelligent traffic system and has good scene adaptability and can be extended to multiple application fields such as intelligent traffic management and virtual testing.

Description

Technical Field

[0001] This application relates to the field of pedestrian contextual awareness and behavior prediction technology, and in particular to a joint prediction method for pedestrian contextual awareness based on multimodal temporal feature fusion. Background Technology

[0002] In autonomous driving and advanced driver assistance systems (ADAS), accurately predicting pedestrian behavior and their situational awareness is crucial for ensuring driving safety. Traditional methods often rely on single-modal information (such as trajectory prediction or pose estimation), which struggles to comprehensively reflect a pedestrian's dynamic cognitive state and behavioral intentions, and are prone to misjudgment, especially in complex traffic scenarios. Existing vision-based pedestrian analysis technologies are largely limited to static features or short-term time-series modeling, lacking the ability to fuse and model multi-dimensional, long-term time-series features, resulting in insufficient prediction accuracy and robustness. Summary of the Invention

[0003] Therefore, it is necessary to provide a pedestrian context awareness joint prediction method based on multimodal temporal feature fusion, including: S1: Perform multi-dimensional feature extraction on the preprocessed continuous vehicle video frame images to extract the head posture features, upper limb behavior features, gait dynamics features, and motion trajectory features of the pedestrians; S2: Perform spatiotemporal attention fusion on the extracted features to generate a fused spatiotemporal feature representation; S3: Input the spatiotemporal feature representation into the joint prediction model and output the situational awareness classification probability, risk level score, and behavior prediction probability.

[0004] Beneficial effects: This method extracts multi-dimensional features from preprocessed continuous vehicle video frames, including pedestrian head posture features, upper limb behavior features, gait dynamics features, and motion trajectory features. The extracted features are then fused using spatiotemporal attention to generate a fused spatiotemporal feature representation. This representation is input into a joint prediction model, which outputs contextual awareness classification probability, risk level score, and behavior prediction probability. This comprehensively improves the accuracy and robustness of pedestrian state prediction. This method provides key technical support for proactive safety protection in intelligent transportation systems and has good scene adaptability, making it applicable to multiple fields such as intelligent traffic management and virtual testing. Attached Figure Description

[0005] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0006] Figure 1 This is a flowchart of the pedestrian context awareness joint prediction method based on multimodal temporal feature fusion in the embodiments of this application. Detailed Implementation

[0007] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0008] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0009] like Figure 1 As shown, this embodiment provides a pedestrian context awareness joint prediction method based on multimodal temporal feature fusion, including: S1: Perform multi-dimensional feature extraction on the preprocessed continuous vehicle video frame images to extract the pedestrian's head posture features, upper limb behavior features, gait dynamics features, and motion trajectory features.

[0010] In this embodiment, the model is trained and tested on the public datasets JAAD and PIE. 30 consecutive frames of in-vehicle video images are captured by the in-vehicle camera at a frame rate of 30fps and an image resolution of 1920×1080 pixels.

[0011] In this embodiment, the preprocessing of consecutive vehicle-mounted video frames includes: The YOLOv5s detector is used to detect pedestrians in any frame of the vehicle video image. The bounding box information of each pedestrian in the corresponding vehicle video frame image is obtained (including the coordinates of the center point of the bounding box, the length and width of the bounding box). The pedestrian area image (256×128 pixels) is then cropped from the corresponding vehicle video frame image.

[0012] Specifically, pedestrian head posture features include 3D head posture angles, gaze direction, and output attention heatmaps, extracted using methods including: Extract the pedestrian's three-dimensional head posture angle and gaze direction from the pedestrian area image corresponding to any frame of the vehicle video image; The pedestrian region image is processed through a ResNet-18 backbone network to extract pedestrian appearance features; the vehicle video frame image is processed through a semantic segmentation network to obtain a scene semantic feature map; the 3D head pose angle, gaze direction, pedestrian appearance features, and scene semantic feature map are processed through a Transformer decoder to output an attention heatmap.

[0013] In this embodiment, the specific extraction process of the three-dimensional head attitude angles (including pitch angle, yaw angle, and roll angle, in degrees) includes: Facial keypoint detection: A heatmap-based convolutional neural network is used, with a lightweight version of the Hourglass Network structure, consisting of four stacked hourglass modules. Each module contains skip connections and residual blocks. The input is a pedestrian region image, and the output is a heatmap of 68 facial keypoints (heatmap resolution 64×32, 68 channels). Each pixel value in the heatmap represents the confidence score of the corresponding keypoint.

[0014] Key point coordinate extraction: For each heatmap channel, the position of the maximum value is taken as the key point coordinate (u). k , v k The algorithm was optimized using subpixel precision (Gaussian fitting within a 3×3 neighborhood around the maximum value). This yielded the coordinates of 68 two-dimensional keypoints.

[0015] Attitude calculation: Prepare a standard 3D face model containing the coordinates of 68 3D key points (unit: mm). This model is obtained by average face scanning.

[0016] The Efficient Perspective-n-Point (EPnP) algorithm is used to solve for the face pose in the camera coordinate system. Assuming the camera intrinsic matrix is ​​known (obtained through calibration), the EPnP algorithm constructs four virtual control points, transforms the 2D-3D correspondence into a system of linear equations, and solves for the rotation matrix R and translation vector.

[0017] Decompose Euler angles from the rotation matrix: pitch angle Yaw angle Roll angle (In ZYX order), the output unit is degrees.

[0018] Pose fine-tuning: To further improve accuracy, the initial pose angle is concatenated with facial image features (extracted from the intermediate layer of the hourglass network), and then input into a small residual network consisting of two fully connected layers (256-dimensional → 128-dimensional → 3-dimensional), outputting the residual. The final attitude angle is The residual network is jointly trained with the keypoint detection network, and the loss function is the mean square error of the attitude angle.

[0019] In this embodiment, the specific extraction process of the line of sight direction (including horizontal and vertical deflection angles, in degrees) includes: Eye area cropping: Based on facial key points (inner and outer corners of the left eye, inner and outer corners of the right eye), crop out image blocks E for the left and right eyes. left E right The size is normalized to 60×36 pixels.

[0020] The gaze regression network employs a lightweight CNN architecture (GazeNet), consisting of four convolutional layers (each with a 3×3 kernel, stride of 2, and 32, 64, 128, and 256 channels respectively) and two fully connected layers (512→128→2). The input is a concatenated left and right eye image (channel overlay, size 60×36×6), and the output is the gaze direction angle.

[0021] Uncertainty modeling: The network outputs an additional variance (through another fully connected branch), and negative log-likelihood loss is used during training, enabling the network to output high uncertainty in occluded or blurred conditions.

[0022] Gaze direction fusion: During inference, if facial occlusion leads to low confidence in gaze estimation (variance > threshold), head pose is used as a prior to approximate the gaze direction as the head orientation (i.e., the gaze and head pose yaw angle are consistent, and the pitch angle is appropriately offset); otherwise, the network prediction is directly output.

[0023] In this embodiment, the specific extraction process of the attention heatmap (resolution 1920×1080, pixel value representing the probability that the location is noticed by a pedestrian) includes: Feature extraction: The pedestrian region image is processed through a ResNet-18 backbone network to extract pedestrian appearance features F. app (Dimensions: 8×4×512).

[0024] Head pose and gaze direction are encoded into two 128-dimensional vectors, which are then mapped to the same space as the appearance features through a fully connected layer.

[0025] Extract local semantic feature maps F centered on pedestrian locations from the scene semantic segmentation map. scene (Size 256×256, cropped and scaled to 64×64), and encoded as a feature.

[0026] Attention Decoder: A Transformer-based decoder structure is adopted, using the above features as queries and global scene features as keys and values, and attention weights are generated through a multi-head attention mechanism (8 heads).

[0027] The decoder outputs an attention probability map of the same size as the original image, obtained through bilinear upsampling and softmax normalization.

[0028] Loss function: During training, cross-entropy loss is calculated using real attention annotations (either via eye-tracking data or manual annotation). If no real annotations are available, a self-supervised approach can be used, leveraging the pedestrian's subsequent behavior (such as whether they walk towards an object) as a weak supervision signal.

[0029] In the above head pose feature extraction process: 1. By combining EPnP and residual fine-tuning, high-precision 3D pose estimation is achieved with an accuracy of ±2°; 2. Gait inference introduces probabilistic modeling to enhance occlusion robustness; 3. Attention prediction is integrated with scene context, which is more consistent with the real cognitive process.

[0030] The pedestrian's upper limb behavioral features include the coordinates of 17 upper limb key points, segmentation masks for various handheld objects, and probability distributions of various interaction patterns between hands and handheld objects. Extraction methods include: Extract the coordinates of 17 upper limb key points of the pedestrian from the pedestrian area image corresponding to any frame of vehicle video; use the coordinates of the 17 upper limb key points as the center to detect the segmentation mask of various handheld objects from the pedestrian area image; based on the coordinates of the 17 upper limb key points and the segmentation mask of various handheld objects, determine the probability distribution of two-handed operation mode, one-handed grip mode or non-handed interaction mode.

[0031] In this embodiment, the specific extraction process of the coordinates of 17 upper limb key points (including shoulder, elbow, wrist, hand, etc., each point including coordinates and confidence level) includes: Backbone network: HRNet-W32 is used, which consists of 4 stages, each containing multiple residual units and maintaining high-resolution representation. The input is a pedestrian region image, and the output is a multi-resolution feature map. Finally, a 1×1 convolution is used to output a heatmap of 17 key points with a size of 64×32×17.

[0032] Post-processing of heatmaps: For each key point, the position of the maximum value in the heatmap is taken as the initial coordinate.

[0033] Subpixel precision coordinates are obtained by Gaussian fitting within a 3×3 neighborhood around the maximum value.

[0034] The confidence score is the normalized value of the maximum value in the heatmap.

[0035] Temporal smoothing: A one-dimensional Kalman filter is used to smooth the keypoint coordinate sequence, and the filter parameters are adaptively adjusted based on the keypoint motion speed.

[0036] In this embodiment, the specific extraction process of segmentation masks for various handheld objects (such as mobile phones, beverage bottles, books, etc.) includes: Candidate region generation: Multiple candidate boxes are generated centered on upper limb key points (such as wrists and hands) (the size of which adapts to the distance between key points). The entire pedestrian region is also used as the context.

[0037] Improved YOLOv8-seg model: Basic architecture: YOLOv8-seg uses CSPDarknet as the backbone, PANet as the feature pyramid, and adds a segmentation head.

[0038] Improvements: Add a coordinate attention module to the neck network to enhance the ability to locate small objects; introduce spatial modulation in the detection head to guide segmentation using key point information.

[0039] Training: The model was trained using a self-built dataset (containing handheld object annotations), and the loss function was a weighted sum of classification loss (cross-entropy), bounding box regression loss (CIoU), and segmentation loss (Dice Loss).

[0040] Inference: Input a pedestrian region image and output the detection results, including the category, confidence score, and segmentation mask for each detection box, retaining detection results with a confidence score > 0.5.

[0041] In this embodiment, the specific extraction process of the probability distribution of various interaction patterns between hands and objects includes: Feature calculation: For each frame, calculate the geometric features between key points of the upper limb: joint angles (elbow joint, shoulder joint), distance between hand and head, and distance between hand and torso.

[0042] The categories of handheld objects are encoded as one-hot vectors (10 categories in total), and key point coordinates are concatenated.

[0043] Obtain the feature vector for each frame.

[0044] Temporal classification networks: The feature vector is input into a temporal convolutional network (TCN), which contains 3 dilated convolutional layers (dilation=1,2,4), with each layer having a kernel size of 3 and 128 channels.

[0045] After global average pooling, two fully connected layers (256→128→3) are applied, and the probability is output using Softmax.

[0046] Attitude-device association model: Establish the rule: When holding a mobile phone and the hand is close to the ear (distance <30 pixels), the category will be changed to the "making a call" subcategory (but the main category will still be classified as single-handed holding).

[0047] When both hands hold objects and are close to each other, it is considered a two-handed operation.

[0048] In the process of extracting the above upper limb behavioral features: 1. Improve the recall rate of small objects to 89.2% by co-optimizing HRNet keypoint detection and handheld object segmentation; 2. Capture dynamic interaction patterns through temporal convolutional networks; 3. Enhance behavioral semantic understanding through pose-device association.

[0049] The gait dynamics features of pedestrians include gait feature vectors, gait dynamic parameters, center of mass displacement, trunk tilt rate of change, gait anomaly index, current gait state probability, and state transition prediction. Extraction methods include: The pedestrian region images in the continuous vehicle-mounted video frames are sequentially passed through the I3D network to output the gait feature vector and gait dynamic parameters of the corresponding pedestrians. Based on the bounding box information sequence of the same pedestrian in the continuous vehicle-mounted video frames and the coordinate sequence of 17 upper limb key points, the centroid displacement, trunk tilt angle change rate and gait anomaly index of the corresponding pedestrians are calculated. The gait feature vector is passed through a hidden Markov model to obtain the current gait state probability and state transition prediction.

[0050] In this embodiment, the specific extraction process of gait feature vectors and gait dynamic parameters (including stride length, stride frequency, and symmetry) includes: 3D Convolutional Networks: The I3D network structure is adopted, which contains multiple 3D convolutional and pooling layers. Specifically, the input is pedestrian region images from 16 consecutive frames of vehicle video images. The input is 7×7×7 convolution (stride 2,2,2) → 3×3×3 max pooling → two Inception modules (containing 1×1×1 and 3×3×3 convolutions) → 3×3×3 max pooling → five Inception modules → global average pooling, and the output is a 2048-dimensional feature vector.

[0051] Parameter decoding: The 2048-dimensional feature vector is input into three independent fully connected regression heads, which output stride (unit pixel), step frequency (steps / second), and symmetry (0~1 value, 1 indicates complete symmetry).

[0052] During training, real gait parameters (obtained through an optical motion capture system) are used as supervision, and the loss function is mean squared error.

[0053] Gait cycle detection: Gait cycle is estimated by autocorrelation analysis using the temporal changes of feature vectors, which assists in step frequency calculation.

[0054] In this embodiment, the specific extraction process of centroid displacement, trunk tilt rate of change, and gait abnormality index includes: Centroid displacement: Using the center of the bounding box as an approximation of the pedestrian's centroid, calculate the displacement difference between adjacent frames to obtain the velocity vector v_t = ( \Delta x_t, \Delta y_t ).

[0055] Torso tilt rate of change: The torso direction vector is calculated using key shoulder points (left and right shoulders), and the angle between this vector and the vertical direction is the torso tilt angle. Calculate the tilt rate of change. .

[0056] Gait abnormality index: Constructing a statistical distribution of normal gait: Gait parameters (stride length, gait frequency, symmetry, and rate of change of trunk tilt angle) are extracted from a large number of normal walking samples and fitted with a multivariate Gaussian distribution.

[0057] For the current gait parameter vector, calculate the Mahalanobis distance as the gait anomaly index.

[0058] In this embodiment, the specific extraction process of the current gait state probability and state transition prediction includes: State definition: The gait pattern is discretized into 4 states: normal walking (S1), deceleration (S2), turning (S3), and running (S4).

[0059] Hidden Markov Model (HMM): Observation probability: Assume that the gait feature vector follows a Gaussian mixture distribution, with each state corresponding to a Gaussian component. Parameters are learned from the training data using the EM algorithm.

[0060] State transition matrix: The frequency of state changes between consecutive frames is statistically analyzed to obtain a 4×4 transition matrix.

[0061] Initial state distribution: Assuming uniform distribution in the first frame or set according to the context.

[0062] Online decoding: The Viterbi algorithm is used to decode the current sequence in real time, outputting the most likely state at each time step and providing the state probability. Simultaneously, the transition matrix is ​​used to predict the state probability at the next time step.

[0063] In the above gait dynamics feature extraction process: 1. Gait spatiotemporal features are extracted end-to-end using 3D-CNN, eliminating the need for manual design; 2. The Mahalanobis distance anomaly index is used to quantify pedestrian instability; 3. Gait dynamic evolution is modeled using HMM to enhance predictive ability.

[0064] Pedestrian trajectory features include the pedestrian's trajectory, behavioral response delay, future trajectory distribution parameters, and trajectory entropy, extracted using methods including: The bounding box information sequence of each pedestrian in the continuous vehicle video frame image is processed through a lightweight ReID network to extract the appearance features of each pedestrian, and the trajectory of the corresponding pedestrian is predicted based on each appearance feature; the behavior response delay of the pedestrian is calculated based on the trajectory change time of each pedestrian and scene event information; and the future trajectory distribution parameters and trajectory entropy are predicted by an LSTM network based on the historical trajectory of the same pedestrian.

[0065] In this embodiment, the specific process of extracting the pedestrian's trajectory includes: Detection and Feature Extraction: For each frame, YOLOv5s is used to obtain all pedestrian detection boxes. At the same time, for each detection box region, appearance features are extracted through a lightweight ReID network (based on ResNet-18, outputting 128-dimensional features).

[0066] Data association: The ByteTrack algorithm is used to classify detection boxes into two categories: high confidence and low confidence.

[0067] For high-confidence detection boxes, Kalman filtering is used to predict the current trajectory position, the IoU between the predicted box and the detection box is calculated, and the Hungarian algorithm is used for matching.

[0068] Unmatched high-confidence detection boxes are matched with low-confidence detection boxes a second time, and the matching conditions are relaxed (IoU threshold is lowered).

[0069] Detection boxes that still cannot be matched are initialized as new trajectories.

[0070] Occlusion handling: For tracks that fail to match but remain within the confidence interval, the tracks are preserved and Kalman filtering is used to continue prediction, while their features are cached. When the occlusion ends and the track reappears, the cached appearance features are used to re-match with the current detection.

[0071] In this embodiment, the specific extraction process of behavior response latency (in frames) includes: Event definition: Define environmental stimulus events, for example: Vehicle approaching: The distance between this vehicle and the pedestrian is less than the safety threshold (e.g., 10 meters).

[0072] Traffic light change: The traffic light at the intersection where pedestrians are about to cross changes from green to red.

[0073] Other vehicles crossing: There is a potential conflict between the trajectories of other vehicles and pedestrians.

[0074] Behavior change detection: Real-time monitoring of pedestrian trajectories and behaviors (such as speed and direction) is performed, and the CUSUM algorithm is used to detect change points. When a significant change is detected (such as a decrease in speed >20% or a change in direction >30°), the time of change is recorded.

[0075] Delay calculation: The time of occurrence of the stimulus event and the time of change in behavior. If there is no significant change, the delay is set to infinity (or marked as no response).

[0076] In this embodiment, the specific extraction process of trajectory distribution parameters (including mean and variance) and trajectory entropy for the next 1-3 seconds includes: Trajectory encoding: The historical trajectory sequence is encoded using an LSTM encoder (2 layers, 128-dimensional hidden layer) to obtain a context vector.

[0077] Decoding Prediction: The decoder uses an LSTM, which generates future trajectory points step by step, starting with the context vector as the initial state. At each time step, it outputs the parameters of a two-dimensional Gaussian distribution.

[0078] The loss function is the negative log-likelihood: K is the number of prediction steps (corresponding to 3 seconds, i.e. 90 frames, which can be downsampled to 30 steps). This represents the coordinates of the actual trajectory point at time t+k. This represents the mean value predicted by the model at time k. This represents the covariance matrix predicted by the model at time k. This represents the probability density function of a two-dimensional Gaussian distribution.

[0079] Trajectory entropy calculation: For the predicted distribution, calculate the entropy. , to measure uncertainty.

[0080] In the process of extracting the above motion trajectory features: 1. Improved occlusion adaptation through ByteTrack to enhance tracking continuity; 2. For the first time, environmental response time was used as a contextual awareness indicator through interaction delay quantization; 3. Provided probabilistic trajectory prediction and entropy output to provide a basis for risk assessment.

[0081] S2: Perform spatiotemporal attention fusion on the extracted features to generate a fused spatiotemporal feature representation.

[0082] Specifically, the steps include: The extracted features are dimensionally aligned using the corresponding first fully connected layer (aligning all features to 256 dimensions). Short-term mode of 3-5 frames: For every consecutive 3-frame short-term window, the head pose features, upper limb behavior features, gait dynamics features, and motion trajectory features of the same pedestrian in each frame of the vehicle video image within the short-term window are stacked along the time axis into a first tensor (size is 3×1024, where 1024 is the dimension of the 256-dimensional head pose features, upper limb behavior features, gait dynamics features, and motion trajectory features). The first tensor is passed through a one-dimensional convolutional layer (Conv1D, kernel size is 3, stride is 1, output channel is 512) and global max pooling to output the first feature. The first feature is then labeled with events based on the features within the short-term window to obtain short-term features. At the same time, an event triggering rule is defined: if a preset high-risk posture combination is detected (e.g., head pitch angle > 25° and confidence of the held object > 0.8 and stride reduction > 15%), an event label (binary vector) is generated.

[0083] Mid-term mode of 10-15 frames: For the mid-term window of 5 frames before and after the current frame, the head pose features, upper limb behavior features, gait dynamics features, and motion trajectory features of the same pedestrian in each frame of the vehicle video image within the short window are sequentially input into the bidirectional LSTM network (256-dimensional hidden layer) according to the time axis. The output of the last time step is used as the mid-term feature (512-dimensional). Hidden Markov model is used to identify typical behavior patterns (such as "observation-decision-action") and output the behavior pattern label probability (5 classes) as auxiliary features.

[0084] Long-term mode (30 frames or more): For all frames in the long-term window, the head pose features, upper limb behavior features, gait dynamics features, and motion trajectory features of the same pedestrian in all frames of vehicle video images are sequentially passed through a Transformer encoder (6 layers, 8 heads of multi-head attention, 1024-dimensional feedforward network) along the time axis to extract global context features (512 dimensions). Scene context information (traffic signal status, lane line position, etc.) is encoded into vectors, and the vectors are passed through a cross-attention mechanism with the global context features to obtain long-term features (512 dimensions). The short-term features, medium-term features, and long-term features are fused to generate a fused spatiotemporal feature representation.

[0085] Furthermore, the fusion based on the short-term features, the medium-term features, and the long-term features includes: The short-term features, medium-term features, and long-term features are concatenated into a concatenated feature (with a dimension of 1536). The features of the same pedestrian in all frames of vehicle video images (256 dimensions per frame) are reduced to 128 dimensions by passing them through the corresponding second fully connected layer, and the dimensions are unified to obtain the modal features corresponding to each feature. In each frame, the modal features are weighted and fused based on attention weights to obtain the corresponding modal fusion features (128 dimensions) for each frame; the attention weight calculation formula is: ,in, This represents the attention weight of the i-th modal feature. Represents the i-th modal feature. This represents the mapping matrix corresponding to the i-th modal feature. Let represent the bias corresponding to the i-th modal feature, tanh(·) represent the hyperbolic tangent function, and q represent the learnable query vector; For the modal fusion features of 30 frames, the self-attention weights of each frame relative to other frames are calculated based on the Transformer self-attention mechanism, and the modal fusion features of each frame are then subjected to global average pooling based on the self-attention weights to obtain the temporal fusion features (dimension 128). The temporal fusion features and the spliced ​​features are residually connected, and then the spliced ​​features are output after layer normalization (dimension 128) for subsequent joint prediction.

[0086] In the above fusion process: 1. Three-level temporal windows correspond to different cognitive levels, which is in line with the multi-scale characteristics of human behavioral understanding; 2. Dynamic focusing between modalities and in time is achieved through spatiotemporal attention mechanisms; 3. The accuracy of long-term intention prediction is enhanced by fusing scene context.

[0087] S3: Input the spatiotemporal feature representation into the joint prediction model and output the situational awareness classification probability, risk level score, and behavior prediction probability.

[0088] Specifically, the joint prediction model includes a first main task branch, a second main task branch, and a third main task branch; Set up three first fully connected layers with the same structure, and three second fully connected layers with the same structure. The dimension of the first fully connected layer is the same as the dimension of the spatiotemporal feature representation, and the dimension of the second fully connected layer is half the dimension of the spatiotemporal feature representation. The first main task branch includes a first fully connected layer, a second fully connected layer, a third fully connected layer with a dimension of 3, and a first output layer based on the Softmax function, arranged in sequence; the spatiotemporal feature representation is processed through the first main task branch to obtain the context awareness classification probability; The second main task branch includes a first fully connected layer, a second fully connected layer, a fourth fully connected layer with dimension 1, and a second output layer based on the Sigmoid function, arranged in sequence; the spatiotemporal feature representation is processed through the second main task branch to obtain the risk level score; The third main task branch includes a first fully connected layer, a second fully connected layer, a fifth fully connected layer with a dimension of 4, and a third output layer based on the Softmax function, arranged in sequence; the spatiotemporal feature representation is processed through the third main task branch to obtain the behavior prediction probability.

[0089] In this embodiment, the joint prediction model further includes a shared feature layer, which includes two residual blocks. The spatiotemporal feature representation is passed through the two residual blocks to obtain two residual features. The spatiotemporal feature representation is then added to the two residual features to obtain a shared representation. By concatenating the shared representation with the spatiotemporal feature representation, the uncertainty estimation feature is obtained; The uncertainty estimation features are sequentially passed through three fully connected layers with the same structure and dimensions as the fully connected layers in the second main task branch, and the final mapping result is passed through the Sigmoid function to output the prediction confidence.

[0090] In this embodiment, the joint prediction model further includes a first auxiliary task branch, a second auxiliary task branch, and a third auxiliary task branch; First auxiliary task branch: Head posture regression: The input is a spatiotemporal feature representation, which passes through a fully connected layer (128→64→3) and outputs a three-dimensional attitude angle, using identity activation.

[0091] Second auxiliary task branch: Device interaction detection: The input is a spatiotemporal feature representation, which passes through a fully connected layer (128→64→1), and the Sigmoid outputs the probability.

[0092] Third auxiliary task branch: Gait anomaly detection: The input is a spatiotemporal feature representation, which passes through a fully connected layer (128→64→1), and the Sigmoid outputs the probability.

[0093] This embodiment employs a joint learning strategy: Loss function: The total loss is the weighted sum of the losses from each task. ; in: L SA Cross-entropy loss: y SA,c p represents the true label of class c.c This represents the predicted probability of class c.

[0094] L risk Mean square error: r represents the model's predicted value (such as risk value, angle, etc.). gt This represents the actual value corresponding to r.

[0095] L act This represents the cross-entropy loss.

[0096] L head For L1 loss: θ represents the prediction angle. gt This indicates the true perspective.

[0097] L device and L gait It represents the cross-entropy of binary classification.

[0098] L conf For confidence loss: When the prediction is correct, c approaches 1, and when the prediction is incorrect, c approaches 0. This loss encourages the model to output high confidence when the prediction is correct and low confidence when the prediction is incorrect.

[0099] Dynamic weight adjustment: An uncertain weighting method is adopted, assuming that the loss of each task follows a Gaussian distribution, and its noise parameter σ i It is learnable. Therefore, the weighted loss is: ; Among them, L i Let σ represent the i-th type of loss. i Initialize to 1 and optimize together with network parameters.

[0100] In this embodiment, the multidimensional prediction results output for each frame ultimately include: context awareness classification probability, risk level score, behavior prediction probability, head pose angle, probability distribution of interaction mode, gait abnormality index, and prediction confidence.

[0101] In the above joint learning process: 1. Fully utilize labeled information through multi-task joint learning to improve the generalization ability of the main task; 2. Use uncertainty estimation to provide prediction credibility, which facilitates downstream decision-making; 3. Simplify hyperparameter tuning through dynamic weight adjustment.

[0102] This embodiment also provides a Bayesian probabilistic fusion framework (HBFM): Objective: To fuse four dimensions of evidence (head, upper limbs, gait, and trajectory) using a hierarchical Bayesian model to obtain the posterior probability of the final situational awareness state, avoiding the assumption of strong independence.

[0103] Mathematical model: Let situational awareness state SA ∈ {weakened, stronger, uncertain}. Let the four dimensions of observational evidence be denoted as E. H E L E G E T A hierarchical Bayesian model is constructed, introducing a latent variable Z (64 dimensions) to represent the pedestrian's internal cognitive state (such as attention level and intention). The model assumes that, given SA and Z, the observations in each dimension are independent. ; To simplify the calculation, Z is discretized into M prototypes (M = 16), which are obtained through clustering.

[0104] Specific implementation steps: Evidence Collection: Obtain observation features from each module. H e L e G e T This is then transformed into a probabilistic form. For example, the output of the head module can be viewed as sampling from a Gaussian distribution, with its mean and variance provided by the network. Continuous features are mapped to likelihood values ​​through a likelihood network.

[0105] Latent variable estimation: Using spatiotemporal features to represent F fina As an estimate of the latent variable Z, or inferred from four observed features by an encoder network q(Z|e). The encoder employs a two-layer MLP (256→128→64) and outputs the mean and variance of Z, assumed to be Gaussian distributed.

[0106] Learning about likelihood functions: For each dimension i and each state SA = c, train a likelihood network P(e i | SA=c, Z), network input is Z and e i The output is a likelihood value. Hybrid density networks (MDNs) can be used for modeling.

[0107] Specifically, given SA=c and Z, the likelihood of the MDN output Gaussian mixture model parameters (mean, variance, weights) is... .

[0108] Prior distribution: P(SA) can be obtained by statistically analyzing the class frequencies in the training data.

[0109] P(Z|SA) is modeled as a Gaussian distribution, whose mean and variance are determined by the state c, and is also learned through a neural network (inputting the one-hot encoding of SA and outputting the distribution parameters).

[0110] Posterior calculation: For the current observation e H e L e G e T First, q(Z|e) is obtained through the encoder as an approximate posterior for the latent variables. Then, the marginal likelihood approximation is: ; Where Q is the number of samples (e.g., Q=10).

[0111] Compute the posterior: ; The state with the highest posterior probability is taken as the final output, and the posterior distribution is output as a measure of uncertainty.

[0112] End-to-end training: The entire Bayesian framework is jointly trained with the preceding feature extraction network, and variational inference is used to optimize the lower bound of evidence (ELBO).

[0113] In the above posterior probability calculation process: 1. By introducing latent variables, the bias caused by the independence assumption is mitigated; 2. The neural network is used to learn the likelihood function and the prior of the latent variables, achieving end-to-end trainability; 3. The posterior probability distribution is output, which is convenient for integration with downstream decision-making modules.

[0114] This embodiment also provides adaptive threshold adjustment and real-time optimization: The adaptive threshold adjustment mechanism aims to dynamically adjust the sensitivity thresholds of each detection module based on environmental conditions, thereby improving the robustness of the model in complex scenarios.

[0115] Specific implementation steps: Environmental parameter perception: Illumination conditions: Calculate the average image brightness L mean and contrast C mean (Standard deviation). The Retinex theory is used to decompose the reflection and illumination components; the illumination component is used to determine the degree of low illumination. If L... mean <50 or C mean If the value is less than 20, it is determined to be low light. The image enhancement module (based on the MSRCR algorithm) is then enabled before the image is input into the network.

[0116] Pedestrian distance: Estimated based on the height h of the pedestrian detection box. Where f is the camera focal length and H is the average height of pedestrians (1.7m). The greater the distance, the more difficult the detection becomes.

[0117] Occlusion level: Mean confidence score of keypoint detection Assess occlusion, if A value less than 0.6 indicates severe occlusion.

[0118] Dynamic threshold function: Head posture angle threshold The normal threshold is 2°, which is relaxed when the distance is far. ; Confidence threshold for handheld object detection Base threshold 0.5, reduced under low light conditions: , ; Gait abnormality index threshold The base Mahalanobis distance threshold is 3.0, which is reduced when occlusion is severe. ; Adaptive event triggering: The above thresholds are used to identify high-risk posture combinations and for post-processing filtering. For example, when a head pitch angle > 100° is detected... And the confidence level of the held object is > At that time, an early warning is triggered.

[0119] Real-time optimization strategy, goal: to achieve real-time inference (<50ms / frame) on automotive platforms (such as NVIDIA Jetson AGX Xavier) while ensuring accuracy.

[0120] Specific technologies: 1. Lightweight model: Knowledge distillation: Using ResNet-101 as the teacher network and MobileNetV3 as the student network, distillation was performed based on ImageNet pre-training. The distillation loss included soft-label cross-entropy and intermediate layer feature matching (using attention transfer). The final student network maintained 95% accuracy while reducing the number of parameters to 4.2M.

[0121] Lightweighting of each submodule: Head pose estimation: The hourglass network was replaced with a smaller network with 4 residual blocks, and the key point heatmap resolution was 32×16.

[0122] Replace HRNet with Lite-HRNet-18.

[0123] 3D-CNN was replaced with R(2+1)D-18.

[0124] The ReID network in ByteTrack has been replaced with MobileNetV2 (128-dimensional).

[0125] 2. Inter-frame feature reuse: For each pedestrian, calculate the cosine similarity of the appearance features between the current frame and the previous frame. If it is greater than 0.95, it is considered that the pedestrian's appearance has not changed significantly, and the pose, key points and other features of the previous frame are reused.

[0126] For trajectory prediction, if the trajectory motion is smooth (small speed change), the LSTM hidden state of the previous frame is reused, and only the current position is updated.

[0127] 3. Asynchronous processing pipeline: Four feature extraction modules are deployed on four CUDA streams and executed in parallel. Each module takes the same pedestrian region image (already cropped) as input and outputs feature vectors and then synchronizes.

[0128] The temporal fusion network and joint prediction model run on the CPU in parallel with feature extraction: feature extraction of the current frame and fusion prediction of the previous frame are performed simultaneously, utilizing asynchronous CPU-GPU copying.

[0129] 4. Mixed-precision inference: All network inference is performed using FP16 precision, and TensorRT optimization reduces memory bandwidth and computation time.

[0130] 5. Dynamic frame rate adjustment: When a static scene or stable pedestrian behavior is detected (e.g., gait anomaly index < exponential threshold and trajectory entropy < entropy threshold), the processing frame rate is reduced to 15fps, and processing is performed once every other frame. Intermediate frames are filled in with features through linear interpolation.

[0131] In the above real-time optimization process: 1. Improve generalization ability by dynamically adjusting the threshold based on environment adaptation; 2. Achieve a balance between accuracy and speed by combining knowledge distillation with a lightweight backbone; 3. Give full play to the hardware parallel capabilities by adopting feature reuse and asynchronous pipelines; 4. Further reduce the computational load by adjusting the dynamic frame rate.

[0132] Tests on the public datasets JAAD and PIE show that our method achieves an accuracy of 92.3% in context-aware classification tasks, which is 18.7% higher than the baseline method.

[0133] The system module corresponding to this method can be deployed on an actual vehicle platform (NVIDIA Jetson AGX Xavier) to achieve real-time pedestrian status monitoring and early warning. The system supports a three-level early warning strategy, taking visual prompts, audible warnings, or active braking intervention based on the risk level.

[0134] The pedestrian context awareness joint prediction method based on multimodal temporal feature fusion provided in this embodiment has the following beneficial effects: 1. By fusing multimodal temporal features, the accuracy and robustness of pedestrian state prediction are comprehensively improved; 2. Introduce hierarchical Bayesian fusion and multi-task learning to enhance the interpretability and generalization ability of the model; 3. Supports real-time edge computing deployment to meet the low latency and high reliability requirements of autonomous driving systems; 4. It has good adaptability to various scenarios and can be extended to multiple application areas such as intelligent traffic management and virtual testing.

[0135] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A pedestrian context awareness joint prediction method based on multimodal temporal feature fusion, characterized in that, include: S1: Perform multi-dimensional feature extraction on the preprocessed continuous vehicle video frame images to extract the head posture features, upper limb behavior features, gait dynamics features, and motion trajectory features of the pedestrians; S2: Perform spatiotemporal attention fusion on the extracted features to generate a fused spatiotemporal feature representation; S3: Input the spatiotemporal feature representation into the joint prediction model and output the situational awareness classification probability, risk level score, and behavior prediction probability.

2. The method according to claim 1, characterized in that, The preprocessing of continuous vehicle-mounted video frames includes: The YOLOv5s detector is used to detect pedestrians in any frame of the vehicle video image, obtain the bounding box information of each pedestrian in the corresponding vehicle video frame image, and then crop the pedestrian area image from the corresponding vehicle video frame image.

3. The method according to claim 2, characterized in that, Pedestrian head posture features include 3D head posture angles, gaze direction, and output attention heatmaps, extracted using methods including: Extract the pedestrian's three-dimensional head posture angle and gaze direction from the pedestrian area image corresponding to any frame of the vehicle video image; The pedestrian region image is processed through a ResNet-18 backbone network to extract pedestrian appearance features; the vehicle video frame image is processed through a semantic segmentation network to obtain a scene semantic feature map; the 3D head pose angle, gaze direction, pedestrian appearance features, and scene semantic feature map are processed through a Transformer decoder to output an attention heatmap.

4. The method according to claim 2, characterized in that, The pedestrian's upper limb behavioral features include the coordinates of 17 upper limb key points, segmentation masks for various handheld objects, and probability distributions of various interaction patterns between hands and handheld objects. Extraction methods include: Extract the coordinates of 17 upper limb key points of the pedestrian from the pedestrian area image corresponding to any frame of vehicle video; use the coordinates of the 17 upper limb key points as the center to detect the segmentation mask of various handheld objects from the pedestrian area image; based on the coordinates of the 17 upper limb key points and the segmentation mask of various handheld objects, determine the probability distribution of two-handed operation mode, one-handed grip mode or non-handed interaction mode.

5. The method according to claim 2, characterized in that, The gait dynamics features of pedestrians include gait feature vectors, gait dynamic parameters, center of mass displacement, trunk tilt rate of change, gait anomaly index, current gait state probability, and state transition prediction. Extraction methods include: The pedestrian region images in the continuous vehicle-mounted video frames are sequentially passed through the I3D network to output the gait feature vector and gait dynamic parameters of the corresponding pedestrians. Based on the bounding box information sequence of the same pedestrian in the continuous vehicle-mounted video frames and the coordinate sequence of 17 upper limb key points, the centroid displacement, trunk tilt angle change rate and gait anomaly index of the corresponding pedestrians are calculated. The gait feature vector is passed through a hidden Markov model to obtain the current gait state probability and state transition prediction.

6. The method according to claim 2, characterized in that, Pedestrian trajectory features include the pedestrian's trajectory, behavioral response delay, future trajectory distribution parameters, and trajectory entropy, extracted using methods including: The bounding box information sequence of each pedestrian in the continuous vehicle video frame image is processed through a lightweight ReID network to extract the appearance features of each pedestrian, and the trajectory of the corresponding pedestrian is predicted based on each appearance feature; the behavior response delay of the pedestrian is calculated based on the trajectory change time of each pedestrian and scene event information; and the future trajectory distribution parameters and trajectory entropy are predicted by an LSTM network based on the historical trajectory of the same pedestrian.

7. The method according to claim 1, characterized in that, The process of performing spatiotemporal attention fusion on the extracted features includes: The extracted features are dimensionally aligned using the corresponding first fully connected layer. For each consecutive 3-frame short window, the head pose features, upper limb behavior features, gait dynamics features, and motion trajectory features of the same pedestrian in each frame of the vehicle video image within the short window are stacked into a first tensor along the time axis. The first tensor is passed through a one-dimensional convolutional layer to output the first feature. Based on each feature within the short window, the first feature is event-labeled to obtain the short-term feature. For the intermediate window of 5 frames before and after the current frame, the head pose features, upper limb behavior features, gait dynamics features, and motion trajectory features of the same pedestrian in each frame of the vehicle video image within the short window are sequentially input into the bidirectional LSTM network according to the time axis, and the output of the last time step is used as the intermediate features. For the long-term window of all frames, the head pose features, upper limb behavior features, gait dynamics features, and motion trajectory features of the same pedestrian in all frames of vehicle video frames are sequentially passed through the Transformer encoder along the time axis to extract global context features; the scene context information is encoded into a vector, and the vector and the global context features are passed through a cross-attention mechanism to obtain long-term features; The short-term features, medium-term features, and long-term features are fused to generate a fused spatiotemporal feature representation.

8. The method according to claim 7, characterized in that, The fusion based on the short-term features, the medium-term features, and the long-term features includes: The short-term feature, the medium-term feature, and the long-term feature are concatenated into a concatenated feature; The features of the same pedestrian in all frames of vehicle video images are aligned and then reduced in dimensionality through the corresponding second fully connected layer. The dimensionality is then unified to obtain the modal features corresponding to each feature. In each frame, the modal features are weighted and fused based on attention weights to obtain the modal fusion features corresponding to each frame; The self-attention weights of each frame relative to other frames are calculated based on the self-attention mechanism, and the modal fusion features of each frame are then subjected to global average pooling based on the self-attention weights to obtain the temporal fusion features. The temporal fusion features and the spliced ​​features are residually connected, and then the fused spatiotemporal feature representation is output after layer normalization.

9. The method according to claim 1, characterized in that, The joint prediction model includes a first main task branch, a second main task branch, and a third main task branch; Set up three first fully connected layers with the same structure, and three second fully connected layers with the same structure. The dimension of the first fully connected layer is the same as the dimension of the spatiotemporal feature representation, and the dimension of the second fully connected layer is half the dimension of the spatiotemporal feature representation. The first main task branch includes a first fully connected layer, a second fully connected layer, a third fully connected layer with a dimension of 3, and a first output layer based on the Softmax function, arranged in sequence; the spatiotemporal feature representation is processed through the first main task branch to obtain the context awareness classification probability; The second main task branch includes a first fully connected layer, a second fully connected layer, a fourth fully connected layer with dimension 1, and a second output layer based on the Sigmoid function, arranged in sequence; the spatiotemporal feature representation is processed through the second main task branch to obtain the risk level score; The third main task branch includes a first fully connected layer, a second fully connected layer, a fifth fully connected layer with a dimension of 4, and a third output layer based on the Softmax function, arranged in sequence; the spatiotemporal feature representation is processed through the third main task branch to obtain the behavior prediction probability.

10. The method according to claim 9, characterized in that, The joint prediction model further includes a shared feature layer, which comprises two residual blocks. The spatiotemporal feature representation is passed through the two residual blocks to obtain two residual features. The spatiotemporal feature representation is then added to the two residual features to obtain the shared representation. By concatenating the shared representation with the spatiotemporal feature representation, the uncertainty estimation feature is obtained; The uncertainty estimation features are sequentially passed through three fully connected layers with the same structure and dimensions as the fully connected layers in the second main task branch, and the final mapping result is passed through the Sigmoid function to output the prediction confidence.

Images