Frame-based multi-modal pedestrian crossing intention prediction method and system

Abstract

This invention discloses a multimodal pedestrian crossing intention prediction method and system based on frame pair coupling. The method includes: acquiring a temporal image containing the target pedestrian and synchronous vehicle motion information; extracting multimodal features including pedestrian trajectory, posture, appearance, and scene semantics; encoding each frame's features into a unified-dimensional embedded representation to form a frame-level feature sequence; inputting the sequence into a feature interaction network based on an attention mechanism, where the network achieves early interaction fusion of intra-frame multimodal features through a self-attention layer; and adaptively fusing information from the previous frame to the current frame through frame pair fusion units embedded between layers to obtain an updated feature sequence; and predicting the pedestrian crossing probability and determining the intention based on the task identifier features in the updated sequence. Through early fusion and explicit frame pair temporal coupling, the accuracy and timeliness of pedestrian crossing intention prediction in complex scenarios are effectively improved.

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving and intelligent transportation technology, and in particular to a multimodal pedestrian crossing intention prediction method and system based on frame pair coupling. Background Technology

[0002] In autonomous and assisted driving systems, accurately and proactively predicting whether pedestrians intend to cross the road is crucial for planning safe driving routes and implementing timely braking or evasive strategies. Effective intent prediction can buy the system valuable decision-making and execution time, thereby significantly improving road traffic safety.

[0003] Existing pedestrian intention prediction solutions primarily rely on information acquired from visual sensors. One type of method focuses on using single-modal information for prediction; for example, predicting future positions solely based on a pedestrian's historical trajectory points, or determining their action state based solely on a pedestrian's posture skeleton map. While these methods may be effective in ideal scenarios, in real-world complex traffic environments, such as those with occlusion, varying lighting, cluttered backgrounds, or when pedestrians are stationary or in a state of slight movement, the cues provided by a single information source are often insufficient and unstable, leading to decreased prediction reliability and insufficient generalization ability.

[0004] To improve robustness, another approach attempts to fuse multi-source information. However, many existing fusion strategies employ a "late fusion" model, where separate encoding networks (such as convolutional neural networks and recurrent neural networks) are used to process data from different modalities (e.g., images, trajectories, speeds) separately. Later in the encoding process or at the decision level, the high-dimensional features of each modality are simply concatenated or weighted averaged. This strategy has significant limitations: the data from different modalities lack interaction in the early stages of encoding, and their inherent fine-grained semantic relationships, such as the synchronous changes in specific body movements and trajectory directions, and the constraints of scene semantics on pedestrian intentions, are not fully modeled and utilized. Independent encoding leads to the compression or loss of cross-modal correlation information, limiting the model's ability to understand complex coupling relationships and thus affecting prediction accuracy.

[0005] Furthermore, the emergence of a pedestrian's intention to cross the street is often accompanied by a series of brief and subtle dynamic changes, such as leaning forward, slightly lifting their feet, and turning their head to observe traffic. These subtle dynamic changes are key clues for determining intent. Existing temporal modeling methods, such as those using 3D convolutions, long short-term memory networks, or temporal Transformers, typically aggregate or recursively update features from a continuous sequence of frames at once, lacking explicit and structured modeling of instantaneous change patterns between adjacent frames. This modeling approach is insufficiently sensitive to these short-term micro-dynamics, making it difficult to capture the subtle signals that trigger intent, resulting in limited lead time for prediction and an inability to provide sufficiently early warnings before the pedestrian actually begins crossing.

[0006] Therefore, a pedestrian crossing intention prediction technology is needed that can overcome the above-mentioned defects. It should be able to achieve fine-grained deep fusion of multimodal information in the early stage and have the ability to efficiently capture and accumulate short-term dynamics between consecutive frames, so as to achieve more accurate and earlier predictions in complex real-world scenarios. Summary of the Invention

[0007] Therefore, the technical problem to be solved by the present invention is to overcome the problems of insufficient multimodal information fusion and interaction and difficulty in capturing short-term micro-dynamics between consecutive frames in the prior art, which leads to insufficient accuracy and lead time in predicting pedestrian crossing intentions.

[0008] To address the aforementioned technical problems, this invention provides a multimodal pedestrian crossing intention prediction method based on frame pair coupling, comprising the following steps: S1: Acquire a temporal image sequence containing the target pedestrian and time-synchronized vehicle motion information; extract multiple modal features from the temporal image sequence and the vehicle motion information to describe the pedestrian's state, behavior, and scene; S2: Encode the multiple modal features of each frame into an embedding representation in a unified dimensional space, add a task identifier, and obtain the initial feature sequence of each frame; S3: Input the initial feature sequence of each frame into the feature interaction network based on the attention mechanism. The feature interaction network fuses and interacts the embedded representations of different modalities within the same frame through the self-attention layer. Through the frame pair fusion unit embedded in at least one network layer, the feature information from the previous frame is fused into the feature information of the current frame to obtain the updated feature sequence that fuses multimodal interaction and strengthens temporal coupling. S4: Predict the pedestrian's crossing probability based on the updated feature sequence to determine the pedestrian's crossing intention.

[0009] In one embodiment of the present invention, in step S3, the frame pair fusion unit is a scalar gated fusion unit, which fuses the feature information from the previous frame into the feature information of the current frame to obtain an updated feature sequence that fuses multimodal interaction and enhances temporal coupling. The method is as follows: generate gate coefficients based on the features of the current frame and the features of the previous frame, and use the gate coefficients to perform weighted fusion of the information of the previous frame and the information of the current frame. After fusion, residual connection and normalization are performed.

[0010] In one embodiment of the present invention, in step S3, the frame pair fusion unit is a channel conditional modulation fusion unit, which fuses the feature information from the previous frame into the feature information of the current frame to obtain an updated feature sequence that fuses multimodal interaction and enhances temporal coupling. The method is as follows: generating a channel-wise scaling coefficient and a channel-wise bias coefficient, performing a channel-wise weighted combination of the features of the current frame and the features of the previous frame, and performing normalization on the fusion result.

[0011] In one embodiment of the present invention, in step S3, the frame pair fusion unit is a shift depth separable fusion unit, which fuses the feature information from the previous frame into the feature information of the current frame to obtain an updated feature sequence that fuses multimodal interaction and enhances temporal coupling. The method is as follows: after concatenating the features of the previous frame and the features of the current frame in the channel dimension, perform channel-wise convolution to characterize the differences between adjacent frames, perform point-wise convolution to perform channel mixing, and normalize after adding the residual of the current frame features.

[0012] In one embodiment of the present invention, in step S2, the method of encoding the multiple modal features of each frame into an embedded representation in a unified dimensional space is as follows: each modal feature is mapped to the same feature dimension through a projection network or a linear mapping layer.

[0013] In one embodiment of the present invention, in step S4, the method for predicting the pedestrian crossing probability based on the updated feature sequence is as follows: extract the features corresponding to the task identifier from the updated feature sequence and input them into a classifier to obtain the pedestrian crossing probability.

[0014] In one embodiment of the present invention, in step S1, the time-series image sequence is a fixed-length observation window, and the last frame of the observation window is located within a preset advance time range before the marked pedestrian crossing event start time.

[0015] In one embodiment of the present invention, in step S1, the multiple modal features used to describe the pedestrian state, behavior and scene include at least: vehicle speed information, target pedestrian bounding box trajectory information, target pedestrian posture key point information, target pedestrian local appearance context information and global scene semantic information.

[0016] In one embodiment of the present invention, the method for obtaining the target pedestrian bounding box trajectory information is as follows: normalizing the bounding box coordinates of each frame; constructing a geometric description vector containing center coordinates, width, height, aspect ratio and area from the normalized coordinates; and calculating the first-order difference of the geometric description vector between adjacent frames to characterize short-term motion changes.

[0017] This invention also provides a multimodal pedestrian crossing intention prediction system based on frame pair coupling, comprising the following modules: The data acquisition and feature extraction module is used to acquire a time-series image sequence containing the target pedestrian and time-synchronized vehicle motion information; and to extract multiple modal features from the time-series image sequence and the vehicle motion information to describe the pedestrian's state, behavior and scene. The feature encoding and serialization module is used to encode the multiple modal features of each frame into an embedded representation in a unified dimensional space, add a task identifier, and obtain the initial feature sequence of each frame. The interactive fusion and temporal coupling module is used to input the initial feature sequence of each frame into the feature interaction network based on the attention mechanism. The feature interaction network fuses and interacts with the embedded representations of different modalities within the same frame through a self-attention layer. Through the frame pair fusion unit embedded in at least one network layer, the feature information from the previous frame is fused into the feature information of the current frame to obtain the updated feature sequence that fuses multimodal interaction and strengthens temporal coupling. The intent prediction module is used to predict the probability of a target pedestrian crossing the street based on the updated feature sequence, and to determine the pedestrian's crossing intent.

[0018] The technical solution of the present invention has the following advantages compared with the prior art: This invention achieves fine-grained deep collaboration between different information sources by performing attention-based early interaction fusion immediately after multimodal feature encoding, effectively improving the relevance and robustness of feature representation. At the same time, it innovatively embeds frame pair fusion units between network layers, explicitly models and accumulates short-term micro-dynamics between adjacent frames layer by layer, making it more sensitive to subtle changes in pedestrian crossing intentions. This enables more accurate and earlier predictions in complex real-world scenarios, providing more reliable pedestrian intention judgment capabilities for autonomous driving systems. Attached Figure Description

[0019] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0020] Figure 1 This is a flowchart illustrating the multimodal pedestrian crossing intention prediction method based on frame pair coupling in an embodiment of the present invention. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0022] Example 1: like Figure 1 As shown, this embodiment of the invention provides a multimodal pedestrian crossing intention prediction method based on frame pair coupling, including the following steps: S1: Acquire a temporal image sequence containing the target pedestrian and time-synchronized vehicle motion information; extract multiple modal features from the temporal image sequence and the vehicle motion information to describe the pedestrian's state, behavior, and scene; S2: Encode the multiple modal features of each frame into an embedding representation in a unified dimensional space, add a task identifier, and obtain the initial feature sequence of each frame; S3: Input the initial feature sequence of each frame into the feature interaction network based on the attention mechanism. The feature interaction network fuses and interacts the embedded representations of different modalities within the same frame through the self-attention layer. Through the frame pair fusion unit embedded in at least one network layer, the feature information from the previous frame is fused into the feature information of the current frame to obtain the updated feature sequence that fuses multimodal interaction and strengthens temporal coupling. S4: Predict the pedestrian's crossing probability based on the updated feature sequence to determine the pedestrian's crossing intention.

[0023] This invention simultaneously acquires temporal images and vehicle motion information, extracting multimodal features of pedestrian state, behavior, and scene. The multimodal features of each frame are mapped to a unified dimensional space and task identifiers are added to form a frame-level feature sequence. This sequence is input into an attention-based network. This network, on the one hand, achieves deep fusion and interaction of multimodal features within the same frame through self-attention; on the other hand, it continuously fuses information from the previous frame into the current frame through frame-pair fusion units embedded between layers, thereby constructing a feature representation that is rich in multimodal association and strengthens temporal coupling. Based on the enhanced feature sequence, the pedestrian's intention to cross the street is predicted.

[0024] In this embodiment of the invention, in step S1, a time-series image sequence containing the target pedestrian and time-synchronized vehicle motion information are acquired; from the time-series image sequence and the vehicle motion information, multiple modal features for describing the pedestrian's state, behavior and scene are extracted.

[0025] Specifically, this involves acquiring a continuous T-frame forward-view image sequence containing the target pedestrian. Simultaneously, it collects vehicle motion data that strictly corresponds to the time of each frame of the image, mainly including vehicle speed. .

[0026] Here, a series of T frames constitute a fixed-length observation window. The length T of the observation window is a positive integer greater than or equal to 2. In practice, it is usually taken as 4 to 32 frames, preferably 8 to 16 frames. The frame rate can be between 10 and 30 fps, so as to achieve a balance between prediction lead and system real-time requirements.

[0027] The last frame of the observation window is set within a preset lead time range (e.g., 1 to 3 seconds before the start time of the manually labeled pedestrian crossing event) to ensure that the model learns and executes a true advance prediction, reserving decision-making and reaction time for the autonomous driving system.

[0028] Further, in step S2, based on the aforementioned raw data, a rich multimodal input set is constructed for each frame to comprehensively describe the pedestrian's state, behavior, and surrounding scene. These modalities include at least: vehicle speed information, target pedestrian bounding box trajectory information, target pedestrian pose keypoint information, target pedestrian local appearance context information, and global scene semantic information. Vehicle speed information is used to characterize the vehicle's own motion state and is an important clue for judging relative motion and collision risk. For speed information, if it is a discrete interval category, a lookup table embedding method is used to obtain the speed token; if it is a continuous scalar, it is first normalized and then mapped to a speed token through a multilayer perceptron.

[0029] The extraction and encoding of target pedestrian bounding box trajectory information aims to accurately capture the position and scale changes of pedestrians, and to process the pixel coordinates of the target pedestrian bounding box detected in each frame. Normalize the vector to the [0,1] interval. Then, construct a six-dimensional geometric description vector. ,in The coordinates of the center point of the bounding box. and Width and height are respectively. Aspect ratio, Let be the area. To further characterize the short-term motion changes of pedestrians, the first-order differences of the center coordinates, width, and height in this geometric description vector between adjacent frames are calculated.

[0030] The original six-dimensional geometric features and the four-dimensional inter-frame difference features together constitute a ten-dimensional temporal descriptor. The time-series descriptor is transmitted through a projection network. Mapped to a unified D-dimensional feature space, bounding box tokens are formed, where, Represents the bounding box feature projection network; Represents a ten-dimensional temporal description subspace; This indicates the unified token embedding space; The token feature dimension; the projection network may consist of at least one fully connected layer and a nonlinear activation function.

[0031] The target pedestrian's pose keypoint information provides prior knowledge of human movement. For each frame, a preset number (e.g., 17) of 2D keypoint coordinates are obtained through a pose estimation algorithm, and these coordinates are unfolded sequentially to form a pose vector. ,in, Indicates time The pedestrian pose key point vector; Indicates the first Two-dimensional coordinates of key points of each joint; .

[0032] This pose vector is then transformed into a D-dimensional pose token through a linear mapping layer, i.e. ,in This is the weight matrix. This is the bias vector.

[0033] In this embodiment, the local appearance context information of the target pedestrian is used to provide visual details of the pedestrian's neighboring areas, such as clothing texture and handheld items. The method is as follows: cropping the original image with the target pedestrian's bounding box as the center to obtain the local image. The local image is divided into P×P blocks. Each image block is flattened and mapped to a D-dimensional vector through a learnable linear projection layer, thus obtaining a local appearance Patch Token sequence. Where P is a positive integer greater than or equal to 8 and N is the number of patches. The bounding box of the local clipping region can be appropriately expanded to include key environmental cues such as road edges and zebra crossing boundaries.

[0034] Global scene semantic information is used to provide high-level semantic cues such as road structure, sidewalks, and zebra crossings. This information is typically presented as a semantic segmentation map or semantic mask for each frame. It exists in the form of, in which and These represent the height and width of the semantic graph, respectively. During encoding, the semantic graph is first represented using channelization (e.g., one-hot encoding), resulting in C semantic channels (C being the number of semantic categories). Then, the semantic graph is divided into non-overlapping patches of size P×P. For each patch, average pooling is performed on its C-dimensional pixel semantic distribution to obtain a C-dimensional semantic distribution histogram vector, which statistically represents the proportion of each semantic category within that region. Finally, each semantic distribution vector is projected onto a D-dimensional space through a linear layer, forming a global semantic patch token sequence. Through the above steps, a comprehensive and fine-grained multimodal feature representation is constructed for each frame, laying a solid foundation for subsequent deep fusion and intent reasoning.

[0035] In the technical field described in the embodiments of this invention, particularly in deep learning models based on attention mechanisms, a token refers to a basic representation unit that maps input data (whether discrete symbols, continuous numerical vectors, or image patches) to a unified, dense vector space (i.e., embedding space). Each token is a real-valued vector of a fixed dimension (e.g., D-dimensional) that encapsulates the semantic information of the original input fragment or features.

[0036] In this embodiment of the invention, the original data of different modalities and structures, such as vehicle speed, bounding box trajectory, and pose keypoints, are converted into their corresponding token representations (e.g., speed token, bounding box token, pose token, etc.) through the aforementioned projection network or linear mapping layer. For image patches, a Patch Token sequence is generated. The core purpose of this tokenization process is to convert heterogeneous multi-source data into homogeneous, dimension-aligned vector sequences, thereby laying the foundation for unified, self-attention-based early cross-modal interaction and fusion within the Transformer architecture.

[0037] The sequence of classification tokens, velocity tokens, bounding box tokens, pose tokens, local appearance patch tokens, and global semantic patch tokens are concatenated in a preset order to form a frame-level token sequence for each frame, and positional encoding is added to distinguish between temporal order and modal type.

[0038] Furthermore, in step S3, an attention-based feature interaction network is used to achieve early fusion of intra-frame multimodal data and temporal coupling between adjacent frames.

[0039] In this embodiment of the invention, the feature interaction network based on the attention mechanism takes stacked Transformer blocks as the main architecture. Within the token sequence of the same frame, the self-attention mechanism is used to realize early and fine-grained interaction fusion between tokens of different modalities. After at least one Transformer block or between layers, a frame pair fusion unit is embedded to explicitly model and fuse the update information from the previous frame into the current frame, thereby realizing enhanced temporal coupling.

[0040] Specifically, the feature interaction network performs early fusion interactions. The constructed, position-encoded sequence of tokens from each frame is input into a network consisting of L stacked Transformer blocks (L is a positive integer greater than or equal to 1). Each Transformer block contains a multi-head self-attention layer and a feedforward network layer, and is typically stabilized by residual connections and layer normalization.

[0041] For a frame-level token sequence input to a Transformer block Its processing follows the standard Transformer formula that combines multi-head attention with feedforward, where... Number of tokens; For the embedded dimension.

[0042] The query is obtained through linear projection. ),key( ),value( )vector: , , , in, , , Learnable projection weights.

[0043] The output of multi-head self-attention is calculated as follows: , in, It's the number of heads that attract attention. It is a single-head dimension.

[0044] Bullish attention will The outputs of each size are spliced ​​together and then projected again: , in, To output the projection matrix; This is for splicing operations.

[0045] This process enables all tokens, including velocity, bounding box geometry, pose, local appearance, and global semantics, to interact freely within a unified semantic space, capturing fine-grained relationships between them, such as the synchronization of specific pose changes and movement directions, thus achieving true early-stage multimodal fusion.

[0046] After further processing by a feedforward network typically consisting of two layers of linear transformation and nonlinear activation, supplemented by residual connections and layer normalization, the final updated token representation is output: , in, For layer normalization.

[0047] Furthermore, Through a feedforward network: , in, It is a non-linear activation function (e.g., GELU or ReLU); , , , These are the parameters of the feedforward network; Output the token sequence for the Transformer block.

[0048] Finally, residual connection and layer normalization are applied again to obtain the output of the Transformer block: .

[0049] Through this process, the sequence of velocity tokens, bounding box tokens, pose tokens, local appearance and global semantics within each frame can fully interact and fuse information in a unified embedding space, updating the representation of each token. In particular, the classification token (task identifier) ​​gathers the global context information of the frame.

[0050] The above mechanism is a manifestation of the early fusion design in the embodiments of the present invention. After the feature encoding is completed, the cross-modal interaction process is immediately started, so that heterogeneous tokens such as speed, trajectory, posture, appearance and semantics can perform token-level fine-grained interaction and information integration in a unified semantic space. This fundamentally avoids the inherent limitation of traditional late fusion strategies, which cause the compression or loss of cross-modal fine-grained correlation information due to the independent deep encoding of each modality.

[0051] Furthermore, in this embodiment, after at least one Transformer block or between layers, the present invention introduces a frame-pair feature fusion (FPFF) operation, which is implemented through a frame-pair fusion unit.

[0052] In this embodiment, the Token updated by the Transformer block in the previous frame is represented as: The token before frame fusion is represented as Then the fusion satisfies the following equation: , in, This indicates the frame pair feature fusion module; This indicates that the Token was updated in the previous frame; the right side of the equals sign... This represents the token representation before fusion of the current frame; the left side of the equals sign. This represents the token of the current frame after fusion. .

[0053] In this embodiment, FPFF is repeatedly executed between multiple Transformer blocks, allowing short-term dynamic information from adjacent frames to propagate and accumulate layer by layer, thus making it more sensitive to micro-dynamics such as pedestrian hesitation and subtle changes in posture before starting.

[0054] Specifically, there are three preferred implementation methods for the frame pair fusion unit.

[0055] The first type is scalar gating fusion, which uses a learnable gating mechanism to adaptively control the fusion ratio of information from the previous frame and the current frame.

[0056] Generate gating coefficients It satisfies the following formula: , in, Indicates the gating coefficient; For the Sigmoid function; Indicates feature splicing; These are weight parameters; This is the bias parameter. The size of the value reflects the importance of the new information in the current frame in the fusion process.

[0057] Fusion is performed based on gating, satisfying the following formula: , in, To integrate intermediate features; and Used to control the contribution ratio of information from the previous frame and the current frame; after fusion, residual connection and layer normalization can be further used to obtain the final output.

[0058] Scalar gating fusion has a simple structure, few parameters, and high computational efficiency, making it suitable for vehicle platforms that are sensitive to computational resources.

[0059] The second type is channel conditional modulation fusion, which learns independent fusion weights for each channel (i.e. dimension) of the feature vector to achieve more refined channel-level information control.

[0060] right Perform global average pooling and generate channel-wise coefficients. It satisfies the following formula: , in, and These represent the channel-level scaling factor and the channel-level weighting factor, respectively. This represents the global average pooling operation, used for statistical aggregation of concatenated features; Represents a linear mapping layer; The features of the spliced ​​frame pair; Consistent with the Token dimension, all are Dimensional vector.

[0061] Furthermore, channel-by-channel fusion is performed to satisfy the following equation: , in, This represents the intermediate features obtained by channel modulation fusion; This represents the channel scaling factor for the current frame's token; This represents the channel weight coefficient for updating the token in the previous frame; symbol This indicates element-wise multiplication; This represents the token representation before the current frame is merged. This indicates that the Token representation was updated in the previous frame; after fusion, layer normalization can be further applied to obtain the final output Token representation.

[0062] Channel conditional modulation fusion enhances the model's selective fusion capability in the channel dimension.

[0063] The third type is shift depth-separable fusion, inspired by depth-separable convolution, used to explicitly model the difference patterns between tokens in adjacent frames.

[0064] The intermediate features are obtained by concatenating the features of the two frames along the channel dimension and then performing a channel-wise convolution with a kernel size of 2. It satisfies the following formula: , in, This indicates that the spliced ​​features are convolved channel by channel at the 1st... Output on each channel; and These represent the parts of the spliced ​​features corresponding to the previous frame and the current frame, respectively, at the [number]th [frame]. The components of each channel; and This indicates that the channel-wise convolution with a kernel size of 2 is in the th... Convolution weights on each channel; Indicates the range of channel indexes; It comes from splicing the features of two frames in the channel dimension.

[0065] After pointwise convolution, channel blending and residual normalization are performed to satisfy the following equation: , in, This represents the intermediate features of the output from the channel-wise convolution; The weight matrix represents the pointwise convolution, used to blend channels and reduce the dimension from... Mapped to ; This represents the fusion increment after channel mixing; the right side of the equals sign... This represents the token of the current frame before fusion; the symbol "+" indicates the addition of residuals. Indicates layer normalization operation; the left side of the equals sign This represents the normalized fused output token.

[0066] The shift depth separable fusion enhances the ability to depict subtle changes between frames (such as phase shift and amplitude change) through a two-step operation of first modeling the differences channel by channel and then performing channel mixing. It is especially beneficial for capturing the micro-dynamics of a pedestrian just before their intention is made.

[0067] The scalar gating fusion, channel conditional modulation fusion, and shift depth separable fusion described above constitute three preferred implementations of FPFF. In practical applications, any one of these methods or a combination thereof can be selected for deployment based on the different application scenarios' emphasis on computational efficiency, fusion precision, or dynamic sensitivity.

[0068] Early deep interaction of multimodal tokens is achieved through Transformer blocks, and explicit, structured coupling between adjacent frames is established between layers through embedded frame pair fusion units. The three optional modes of FPFF provide flexible choices for different application scenarios. The synergy between early fusion and frame pair temporal coupling ensures that it can both understand rich cross-modal associations and keenly capture short-term dynamic sequence patterns that indicate street crossing intentions.

[0069] Furthermore, in step S4, the pedestrian's crossing probability is predicted based on the updated feature sequence to determine the pedestrian's crossing intention.

[0070] Specifically, feature vectors corresponding to preset task identifiers (i.e., classification tokens) are extracted from the updated feature sequence. This classification token is located at the beginning or a specified position of the token sequence in each frame. During the early fusion and frame pair coupling process of the Transformer, it has gathered all modal interaction information within that frame, including velocity, bounding box trajectory, pose, local appearance, and global semantics, as well as temporal dynamics from historical frames. Therefore, it carries a comprehensive contextual representation for intent inference. This classification token is input into a lightweight classifier to map and obtain the predicted probability of the target pedestrian's intent to cross the street at the current moment. The classifier is typically a classification head composed of a multilayer perceptron (MLP), and its process can be formally represented as follows: , in, This represents the feature sequence extracted from the output of the last Transformer block, corresponding to time step [time]. Token classification; This represents a multilayer perceptron consisting of one or two fully connected layers and a nonlinear activation function. This represents the Sigmoid activation function, used to map the network output to... The interval is used to obtain the probability of crossing the street. .

[0071] To obtain the final binary intention judgment ("cross the street" or "do not cross the street"), the predicted probabilities are... With an adjustable decision threshold (usually meets) Compare: If If the pedestrian has the intention to "cross the street", then it is determined that the pedestrian intends to "cross the street"; otherwise, it is determined that the pedestrian "does not cross the street".

[0072] During model training, supervised learning is used to optimize the overall network parameters. Since pedestrian crossing intentions in real-world datasets often exhibit class imbalance (positive "crossing" samples are far fewer than negative "not crossing" samples), to improve the model's ability to identify minority classes (i.e., recall) and mitigate training bias caused by sample imbalance, the weighted binary cross-entropy loss function is preferentially chosen. This loss function amplifies the contribution of positive or difficult samples to gradient updates by assigning different weights to positive and negative samples. The specific form of the loss function is as follows: , in, This represents the batch sample size. It is the first The true intention to cross the street is labeled for each sample (1 indicates crossing the street, 0 indicates not crossing the street). It is the predicted probability corresponding to the model; and These are the loss weights for the positive and negative classes, respectively. The weights are typically set based on the proportion of positive and negative samples in the training set, or their reciprocals, to balance the influence of each class. For training stability, the Adam optimizer is widely used, coupled with appropriate learning rate decay strategies (such as cosine annealing with warm restart). To suppress overfitting, regularization techniques such as Dropout and Stochastic Depth are commonly introduced during network training; the Dropout rate and Stochastic Depth rate are typically set between 0.1 and 0.2.

[0073] To improve the temporal stability of the model's output in actual deployment and reduce prediction jitter caused by single-frame noise, the inference phase can be performed on several consecutive frames (e.g., consecutive frames). The predicted probability of a frame is smoothed over time by using a moving average, taking the current frame and the previous frames. The average of the frame prediction probabilities is used as the final probability output for the current frame, and then compared with a threshold. Compare.

[0074] Another method is to use continuous Frame consistency is determined only when consecutive frames are... More than half of the frames have a raw prediction probability exceeding a threshold. Only then is it finally determined that there was an intention to "cross the street" during that period. These post-processing techniques can effectively smooth the output and reduce the false alarm rate.

[0075] In this embodiment of the invention, not only is the prediction accuracy and robustness in complex scenarios effectively improved through the collaborative design of multimodal early fusion and FPFF, but also flexible and scalable technical options are provided for different application scenarios by offering diverse FPFF implementation schemes.

[0076] Example 2: Based on the same inventive concept as Embodiment 1, the present invention also provides a multimodal pedestrian crossing intention prediction system based on frame pair coupling, used to implement the steps of the multimodal pedestrian crossing intention prediction method based on frame pair coupling described in Embodiment 1, including the following modules: The data acquisition and feature extraction module is used to acquire a time-series image sequence containing the target pedestrian and time-synchronized vehicle motion information; and to extract multiple modal features from the time-series image sequence and the vehicle motion information to describe the pedestrian's state, behavior and scene. The feature encoding and serialization module is used to encode the multiple modal features of each frame into an embedded representation in a unified dimensional space, add a task identifier, and obtain the initial feature sequence of each frame. The interactive fusion and temporal coupling module is used to input the initial feature sequence of each frame into the feature interaction network based on the attention mechanism. The feature interaction network performs early fusion and interaction on the embedded representations of different modalities within the same frame through the self-attention layer. Through the frame pair fusion unit embedded in at least one network layer, the feature information from the previous frame is fused into the feature information of the current frame to obtain the updated feature sequence that fuses multimodal interaction and strengthens temporal coupling. The intent prediction module is used to predict the probability of a target pedestrian crossing the street based on the updated feature sequence, and to determine the pedestrian's crossing intent.

[0077] The data acquisition and feature extraction module, feature encoding and serialization module, interactive fusion and temporal coupling module, and intent prediction module of the multimodal pedestrian crossing intent prediction system based on frame pair coupling proposed in this embodiment are respectively used to implement steps S1, S2, S3, and S4 in the multimodal pedestrian crossing intent prediction method based on frame pair coupling in Embodiment 1. To avoid redundancy, they will not be described in detail here.

[0078] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0079] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0080] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0081] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0082] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A multimodal pedestrian crossing intention prediction method based on frame pair coupling, characterized in that, Includes the following steps: S1: Acquire a temporal image sequence containing the target pedestrian and time-synchronized vehicle motion information; extract multiple modal features from the temporal image sequence and the vehicle motion information to describe the pedestrian's state, behavior, and scene; S2: Encode the multiple modal features of each frame into an embedding representation in a unified dimensional space, add a task identifier, and obtain the initial feature sequence of each frame; S3: Input the initial feature sequence of each frame into the feature interaction network based on the attention mechanism. The feature interaction network fuses and interacts the embedded representations of different modalities within the same frame through the self-attention layer. Through the frame pair fusion unit embedded in at least one network layer, the feature information from the previous frame is fused into the feature information of the current frame to obtain the updated feature sequence that fuses multimodal interaction and strengthens temporal coupling. S4: Predict the pedestrian's crossing probability based on the updated feature sequence to determine the pedestrian's crossing intention.

2. The multimodal pedestrian crossing intention prediction method based on frame pair coupling according to claim 1, characterized in that: In step S3, the frame pair fusion unit is a scalar gated fusion unit, which fuses the feature information from the previous frame into the feature information of the current frame to obtain the updated feature sequence that fuses multimodal interaction and enhances temporal coupling. The method is as follows: generate gate coefficients based on the features of the current frame and the features of the previous frame, and use the gate coefficients to perform weighted fusion of the information of the previous frame and the information of the current frame. After fusion, residual connection and normalization are performed.

3. The multimodal pedestrian crossing intention prediction method based on frame pair coupling according to claim 1, characterized in that: In step S3, the frame pair fusion unit is a channel conditional modulation fusion unit, which fuses the feature information from the previous frame into the feature information of the current frame to obtain the updated feature sequence that fuses multimodal interaction and strengthens temporal coupling. The method is as follows: generate channel-wise scaling coefficients and channel-wise bias coefficients, perform channel-wise weighted combination of the features of the current frame and the features of the previous frame, and perform normalization on the fusion result.

4. The multimodal pedestrian crossing intention prediction method based on frame pair coupling according to claim 1, characterized in that: In step S3, the frame pair fusion unit is a shift depth separable fusion unit, which fuses the feature information from the previous frame into the feature information of the current frame to obtain the updated feature sequence that fuses multimodal interaction and enhances temporal coupling. The method is as follows: after concatenating the features of the previous frame and the features of the current frame in the channel dimension, perform channel-wise convolution to characterize the differences between adjacent frames, perform point-wise convolution to perform channel mixing, and normalize after adding with the feature residual of the current frame.

5. The multimodal pedestrian crossing intention prediction method based on frame pair coupling according to claim 1, characterized in that: In step S2, the method for encoding the multiple modal features of each frame into an embedding representation in a unified dimensional space is as follows: each modal feature is mapped to the same feature dimension through a projection network or a linear mapping layer.

6. The multimodal pedestrian crossing intention prediction method based on frame pair coupling according to claim 1, characterized in that: In step S4, the method for predicting the pedestrian crossing probability based on the updated feature sequence is as follows: extract the features corresponding to the task identifier from the updated feature sequence and input them into the classifier to obtain the pedestrian crossing probability.

7. The multimodal pedestrian crossing intention prediction method based on frame pair coupling according to claim 1, characterized in that: In step S1, the time-series image sequence is a fixed-length observation window, and the last frame of the observation window is located within a preset advance time range before the marked start time of the pedestrian crossing event.

8. The multimodal pedestrian crossing intention prediction method based on frame pair coupling according to claim 1, characterized in that: In step S1, the multiple modal features used to describe the pedestrian's state, behavior, and scene include at least: vehicle speed information, target pedestrian bounding box trajectory information, target pedestrian pose key point information, target pedestrian local appearance context information, and global scene semantic information.

9. The multimodal pedestrian crossing intention prediction method based on frame pair coupling according to claim 8, characterized in that: The method for obtaining the target pedestrian bounding box trajectory information is as follows: normalize the bounding box coordinates of each frame; construct a geometric description vector containing center coordinates, width, height, aspect ratio and area from the normalized coordinates; calculate the first-order difference of the geometric description vector between adjacent frames to characterize short-term motion changes.

10. A multimodal pedestrian crossing intention prediction system based on frame pair coupling, characterized in that, Includes the following modules: The data acquisition and feature extraction module is used to acquire a time-series image sequence containing the target pedestrian and time-synchronized vehicle motion information; and to extract multiple modal features from the time-series image sequence and the vehicle motion information to describe the pedestrian's state, behavior and scene. The feature encoding and serialization module is used to encode the multiple modal features of each frame into an embedded representation in a unified dimensional space, add a task identifier, and obtain the initial feature sequence of each frame. The interactive fusion and temporal coupling module is used to input the initial feature sequence of each frame into the feature interaction network based on the attention mechanism. The feature interaction network fuses and interacts with the embedded representations of different modalities within the same frame through a self-attention layer. Through the frame pair fusion unit embedded in at least one network layer, the feature information from the previous frame is fused into the feature information of the current frame to obtain the updated feature sequence that fuses multimodal interaction and strengthens temporal coupling. The intent prediction module is used to predict the cross-street probability of a target pedestrian based on the updated feature sequence and to determine the pedestrian's cross-street intent.

Images