Rider intention recognition method considering deep space-time fusion
By employing a deep spatiotemporal fusion-based cyclist intent recognition method, and utilizing a lightweight network and a Transformer encoder with a sparse attention mechanism, the accuracy and real-time performance issues of cyclist intent recognition in complex traffic scenarios are addressed, achieving highly efficient intent recognition results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-22
- Publication Date
- 2026-03-27
AI Technical Summary
Existing technologies for cyclist intent recognition in complex traffic scenarios suffer from problems such as difficulty in feature extraction, insufficient fusion of multimodal information, redundancy in temporal modeling leading to low recognition accuracy, poor real-time performance, and insufficient robustness.
We employ lightweight convolutional networks and convolutional feedforward network modules to extract deep visual features, combine a Transformer encoder with a sparse attention mechanism to perform cross-modal feature fusion and long-range temporal dependency modeling, compress features through bottleneck structures, and use a temporal attention module for intent classification.
It significantly improves the accuracy and stability of rider intent recognition, reduces the number of model parameters, and meets the real-time requirements of intelligent driving systems.
Smart Images
Figure CN121743942A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of autonomous driving, and in particular to a cyclist intention recognition method considering deep spatio-temporal fusion. BACKGROUND
[0002] In intelligent driving systems, accurately recognizing and understanding the behavior intention of traffic participants is one of the key technologies to ensure driving safety and achieve human-like decision-making. Cyclists, as an important type of traffic participants, have characteristics such as fast movement, strong maneuverability, and high behavior uncertainty. Their intentions are of great significance to the active avoidance, cooperative driving, and risk prediction of intelligent driving vehicles. However, due to the complexity of traffic scenes, the dynamic variability of cycling behavior, and the limitations of perception conditions, achieving high-precision and high-robustness cyclist intention recognition still faces many challenges.
[0003] Currently, research on cyclist intention recognition mainly falls into two categories: methods based on visual features and methods based on motion trajectories. Methods based on visual features mainly use visual information such as the cyclist's posture, gestures, or head orientation to infer intentions. For example, by using a pose estimation algorithm to extract human key points, and then combining a time series modeling network to recognize the cyclist's turning or lane-changing intention. However, this method is easily affected by factors such as changes in lighting, target occlusion, and shooting distance in actual applications. The quality of visual feature extraction significantly decreases in low-resolution or partially occluded situations, and the recognition performance is unstable. In addition, pure vision-based methods often ignore the motion trend information of cyclists, making it difficult to effectively predict medium and long-term intentions. Methods based on motion trajectories mainly use the cyclist's historical position information to infer intentions through trajectory prediction models. Although this method can better reflect the macro motion law of cyclists, it lacks the ability to perceive fine-grained actions such as hand signals and head rotation, and its prediction reliability is limited when observing for a short period or when trajectory noise is large. More importantly, single trajectory information cannot fully represent the behavior semantics of cyclists in complex interactive scenarios, making it difficult to cope with sudden changes in intentions. Although existing research has attempted to integrate visual and trajectory information to improve recognition performance, existing fusion methods mostly use simple feature concatenation or late decision fusion strategies, and the information complementarity has not been fully explored. In addition, existing methods often use network structures with high computational complexity (such as dense 3D convolution), resulting in large model parameter quantities and slow inference speed. Therefore, a cyclist intention recognition method that can deeply integrate multi-modal information, efficiently model long-term temporal dependencies, and balance recognition accuracy and computational efficiency is an important research content. SUMMARY
[0004] To solve the problems of low recognition accuracy, poor real-time performance and insufficient robustness caused by difficulty in feature extraction, insufficient multi-modal information fusion and redundant time sequence modeling in existing technologies in complex traffic scenarios, the present application proposes a cyclist intention recognition method considering deep spatio-temporal fusion, which can not only significantly improve the accuracy and efficiency of intention recognition, but also enhance the adaptability to complex scenes such as occlusion and light change, providing more reliable and real-time behavior understanding basis for intelligent driving systems.
[0005] To achieve the above-mentioned purpose, the basic idea of the present application is: first, the historical visual image sequence and spatio-temporal trajectory sequence of the target cyclist are obtained, the deep visual features are extracted through the lightweight convolutional network and convolutional feedforward network module respectively, and the trajectory information is embedded into the high-dimensional feature space through the fully connected layer; second, the extracted visual features and trajectory features are aligned and spliced, input into the lightweight Transformer encoder constructed based on the sparse attention mechanism, realizing deep fusion of cross-modal features and long-range time sequence dependence modeling; then, the fused features are compressed and aggregated by using the bottleneck structure, and the key frame information is weighted by the time attention module; finally, the intention category of the cyclist is output through the classifier. The internal correlation between visual appearance and motion trajectory is deeply mined, and time sequence reasoning is realized by using an efficient structure.
[0006] The technical scheme of the present application is as follows: a cyclist intention recognition method considering deep spatio-temporal fusion, comprising the following steps: A、Obtaining multi-modal time sequence data of the target cyclist A1、Obtaining local visual image sequence and spatio-temporal trajectory information of the cyclist The motion trend of the cyclist is described by the historical sequence of the boundary box of the implicit reaction intention, and the local visual information further reveals the intention of the cyclist. Therefore, the recognition variables of the cyclist intention recognition model are: cyclist trajectory feature and visual feature ; wherein, , is the horizontal coordinate of the top-left corner point of the boundary box in the t th frame, is the vertical coordinate of the top-left corner point of the boundary box in the t th frame, is the horizontal coordinate of the bottom-right corner point of the boundary box in the t th frame, is the vertical coordinate of the bottom-right corner point of the boundary box in the t th frame. represents the local visual information of the t th frame, T represents the length of the observed time sequence.
[0007] A2, Data preprocessing and augmentation Standardization of the extracted local image sequence: first, each bounding box is enlarged by 1 times in each direction to contain more rich context information; then, all images are adjusted to a uniform size of 224x224 pixels; finally, normalization processing is performed, that is, the mean [0.485, 0.456, 0.406] is subtracted and the standard deviation [0.229, 0.224, 0.225] is divided. Time alignment is performed on the trajectory sequence to ensure that the visual image and the trajectory data strictly correspond in time sequence.
[0008] B, Extraction and embedding of visual and trajectory features of the cyclist B1, Extraction of visual features The preprocessed image sequence is input into a lightweight convolutional neural network MobileNetV2 for feature extraction. To further enhance the expression ability of local features and reduce the number of model parameters, a convolutional feedforward network module is input after the MobileNetV2 network processing. The convolutional feedforward network module finally outputs the visual feature tensor , C , H , W , ,
[0009] B2, Embedding of spatio-temporal features The motion trend of the cyclist is described by the historical sequence of the boundary box representing the implicit reaction intention. Considering that the original boundary box data has a low dimension and limited expression, and the local visual features after feature extraction have a high dimension, the trajectory feature B and the visual feature V are embedded into the same feature space through a fully connected layer for unified processing and feature fusion, respectively obtaining , , , , , ,
[0010] , MLP , ,
[0011] C, Cross-modal feature fusion based on sparse attention mechanism C1, Modeling of time sequence and modal interaction The inherent time correlation of visual features and spatial features and the cross-correlation between different features are captured by a sparse attention encoder. The training weights of the two mappings B and V are first processed by a fully connected layer to obtain two class labels and Add it to the spatial features according to the time axis. and visual features Previously, the initial fusion feature sequence was obtained as follows: The formula is as follows:
[0012] C2. Initial fusion feature sequence Add sinusoidal positional encoding. Sinusoidal positional encoding preserves positional information, ensuring permutation invariance for subsequent attention operations. Its definition is as follows:
[0013] In the formula, PE A position vector generated for each time frame. This indicates that the data at a certain time has a length of T The index position in the input. d For the hidden layer dimension of the model, k This is the dimension index of the position vector.
[0014] C3. Input the feature sequence that incorporates location information into a... L In a Transformer encoder composed of stacked identical layers, each layer contains an attention module and a feedforward network. The attention module employs a sparse attention mechanism, which reduces computational complexity by sparsifying the standard attention matrix. Reduced to approximately It can efficiently model the dependencies of long sequences.
[0015] Specifically, the sparse attention mechanism takes a feature sequence incorporating location information as input and generates a query, key, and value vector for each attention head h. This multidimensional vector is represented as follows:
[0016] In the formula, It is the first Query vector in the header dimension It is the first Keyword vectors in the head dimension It is the first The value vector in the head dimension. It is a learnable parameter matrix for generating query vectors. It is a learnable parameter matrix that generates the key vectors. It is a learnable parameter matrix that generates value vectors.
[0017] Next, the multi-dimensional vectors are aggregated to generate multi-dimensional output features. The calculation formula is as follows:
[0018] In the formula, spare attention(.) is the sparse attention layer.
[0019] C4. After completing the modeling of visual and spatial features, obtain the aggregated... B and V A vector of all time information Mark it and the learnable class. Integrating to obtain Then, Transformer encoding is used to mine the relationships between different modalities. Then use replace Class tags in Finally, layer normalization is performed through fully connected layers and residual connections.
[0020] C5. Feature compression and aggregation.
[0021] To reduce redundant information and lower parameters in the two fused features, a bottleneck structure is used to compress the features, as shown below:
[0022]
[0023] in, , This represents the compressed features. Following this method, B and V The two features are segmented to obtain:
[0024]
[0025] Finally, feature aggregation is performed by addition to obtain the final result. ,in, It indicates the first i The feature vectors at each time step.
[0026] D. Time-weighted intent classification D1. To aggregate information along the time dimension, an attention module is used to learn different frames. The importance is shown in the following formula:
[0027] In the formula, h and This represents the learnable parameters. The normalized attention weight of the calculated first frame. i
[0028] D2, obtain a global feature representation for intent classification by weighted summation z
[0029] D3, input the global feature representation focusing on key timing information into a classifier for intent discrimination. z Specifically, the global feature representation is mapped to a dimensional space with the same number of candidate intent categories through a fully connected layer, and then normalized through a Softmax function to obtain the probability belonging to each intent category. z
[0030] Compared with the prior art, the beneficial effects of the present application are as follows: 1. The present application can extract the deep visual features and high-order trajectory features of the rider in parallel, and realize the deep fusion of cross-modal features by using the Transformer encoder based on the sparse attention mechanism, effectively capturing the long-term temporal dependence and inter-modal interaction, and significantly improving the accuracy of intent recognition, especially the recognition stability in complex traffic scenes.
[0031] 2. The present application can greatly reduce the model parameter amount while maintaining high recognition accuracy through lightweight design (such as convolutional feedforward network module and sparse attention mechanism), achieving a balance between model efficiency and performance, and meeting the requirements of real-time intent recognition of intelligent driving systems. BRIEF DESCRIPTION OF DRAWINGS
[0032] The present application has a total of Figure 4 Zhang, wherein: Figure 1 is a flowchart of the present application.
[0033] Figure 2 is a convolutional feedforward network module diagram of the present application.
[0034] Figure 3 is a spatiotemporal feature deep fusion diagram of the present application.
[0035] Figure 4 is a Transformer encoding diagram of the present application DETAILED DESCRIPTION The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand the advantages of the present invention from the content disclosed in this specification. The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. To provide a deep understanding of the present invention, the following description will include many specific details. Furthermore, to avoid confusion or obscuring the focus of the present invention, some specific details have been omitted in the description.
[0036] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
[0037] like Figure 1 As shown, a cyclist intent recognition method considering deep spatiotemporal fusion includes the following steps: A. Obtain multimodal time-series data of the target cyclist. A1. A cyclist's movement trends can be described by the historical sequence of bounding boxes implicitly reflecting their intentions. Furthermore, local visual information such as changes in gestures and head turning can further reveal the cyclist's intentions. Therefore, this invention proposes identifying the cyclist's intentions in the interaction area based on two types of basic information. Thus, the model's identification variables are: the cyclist's trajectory features... and visual features .in, , It is the first t The x-coordinate of the top-left corner of the frame bounding box. It is the first t The ordinate of the top-left corner of the frame bounding box. It is the first t The x-coordinate of the bottom right corner of the frame bounding box. It is the first t The ordinate of the bottom right corner of the frame bounding box. It indicates the first t Local visual information of a frame, T This indicates the length of the observed time series.
[0038] A2, Standardization processing of the extracted local image sequence: first, each bounding box is expanded by 1 times in each direction to contain more rich context information; then, all images are adjusted to a uniform size of 224x224 pixels; finally, normalization processing is performed, that is, the mean [0.485, 0.456, 0.406] is subtracted and the standard deviation [0.229, 0.224, 0.225] is divided. The trajectory sequence is time-aligned to ensure that the visual image and the trajectory data strictly correspond in time sequence.
[0039] B, Extraction and embedding of the cyclist's visual and trajectory features B1, Visual feature extraction The preprocessed image sequence is input into a lightweight convolutional neural network MobileNetV2 for feature extraction. To further enhance the expression ability of local features and reduce the model parameter quantity, after processing by the MobileNetV2 network, it is input into a convolutional feedforward network module. The convolutional feedforward network module is as shown in Figure 2 , which performs point-by-point dimensionality increase and spatial information fusion on the input features through convolution operation with a step of 1, and finally outputs the visual feature tensor , where C , H , W are the channel number, height and width, respectively.
[0040] B2, The motion trend of the cyclist can be described by the historical sequence of the boundary box of the implicit reaction intention. Considering that the original boundary box data has a low dimension and limited expressible features, and after feature extraction, the local visual features have a high dimension, therefore, the trajectory feature B and the visual feature V are embedded into the same feature space through a fully connected layer for unified processing and feature fusion to obtain , , and represent the embedding dimensions.
[0041]
[0042] wherein, MLP represents a fully connected layer.
[0043] C, Cross-modal feature fusion based on sparse attention mechanism As shown in Figure 3 , the visual features and spatial features of the cyclist are deeply spatio-temporally fused, and the specific fusion steps are as follows.
[0044] C1, Two trainable weights that can map B and V are processed through a fully connected layer to obtain two class labels and adding them to the spatial features along the time axis and visual features The resulting feature map .
[0045]
[0046] C2, the initial fused feature sequence Add sinusoidal position encoding, which preserves the positional information and ensures the permutation invariance of the subsequent attention operation, which is defined as follows.
[0047]
[0048] where PE is the position vector generated for each time frame, denotes the index position of the data at a certain time in the input with length T , d is the dimension of the model hidden layer, d and dpos is the dimension index of the position vector. k
[0049] C3, the feature sequence fused with position information is input into a Transformer encoder stacked by L the same layers, as shown in Figure 4 Each layer of the Transformer encoder contains an attention module and a feedforward network. The attention module adopts a sparse attention mechanism, which reduces the computational complexity from to about efficiently modeling long sequence dependencies.
[0050] Specifically, the sparse attention mechanism takes the feature sequence fused with position information as input, and generates query, key and value vectors for each attention head. The multi-dimensional vector is represented as follows:
[0051] where is the query vector in the th head dimension, is the key vector in the th head dimension, is the value vector in the th head dimension. is the learnable parameter matrix for generating the query vector, is the learnable parameter matrix for generating the key vector, is the learnable parameter matrix for generating the value vector.
[0052] Then, the multi-dimensional vector is aggregated to generate a multi-dimensional output feature , and the calculation formula is as shown below:
[0053] In the formula, spare attention(.) is a sparse attention layer.
[0054] C4, after modeling the visual features and spatial features, a vector B aggregated with all the time information of V is obtained , and the learnable class label is integrated to obtain , and then the relationship between different modalities is mined through Transformer encoding , then is used to replace the class label in , and finally the layer normalization processing is performed through the full connection layer and the residual connection.
[0055] C5, in order to reduce the redundant information that may exist in the two fusion features and reduce the parameters, the application compresses the features through a bottleneck structure, which is represented as follows:
[0056]
[0057] Among them, , represents the compressed feature, , according to this way, the two features of B and V are segmented to obtain:
[0058]
[0059] Finally, the features are aggregated by adding to obtain .
[0060] D, intent classification based on time weighting D1, in order to aggregate information along the time dimension, the importance of different frames is learned through an attention module, and the formula is as follows:
[0061] In the formula, h and represent learnable parameters, For the calculated first i Normalized attention weights for frames.
[0062] D2. Obtain the global feature representation for intent classification by weighted summation. z :
[0063] D3. Obtain a global feature representation focusing on key time-series information. z The input is fed into a classifier for intent determination. Specifically, this is done through a fully connected layer. z Mapped to a dimensional space with the same number of candidate intent categories, and then normalized using the Softmax function, the probability of belonging to each intent category is obtained.
[0064] The basic principles, main features, and advantages of this invention have been described above. Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this invention. Various changes and modifications can be made to this invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed.
Claims
1. A method for cyclist intent recognition considering deep spatiotemporal fusion, characterized in that: Includes the following steps: A. Obtain multimodal time-series data of the target cyclist; B. Extract and embed the cyclist's visual and trajectory features; C. Cross-modal feature fusion based on sparse attention mechanism; D. Time-weighted intent classification.
2. The cyclist intent recognition method considering deep spatiotemporal fusion according to claim 1, characterized in that: The method for obtaining multimodal time-series data of the target cyclist described in step A includes the following steps: A1. Obtain the cyclist's local visual image sequence and spatiotemporal trajectory information. A cyclist's movement trends are described by the historical sequence of bounding boxes implicitly reflecting their intentions. Simultaneously, local visual information further reveals the cyclist's intentions. Therefore, this paper proposes a method to identify the cyclist's intentions in the interaction area based on two types of basic information. Thus, the identification variables of the cyclist intention recognition model are: cyclist trajectory features... and visual features ;in, , It is the first t The x-coordinate of the top-left corner of the frame bounding box. It is the first t The ordinate of the top-left corner of the frame bounding box. It is the first t The x-coordinate of the bottom right corner of the frame bounding box. It is the first t The ordinate of the bottom right corner of the frame bounding box; Indicates the first t Local visual information of a frame, T This indicates the length of the observed time series; A2. Data Preprocessing and Augmentation The extracted local image sequences were standardized as follows: First, each bounding box was enlarged by a factor of 1 in each direction to include richer contextual information; then, all images were adjusted to a uniform size of 224×224 pixels; finally, normalization was performed by subtracting the mean [0.485, 0.456, 0.406] and dividing by the standard deviation [0.229, 0.224, 0.225]; the trajectory sequences were time-aligned to ensure that the visual images and trajectory data corresponded strictly in time.
3. The cyclist intent recognition method considering deep spatiotemporal fusion according to claim 1, characterized in that: Step B, the method for extracting and embedding the cyclist's visual and trajectory features, includes the following steps: B1. Extracting visual features The preprocessed image sequence is input into a lightweight convolutional neural network, MobileNetV2, for feature extraction. To further enhance the expressive power of local features and reduce the number of model parameters, the image is then fed into a convolutional feedforward network module after processing by the MobileNetV2 network. The convolutional feedforward network module outputs a visual feature tensor through convolution operations with a stride of 1. ,in C , H , W These are the number of channels, height, and width, respectively. B2. Embedded Spatiotemporal Features Cyclists' movement trends are described by the historical sequence of bounding boxes that implicitly reflect their intentions. Considering that the original bounding box data has low dimensionality and limited features, while the extracted local visual features have high dimensionality, a fully connected layer is used to integrate the trajectory features. B and visual features V They are embedded into the same feature space for unified processing and feature fusion, resulting in... , , The dimension of the embedding is represented by the following formula: in, MLP This indicates a fully connected layer.
4. The cyclist intent recognition method considering deep spatiotemporal fusion according to claim 1, characterized in that: The cross-modal feature fusion method based on sparse attention mechanism described in step C includes the following steps: C1. Temporal and Modal Interaction Modeling The inherent temporal correlations of visual and spatial features, as well as the cross-correlations between different features, are captured using a sparse attention encoder. First, the training weights of two mappings, B and V, are processed through a fully connected layer to obtain two class labels. and Add it to the spatial features according to the time axis. and visual features Previously, the initial fusion feature sequence was obtained as follows: The formula is as follows: C2. Initial fusion feature sequence Add sinusoidal position encoding; sinusoidal position encoding preserves positional information and ensures the permutation invariance of subsequent attention operations, and its definition is as follows: In the formula, PE A position vector generated for each time frame. This indicates that the data at a certain time has a length of T The index position in the input. d For the hidden layer dimension of the model, k The dimension index of the position vector; C3. Input the feature sequence that incorporates location information into a... L In a Transformer encoder composed of stacked identical layers, each layer contains an attention module and a feedforward network; the attention module employs a sparse attention mechanism, which reduces computational complexity by sparsifying the standard attention matrix. Reduced to approximately It efficiently models the dependencies of long sequences; Specifically, the sparse attention mechanism takes a feature sequence incorporating location information as input and generates a query, key, and value vector for each attention head h. This multidimensional vector is represented as follows: In the formula, It is the first Query vector in the header dimension It is the first Keyword vectors in the head dimension It is the first The value vector along the head dimension; It is a learnable parameter matrix for generating query vectors. It is a learnable parameter matrix that generates the key vectors. It is a learnable parameter matrix that generates value vectors; Next, the multi-dimensional vectors are aggregated to generate multi-dimensional output features. The calculation formula is as follows: In the formula, spare attention(.) is the sparse attention layer; C4. After completing the modeling of visual and spatial features, obtain the aggregated... B and V A vector of all time information Mark it and the learnable class. Integrating to obtain Then, Transformer encoding is used to mine the relationships between different modalities. Then use replace Class tags in Finally, layer normalization is performed through fully connected layers and residual connections; C5. Feature compression and aggregation; To reduce redundant information and lower parameters in the two fused features, a bottleneck structure is used to compress the features, as shown below: in, , This represents the compressed features. Following this method, B and V The two features are segmented to obtain: Finally, feature aggregation is performed by addition to obtain the final result. ,in, It indicates the first i The feature vectors at each time step.
5. The cyclist intent recognition method considering deep spatiotemporal fusion according to claim 1, characterized in that: Step D, the time-weighted intent classification method, includes the following steps: D1. To aggregate information along the time dimension, an attention module is used to learn different frames. The importance is shown in the following formula: In the formula, h and This represents the learnable parameters. For the calculated first i Normalized attention weights for frames; D2. Obtain the global feature representation for intent classification by weighted summation. z : D3. Obtain a global feature representation focusing on key time-series information. z The input is fed into a classifier for intent determination; specifically, it is processed through a fully connected layer. z Mapped to a dimensional space with the same number of candidate intent categories, and then normalized using the Softmax function, the probability of belonging to each intent category is obtained.