Pedestrian abnormal behavior video generation method and system based on sparse attention
By employing a sparse attention video generation method and utilizing the ST-DiT model to dynamically focus on key spatiotemporal regions, the generation efficiency and quality issues of abnormal pedestrian behavior scenarios in autonomous driving are addressed. This method achieves efficient generation and high-quality video support, making it suitable for simulation testing and safety assessment of autonomous driving systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING UNION UNIVERSITY
- Filing Date
- 2026-02-04
- Publication Date
- 2026-05-15
AI Technical Summary
Existing video generation technologies struggle to efficiently generate rare but critical pedestrian abnormal behavior scenarios for autonomous driving, especially special scenarios such as pedestrians suddenly crossing the road or illegally crossing at night, resulting in insufficient model generalization ability and wasted computing resources.
A sparse attention-based method for generating videos of abnormal pedestrian behavior is adopted. By using a semantically aware spatiotemporal sparse diffusion transformer (ST-DiT), the sparse attention mechanism of the spatial and temporal Transformer modules is utilized, combined with high-granularity and low-granularity clustering strategies, to dynamically focus on key spatiotemporal regions and generate high-quality videos of abnormal pedestrian behavior.
It significantly improves computational efficiency, achieving a speedup of 2.3 times, while maintaining the quality of the generated videos. The generated videos outperform baseline methods in metrics such as PSNR, SSIM, and LPIPS, providing high realism and diverse data support, and are suitable for simulation testing and safety assessment of autonomous driving systems.
Smart Images
Figure CN122049983A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the fields of computer vision and intelligent transportation technology, and in particular relates to a method and system for generating videos of abnormal pedestrian behavior based on sparse attention. Background Technology
[0002] In the field of autonomous driving, special scenarios occur very infrequently in actual road surveys, making it difficult to cover real road conditions. Examples include pedestrians suddenly crossing the road, illegal crossings at night, and visual occlusion behaviors in adverse weather. These difficult-to-capture special scenarios are often directly related to driving safety. Therefore, high-quality video generation technology has crucial strategic value in the development and verification of autonomous driving systems, especially in achieving high-fidelity synthesis and prediction of pedestrian interactions in these special scenarios. Accurate generation and behavioral simulation of these critical hazardous situations can significantly enhance the perception and decision-making robustness of autonomous driving systems in complex environments, compensate for the insufficient model generalization ability caused by the scarcity of real-world data, and thus provide controllable, rich, and diverse data support for system simulation testing, safety assessment, and closed-loop optimization. The development of video generation methods has gone through several stages, each with its own unique advantages and limitations.
[0003] In the early days, generative adversarial networks (GANs) were widely used for video generation. They could generate visually plausible frame sequences through adversarial training mechanisms, but they suffered from problems such as training instability, temporal inconsistencies, and pattern collapse in the generated videos. Subsequently, variational autoencoders improved the ability to model the latent distribution of videos by introducing latent variable structures, resulting in more continuous generated results. However, they still struggled to handle long-range dependencies in complex dynamic scenes.
[0004] In recent years, diffusion models have demonstrated powerful capabilities in generation tasks, particularly their advantages in detail preservation and training stability, gradually becoming the mainstream paradigm for video generation. Building on this foundation, a research team from a Chinese university has further introduced the Transformer architecture, proposing the Video Diffusion Transformer (VDT) framework. VDT leverages the global tokenization and self-attention mechanisms of the Transformer to significantly improve its ability to model long sequence temporal dependencies and supports various video generation tasks through a unified spatiotemporal masking mechanism, marking a significant breakthrough in video generation technology.
[0005] However, VDT still faces significant bottlenecks: the attention mechanism in its core Transformer module has quadratic computational complexity, leading to high computational overhead and memory consumption, severely limiting generation efficiency and application scope. Although some sparse attention methods have attempted to reduce computational costs, they often rely on predefined fixed patterns or simple heuristics, failing to dynamically focus on semantically key regions based on video content, resulting in insufficient recognition accuracy and wasted computational resources. Summary of the Invention
[0006] To address the aforementioned technical issues, this invention proposes a method and system for generating videos of abnormal pedestrian behavior based on sparse attention. Based on the semantically aware spatiotemporal sparse diffusion transformer ST-DiT, it significantly improves the computational and memory efficiency of the model while maintaining high-fidelity video generation quality.
[0007] To achieve the above objectives, this invention provides a method for generating videos of abnormal pedestrian behavior based on sparse attention, comprising:
[0008] Obtain the video frame sequence, map each frame to the latent space, and obtain the latent representation;
[0009] The latent representation is converted into a token sequence through a block embedding layer, and position encoding is added to the token sequence to obtain a position-optimized sequence.
[0010] The position-optimized sequence is input into the spatial Transformer module. The spatial Transformer module uses a semantically aware sparse attention mechanism and a high-granularity clustering strategy to obtain spatial context features and update the sequence content. The token sequence is then reverse-rearranged to restore the original position-optimized sequence.
[0011] The updated position-optimized sequence is input into the temporal Transformer module, which uses a semantically aware sparse attention mechanism and a low-granularity clustering strategy to obtain spatial context features and update the sequence content. The token sequence is then reverse-rearranged to restore the original position-optimized sequence.
[0012] The sequence processed in the spatiotemporal domain is reconstructed into video frames through the decoder of the variational autoencoder to generate the target video sequence;
[0013] The trained model is fine-tuned using a dataset of abnormal pedestrian behavior scenarios to generate videos of abnormal pedestrian behavior.
[0014] Optionally, acquiring the video frame sequence and mapping each frame to the latent space includes:
[0015] Extract consecutive frames from the video at a fixed sampling rate to form a video frame sequence;
[0016] Each frame in the video frame sequence is input into the encoder of the variational autoencoder to obtain the latent representation of each frame;
[0017] The number of frames in the video frame sequence is determined by the fixed sampling rate, and the latent representation is used for subsequent block embedding layer processing.
[0018] Optionally, converting the latent representation into a token sequence through a block embedding layer and adding position encoding to the token sequence includes:
[0019] The block embedding layer is implemented by a linear projection layer, which is used to convert the feature vector of each spatial location in the latent representation into a token;
[0020] The token sequence is added to the position code to obtain the position-optimized sequence; wherein the position code is embedded in the token sequence to preserve the position information of the sequence.
[0021] Optionally, the Transformer module that optimizes the positional input sequence space includes:
[0022] The k-means clustering algorithm is used to perform cluster analysis on the token sequences. Clustering is performed by calculating the similarity of the token semantic features. The spatial Transformer module adopts high-granularity clustering, and the number of clusters is in the range of 150 to 300.
[0023] Based on the clustering results, the token sequence is rearranged using a permutation matrix so that tokens belonging to the same cluster are arranged consecutively.
[0024] Attention weights are calculated on the rearranged token sequence, and a Top-p selection strategy is used to calculate attention only for the top 90% of clusters with the highest total attention scores.
[0025] After the calculation is complete, update the sequence content and use the inverse permutation matrix to reverse the token sequence to restore the sequence order.
[0026] Optionally, the spatially processed sequence can be input into the time Transformer module, which includes:
[0027] The k-means clustering algorithm is used to perform cluster analysis on the token sequence. Clustering is performed by calculating the similarity of the token semantic features. The temporal Transformer module adopts low-granularity clustering, and the number of clusters is in the range of 20 to 80.
[0028] Based on the clustering results, the token sequence is rearranged using a permutation matrix so that tokens belonging to the same cluster are arranged consecutively in memory.
[0029] Attention weights are calculated on the rearranged token sequence, and a Top-p selection strategy is used to calculate attention only for the top 90% of clusters with the highest total attention scores.
[0030] After the calculation is complete, update the sequence content and use the inverse permutation matrix to reverse the token sequence to restore the sequence order.
[0031] Optionally, reconstructing the spatiotemporally processed sequence into video frames using a variational autoencoder decoder includes:
[0032] The processed token sequence is input into the decoder of the variational autoencoder;
[0033] The decoder converts the token sequence back into pixel space to generate video frames;
[0034] The video frames are combined to form a target video sequence, thus completing the video generation of abnormal pedestrian behavior.
[0035] Optionally, fine-tuning the spatiotemporal sparse diffusion transform with a dataset of pedestrian abnormal behavior scenarios includes:
[0036] The dataset contains video scenes of pedestrians waving to signal for a stop, pedestrians climbing over railings, and pedestrians suddenly rushing into the lane when their view is obscured by a car.
[0037] The fine-tuning uses an adaptive moment estimation optimizer with a learning rate set to 1e-5 and a batch size set according to GPU memory.
[0038] The fine-tuning training cycle is 50 epochs to optimize the spatiotemporal consistency and behavioral accuracy of the generated videos.
[0039] On the other hand, to achieve the above objectives, the present invention also provides a pedestrian abnormal behavior video generation system based on sparse attention, comprising:
[0040] The data acquisition and encoding module is used to acquire video frame sequences, map each frame to the latent space, and obtain the latent representation.
[0041] The tokenization module is used to convert the latent representation into a token sequence through a block embedding layer, and add position encoding to the token sequence to obtain a position-optimized sequence.
[0042] The spatial processing module is used to input the position optimization sequence into the spatial Transformer module. The spatial Transformer module uses a semantically aware sparse attention mechanism and a high-granularity clustering strategy to obtain spatial context features and update the sequence content. It also performs inverse rearrangement of the token sequence to restore the original position optimization sequence.
[0043] The time processing module is used to input the updated position optimization sequence into the time Transformer module. The time Transformer module uses a semantically aware sparse attention mechanism and a low-granularity clustering strategy to obtain spatial context features and update the sequence content. It also performs inverse rearrangement of the token sequence to restore the original position optimization sequence.
[0044] The video generation module is used to reconstruct video frames from the spatiotemporal processed sequence through the decoder of the variational autoencoder, thereby generating the target video sequence.
[0045] The model fine-tuning module is used to fine-tune the spatiotemporal sparse diffusion transformer with the dataset of pedestrian abnormal behavior scenarios to generate videos of pedestrian abnormal behavior.
[0046] Technical effects of the invention:
[0047] Significantly improved computational efficiency: Through semantically aware clustering and rearrangement mechanisms, the computational complexity of attention is reduced from that of the traditional Transformer. Reduce to This significantly reduces computational and memory overhead, achieving a speedup of up to 2.3 times while maintaining generation quality, which is significantly better than existing sparse attention methods.
[0048] Optimized generation quality: The semantic-based dynamic token selection mechanism effectively focuses on key spatiotemporal regions, avoiding information loss caused by fixed sparse patterns. The generated video outperforms the baseline method in terms of PSNR, SSIM, LPIPS, and other metrics, especially in terms of pedestrian abnormal behavior details and motion coherence.
[0049] Enhanced applicability and practicality: This invention can efficiently generate rare but critical pedestrian dangerous behavior scenarios in autonomous driving, providing highly realistic and diverse synthetic data support for simulation testing, decision planning, and safety assessment of autonomous driving systems, and has significant industrial application value.
[0050] In summary, this invention, by introducing a semantically guided sparse computational structure, achieves significant optimization of computational efficiency while maintaining the generation effect, and is particularly suitable for video generation in autonomous driving scenarios with high real-time requirements. Attached Figure Description
[0051] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0052] Figure 1 This is a flowchart illustrating a method for generating videos of abnormal pedestrian behavior based on sparse attention, according to an embodiment of the present invention.
[0053] Figure 2 This is a diagram of the ST-DiT network structure according to an embodiment of the present invention;
[0054] Figure 3 This is a flowchart illustrating the computation of the semantic-aware sparse attention mechanism in an embodiment of the present invention. Detailed Implementation
[0055] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0056] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0057] like Figure 1 As shown, this embodiment provides a method for generating videos of abnormal pedestrian behavior based on sparse attention, including:
[0058] Obtain the video frame sequence, map each frame to the latent space, and obtain the latent representation;
[0059] The latent representation is converted into a token sequence through a block embedding layer, and position encoding is added to the token sequence to obtain a position-optimized sequence.
[0060] The position-optimized sequence is input into the spatial Transformer module. The spatial Transformer module uses a semantically aware sparse attention mechanism and a high-granularity clustering strategy to obtain spatial context features and update the sequence content. The token sequence is then reverse-rearranged to restore the original position-optimized sequence.
[0061] The updated position-optimized sequence is input into the temporal Transformer module, which uses a semantically aware sparse attention mechanism and a low-granularity clustering strategy to obtain spatial context features and update the sequence content. The token sequence is then reverse-rearranged to restore the original position-optimized sequence.
[0062] The sequence processed in the spatiotemporal domain is reconstructed into video frames through the decoder of the variational autoencoder to generate the target video sequence;
[0063] The trained model is fine-tuned using a dataset of abnormal pedestrian behavior scenarios to generate videos of abnormal pedestrian behavior.
[0064] Furthermore, acquiring the video frame sequence and mapping each frame to the latent space includes:
[0065] Extract consecutive frames from the video at a fixed sampling rate to form a video frame sequence;
[0066] Each frame in the video frame sequence is input into the encoder of the variational autoencoder to obtain the latent representation of each frame;
[0067] The number of frames in the video frame sequence is determined by the fixed sampling rate, and the latent representation is used for subsequent block embedding layer processing.
[0068] Furthermore, converting the latent representation into a token sequence through a block embedding layer and adding positional encoding to the token sequence includes:
[0069] The block embedding layer is implemented by a linear projection layer, which is used to convert the feature vector of each spatial location in the latent representation into a token;
[0070] The token sequence is added to the position code to obtain the position-optimized sequence; wherein the position code is embedded in the token sequence to preserve the position information of the sequence.
[0071] Furthermore, the Transformer module for optimizing the positional input sequence space includes:
[0072] The k-means clustering algorithm is used to perform cluster analysis on the token sequences. Clustering is performed by calculating the similarity of the token semantic features. The spatial Transformer module adopts high-granularity clustering, and the number of clusters is in the range of 150 to 300.
[0073] Based on the clustering results, the token sequence is rearranged using a permutation matrix so that tokens belonging to the same cluster are arranged consecutively.
[0074] Attention weights are calculated on the rearranged token sequence, and a Top-p selection strategy is used to calculate attention only for the top 90% of clusters with the highest total attention scores.
[0075] After the calculation is complete, update the sequence content and use the inverse permutation matrix to reverse the token sequence to restore the sequence order.
[0076] Furthermore, the spatially processed sequence is input into the time-varying Transformer module, which includes:
[0077] The k-means clustering algorithm is used to perform cluster analysis on the token sequence. Clustering is performed by calculating the similarity of the token semantic features. The temporal Transformer module adopts low-granularity clustering, and the number of clusters is in the range of 20 to 80.
[0078] Based on the clustering results, the token sequence is rearranged using a permutation matrix so that tokens belonging to the same cluster are arranged consecutively in memory.
[0079] Attention weights are calculated on the rearranged token sequence, and a Top-p selection strategy is used to calculate attention only for the top 90% of clusters with the highest total attention scores.
[0080] After the calculation is complete, update the sequence content and use the inverse permutation matrix to reverse the token sequence to restore the sequence order.
[0081] Furthermore, reconstructing the video frames from the spatiotemporally processed sequence using a variational autoencoder decoder includes:
[0082] The processed token sequence is input into the decoder of the variational autoencoder;
[0083] The decoder converts the token sequence back into pixel space to generate video frames;
[0084] The video frames are combined to form a target video sequence, thus completing the video generation of abnormal pedestrian behavior.
[0085] Furthermore, fine-tuning the spatiotemporal sparse diffusion transform with a dataset of pedestrian abnormal behavior scenarios includes:
[0086] The dataset contains video scenes of pedestrians waving to signal for a stop, pedestrians climbing over railings, and pedestrians suddenly rushing into the lane when their view is obscured by a car.
[0087] The fine-tuning uses an adaptive moment estimation optimizer with a learning rate set to 1e-5 and a batch size set according to GPU memory.
[0088] The fine-tuning training cycle is 50 epochs to optimize the spatiotemporal consistency and behavioral accuracy of the generated videos.
[0089] Specifically, the implementation process of this embodiment includes:
[0090] The dataset, used for training and fine-tuning, contains videos of abnormal pedestrian behavior in various specific scenarios. These include pedestrians waving to signal for a stop, pedestrians climbing over railings, and pedestrians suddenly rushing into the roadway while obscured by vehicles. Approximately 450 videos are collected for each scenario, with a total duration of about one hour. The scenarios must cover different weather conditions (sunny, rainy), lighting conditions (daytime, nighttime), and road environments (urban, rural). The videos are formatted at 1080p resolution and 30fps.
[0091] First, the input video is presampled at a sampling rate of 16 frames per second for a fixed period of time to obtain a video frame sequence. ,in Indicates the first in the video Frame images. Next, each frame is placed into the latent space to obtain a latent representation. ,in For frame number, and Here, represents the height and width of the latent feature map, and C represents the number of channels.
[0092] The potential meaning Through a block embedding layer, implemented by a linear projection layer, each spatial location is... The feature vectors are converted into tokens of dimension D in the hidden layer, and the output is a token sequence. Then encode the position. By embedding the token sequence, a position-optimized sequence is obtained. Hidden layer dimensions N is a hyperparameter, preferably 1152, and the block size is determined by the downsampling rate of the pre-trained VAE, with a typical value of (1,2,2), where N=T*H*W is the total number of tokens.
[0093] Optimize the position sequence The input space Transformer module is responsible for modeling the spatial context relationships within each frame (such as pedestrian poses and background object layouts). Its core is a semantically aware sparse attention mechanism, and its operation is as follows: Figure 3 As shown, it specifically includes:
[0094] For the query, key, and value tensors in the attention mechanism, a clustering algorithm is applied based on their semantic features to assign semantically similar tokens to the same cluster.
[0095] Arrange the tokens consecutively within the same cluster to obtain the rearranged order;
[0096] Calculate attention weights on the rearranged contiguous layout, and calculate sparse attention weights based on the cluster;
[0097] Update the sequence content and restore the sequence to the original token order using a reverse rearrangement operation.
[0098] Clustering Analysis: For each attention head, the k-means clustering algorithm is applied to the Query and Key tensors respectively. The clustering is based on the semantic features of the tokens; in this embodiment, visual features extracted by the pre-trained CLIP-ViT model are used as the clustering basis. The spatial module adopts high-granularity clustering with a large number of clusters (preferably 150-300) to capture subtle spatial details.
[0099] The clustering algorithm is k-means, and the clustering is applied independently to the query token at each attention head and each Transformer layer. Key tokens ,create Query cluster and bond clusters ,in and belong and The number of tokens, A set of data points For the set of cluster centers, It is a distance function;
[0100] .
[0101] Token rearrangement: Based on the clustering results, a permutation matrix is used. Tokens belonging to the same cluster are arranged consecutively in memory.
[0102] Sparse attention computation: Attention weights are calculated on contiguous memory blocks after rearrangement. A Top-p strategy (p=0.9) is adopted, calculating only the top 90% of clusters with the highest total attention scores, further saving computational resources, and obtaining the rearranged attention. The attention output after the arrangement is obtained, where Let the permutation matrix acting on the query tensor Q satisfy the following condition: Where I is the identity matrix. Let the permutation matrix acting on the key tensor K and the value tensor V satisfy the following condition: , , , This represents the query, key, and value tensor after permutation and rearrangement. This represents the block sparse attention computation performed on the permuted contiguous memory layout. The transpose matrix of Πq, i.e., the inverse permutation matrix, is used to restore the output O′ to the original token order. This is a scaling factor used to prevent the gradient from vanishing due to an excessively large dot product result;
[0103] ;
[0104] ;
[0105] A Top-p strategy is used to dynamically select key clusters with high importance to participate in attention calculation. Importance is determined by the sum of attention scores of tokens within the cluster, calculated as follows:
[0106] ;
[0107] ;
[0108] in, As a scalar, Let be a vector, representing the centroid of the i-th query cluster. Denotes the centroid of the j-th bond cluster. The scaling factor. The approximate attention weights between the i-th query cluster and the j-th key cluster are... Let be the number of tokens in the j-th key cluster. Indicates the total number of bond clusters. This represents the natural exponential function.
[0109] The spatially processed sequence is input into the temporal Transformer module. This module is responsible for modeling the temporal context relationships between frames. Its structure is similar to the spatial module, but it uses low-granularity clustering, reducing the number of clusters. A smaller value, preferably in the range of 20-80, is preferred because changes in the time dimension are generally smoother and more global than those in the spatial dimension. A permutation matrix using the time dimension is employed. and .
[0110] Using specific pedestrian abnormal behavior datasets, such as Figure 2 The entire ST-DiT model is subjected to supervised fine-tuning, as shown. The fine-tuning uses an adaptive moment estimation optimizer with a learning rate of 1e-5, a batch size of 8 based on GPU memory, and a training period of 50 epochs. The goal of the fine-tuning is to enable the model to learn to generate video sequences with high spatiotemporal consistency that conform to real pedestrian abnormal behavior patterns.
[0111] This embodiment also provides a pedestrian abnormal behavior video generation system based on sparse attention, including:
[0112] The data acquisition and encoding module is used to acquire video frame sequences. It uses a pre-trained variational autoencoder (VAE) to map each frame image from the pixel space to the latent space to obtain the corresponding latent representation.
[0113] The tokenization module divides the latent representation into non-overlapping image patches through the Patch Embedding Layer and converts them into a token sequence. Then, it adds a spatiotemporal location code P to the token sequence to generate a location-optimized sequence containing location information, so as to maintain the model's ability to perceive spatial structure and temporal order.
[0114] The spatial processing module inputs the position optimization sequence into the spatial Transformer module. This module adopts a semantically aware sparse attention mechanism. First, it performs fine-grained k-means clustering on the query, key, and value tokens. Based on semantic similarity, the tokens are divided into multiple compact clusters. Then, sparse attention calculation is performed only on the clustered key token clusters, and the sequence content is updated and then reversed to restore the original spatial structure.
[0115] The temporal processing module inputs the spatially processed sequence into the temporal Transformer module. This module also adopts a semantically aware sparse attention mechanism, but performs low-granularity (coarse-grained) k-means clustering on the token sequence to capture long-range temporal dependencies across frames. By performing sparse attention calculation on key temporal token clusters and updating the sequence content before reordering, it effectively models the temporal dynamics between video frames.
[0116] The video generation module reconstructs the spatiotemporally processed sequence into pixel-space video frames through a variational autoencoder (VAE) decoder, ultimately generating a high-quality, spatiotemporally consistent target video sequence.
[0117] The model fine-tuning module is used to fine-tune the spatiotemporal sparse diffusion transformer with the dataset of pedestrian abnormal behavior scenarios to generate videos of pedestrian abnormal behavior.
[0118] Advantages of this invention:
[0119] Significantly improved computational efficiency: Through semantically aware clustering and rearrangement mechanisms, the computational complexity of attention is reduced from that of the traditional Transformer. Reduce to This invention significantly reduces computational and memory overhead, achieving a speedup of up to 2.3 times while maintaining generation quality, significantly outperforming existing sparse attention methods. Generation quality optimization: A semantically guided dynamic token selection mechanism effectively focuses on key spatiotemporal regions, avoiding information loss due to fixed sparse patterns. The generated videos outperform baseline methods in PSNR, SSIM, and LPIPS metrics, particularly excelling in capturing details of abnormal pedestrian behavior and motion coherence. Enhanced applicability and practicality: This invention can efficiently generate rare but critical pedestrian dangerous behavior scenarios in autonomous driving, providing highly realistic and diverse synthetic data support for simulation testing, decision planning, and safety assessment of autonomous driving systems, possessing significant industrial application value. In summary, this invention, by introducing a semantically guided sparse computational structure, achieves significant optimization of computational efficiency while maintaining generation quality, making it particularly suitable for video generation in autonomous driving scenarios with high real-time requirements.
[0120] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for generating videos of abnormal pedestrian behavior based on sparse attention, characterized in that, include: Obtain the video frame sequence, map each frame to the latent space, and obtain the latent representation; The latent representation is converted into a token sequence through a block embedding layer, and position encoding is added to the token sequence to obtain a position-optimized sequence. The position-optimized sequence is input into the spatial Transformer module. The spatial Transformer module uses a semantically aware sparse attention mechanism and a high-granularity clustering strategy to obtain spatial context features and update the sequence content. The token sequence is then reverse-rearranged to restore the original position-optimized sequence. The updated position-optimized sequence is input into the temporal Transformer module, which uses a semantically aware sparse attention mechanism and a low-granularity clustering strategy to obtain spatial context features and update the sequence content. The token sequence is then reverse-rearranged to restore the original position-optimized sequence. The sequence processed in the spatiotemporal domain is reconstructed into video frames through the decoder of the variational autoencoder to generate the target video sequence; The trained model is fine-tuned using a dataset of abnormal pedestrian behavior scenarios to generate videos of abnormal pedestrian behavior.
2. The method for generating videos of abnormal pedestrian behavior based on sparse attention as described in claim 1, characterized in that, Obtaining the video frame sequence and mapping each frame to the latent space includes: Extract consecutive frames from the video at a fixed sampling rate to form a video frame sequence; Each frame in the video frame sequence is input into the encoder of the variational autoencoder to obtain the latent representation of each frame; The number of frames in the video frame sequence is determined by the fixed sampling rate, and the latent representation is used for subsequent block embedding layer processing.
3. The method for generating videos of abnormal pedestrian behavior based on sparse attention as described in claim 1, characterized in that, Converting the latent representation into a token sequence through a block embedding layer and adding positional encoding to the token sequence includes: The block embedding layer is implemented by a linear projection layer, which is used to convert the feature vector of each spatial location in the latent representation into a token; The token sequence is added to the position code to obtain the position-optimized sequence; wherein the position code is embedded in the token sequence to preserve the position information of the sequence.
4. The method for generating videos of abnormal pedestrian behavior based on sparse attention as described in claim 1, characterized in that, The Transformer module optimizes the positional input sequence space and includes: The k-means clustering algorithm is used to perform cluster analysis on the token sequences. Clustering is performed by calculating the similarity of the token semantic features. The spatial Transformer module adopts high-granularity clustering, and the number of clusters is in the range of 150 to 300. Based on the clustering results, the token sequence is rearranged using a permutation matrix so that tokens belonging to the same cluster are arranged consecutively. Attention weights are calculated on the rearranged token sequence, and a Top-p selection strategy is used to calculate attention only for the top 90% of clusters with the highest total attention scores. After the calculation is complete, update the sequence content and use the inverse permutation matrix to reverse the token sequence to restore the sequence order.
5. The method for generating videos of abnormal pedestrian behavior based on sparse attention as described in claim 1, characterized in that, The spatial domain processed sequence is input into the time Transformer module, which includes: The k-means clustering algorithm is used to perform cluster analysis on the token sequences. Clustering is performed by calculating the similarity of the token semantic features. The temporal Transformer module adopts low-granularity clustering, and the number of clusters is in the range of 20 to 80. Based on the clustering results, the token sequence is rearranged using a permutation matrix so that tokens belonging to the same cluster are arranged consecutively in memory. Attention weights are calculated on the rearranged token sequence, and a Top-p selection strategy is used to calculate attention only for the top 90% of clusters with the highest total attention scores. After the calculation is complete, update the sequence content and use the inverse permutation matrix to reverse the token sequence to restore the sequence order.
6. The method for generating videos of abnormal pedestrian behavior based on sparse attention as described in claim 1, characterized in that, The process of reconstructing video frames from the spatiotemporally processed sequence using a variational autoencoder decoder includes: The processed token sequence is input into the decoder of the variational autoencoder; The decoder converts the token sequence back into pixel space to generate video frames; The video frames are combined to form a target video sequence, thus completing the video generation of abnormal pedestrian behavior.
7. The method for generating videos of abnormal pedestrian behavior based on sparse attention as described in claim 1, characterized in that, Fine-tuning the spatiotemporal sparse diffusion transformer using a dataset of pedestrian abnormal behavior scenarios includes: The dataset contains video scenes of pedestrians waving to signal for a stop, pedestrians climbing over railings, and pedestrians suddenly rushing into the lane when their view is obscured by a car. The fine-tuning uses an adaptive moment estimation optimizer with a learning rate set to 1e-5 and a batch size set according to GPU memory. The fine-tuning training cycle is 50 epochs to optimize the spatiotemporal consistency and behavioral accuracy of the generated videos.
8. A system for generating videos of abnormal pedestrian behavior based on sparse attention, characterized in that, A method for generating videos of abnormal pedestrian behavior based on sparse attention as described in any one of claims 1-7, comprising: The data acquisition and encoding module is used to acquire video frame sequences, map each frame to the latent space, and obtain the latent representation. The tokenization module is used to convert the latent representation into a token sequence through a block embedding layer, and add position encoding to the token sequence to obtain a position-optimized sequence. The spatial processing module is used to input the position optimization sequence into the spatial Transformer module. The spatial Transformer module uses a semantically aware sparse attention mechanism and a high-granularity clustering strategy to obtain spatial context features and update the sequence content. It also performs inverse rearrangement of the token sequence to restore the original position optimization sequence. The time processing module is used to input the updated position optimization sequence into the time Transformer module. The time Transformer module uses a semantically aware sparse attention mechanism and a low-granularity clustering strategy to obtain spatial context features and update the sequence content. It also performs inverse rearrangement of the token sequence to restore the original position optimization sequence. The video generation module is used to reconstruct video frames from the spatiotemporal processed sequence through the decoder of the variational autoencoder, thereby generating the target video sequence. The model fine-tuning module is used to fine-tune the spatiotemporal sparse diffusion transformer with the dataset of pedestrian abnormal behavior scenarios to generate videos of pedestrian abnormal behavior.