Traffic video abnormal event detection method, device and equipment and storage medium

Abstract

The application relates to a traffic video abnormal event detection method, device and equipment and a storage medium. The method comprises the following steps: performing data preprocessing on a traffic monitoring video stream, converting continuous videos into a video block sequence, then constructing and training a frame prediction model, calculating a prediction error according to the difference between a predicted frame and an actual frame, realizing coarse-grained detection of an abnormal event and abnormal segment positioning through the prediction error, and generating abnormal space-time features; performing non-uniform sampling in the time domain and key frame compression processing, generating a key frame video sequence containing fine-grained abnormal categories and inputting the key frame video sequence into a double-modal frame classification model, calculating the matching relationship between the video and the abnormal category semantics, generating the probability distribution of each abnormal category, and generating a video abnormal classification result according to the probability distribution. The application does not need to rely on a large amount of abnormal labeled data, can realize accurate discovery and qualitative classification of various rare traffic events in a complex road network long video under the premise of effectively reducing the calculation time consumption.

Description

Technical Field

[0001] This application relates to the field of traffic image detection technology, and more specifically to a method, apparatus, device, and storage medium for detecting abnormal events in traffic videos. Background Technology

[0002] With the rapid development of intelligent transportation and intelligent monitoring technologies, large-scale traffic video surveillance systems have been widely deployed on highways, urban roads, and important transportation hubs. By continuously collecting video data on road operation status, these systems provide crucial data support for traffic safety management and early warning of abnormal events. Traffic anomaly detection technology using video analytics can automatically identify abnormal behaviors such as traffic accidents, road congestion, illegal occupation of lanes, littering, and pedestrians or animals entering the road. This is of great significance for improving traffic management efficiency, reducing manual inspection costs, and ensuring road safety.

[0003] Currently, most existing traffic anomaly detection methods are based on supervised learning frameworks, using deep learning models to classify and identify abnormal events in videos. However, in practical applications of traffic anomaly detection for long-form highway videos, several challenges remain. On the one hand, anomalies such as traffic accidents, debris spills, and road damage occur infrequently and exhibit a typical long-tail distribution, resulting in a severe shortage of labeled anomaly samples. This makes it easy for models to overfit during training, making it difficult to learn anomaly features with good generalization ability, thus leading to a high false negative rate. On the other hand, traffic monitoring videos are usually non-stationary time series collected continuously over long periods. The data distribution changes dynamically over time and contains a large number of redundant frames and complex temporal relationships, resulting in high computational overhead and a high false alarm rate, making it difficult to balance efficiency and accuracy. Furthermore, traffic scenarios are open and unpredictable, with a large number of unlabeled unknown anomaly types. Traditional closed-domain models struggle to effectively understand and automatically discover these anomalies, and the frequent occurrence of such unknown anomalies seriously affects traffic safety monitoring. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for detecting abnormal events in traffic videos. It adopts a two-stage architecture, does not rely on massive amounts of anomaly annotation data, and can accurately detect and classify various rare traffic events in long videos of complex road networks while effectively reducing computation time.

[0005] In a first aspect, this application provides a method for detecting abnormal events in traffic videos. The method includes: preprocessing the acquired traffic monitoring video stream to convert continuous video into a standardized video block sequence, wherein the video block sequence includes at least channel dimension, time dimension, and spatial dimension information; constructing and training a spatiotemporal memory Mamba frame prediction model using unsupervised learning based on the video block sequence to model normal traffic behavior and calculate the prediction error based on the difference between the predicted frame and the real frame, thereby achieving coarse-grained detection of abnormal events and localization of abnormal segments through the prediction error, and generating abnormal spatiotemporal features; performing non-uniform sampling and keyframe compression processing in the time domain based on the abnormal spatiotemporal features to generate a keyframe video sequence containing fine-grained abnormal categories; and inputting the keyframe video sequence into a bimodal frame classification model using contrastive learning, calculating the matching relationship between the video and the semantics of the abnormal category through cross-modal alignment between visual features and text semantic features, generating the probability distribution of each abnormal category, and generating a video abnormality classification result based on the probability distribution.

[0006] In one optional embodiment of the first aspect, the data preprocessing of the acquired traffic monitoring video stream to convert continuous video into a standardized video block sequence includes: decoding the acquired traffic monitoring video stream and extracting continuous RGB image frames according to a preset frame rate to construct an image temporal set; performing a spatial scale unification operation on each frame in the image temporal set to scale the image resolution to a preset size to eliminate resolution differences between different video sources; normalizing each frame in the scaled image temporal set to reduce color distribution differences caused by different acquisition devices and lighting conditions by unifying pixel distribution range and color space features; segmenting each normalized frame along the time dimension into multiple fixed-length short video segments, each short video segment containing a preset number of continuous frames and segmented using a non-overlapping sliding window method; and performing dimensional recombination and tensor encapsulation on each short video segment to generate a multi-dimensional tensor form video block sequence that conforms to the requirements of a deep learning framework, wherein each video block sequence includes at least channel dimension, time dimension, and spatial dimension information.

[0007] In one alternative embodiment of the first aspect, the step of constructing and training a spatiotemporal memory Mamba frame prediction model using unsupervised learning based on the video block sequence includes: constructing a spatiotemporal memory Mamba network with a symmetric encoder and decoder structure, wherein the encoder is used to perform hierarchical feature compression and abstract representation on the input video block sequence, and the decoder is used to progressively restore spatial resolution and generate prediction frames; wherein a normal behavior feature memory is constructed at the encoder output, and the normal pattern feature representation is enhanced and the ability to reconstruct abnormal features is suppressed by performing similarity retrieval and reconstruction on the input features; and the difference between the prediction frame and the real frame is calculated as a prediction error and the prediction error is mapped to an anomaly score, and when the anomaly score is greater than a preset threshold, it is determined that there is abnormal behavior at the corresponding time position, and the corresponding features are extracted as abnormal spatiotemporal features; and during the model training process, a multi-constraint joint loss function based on unsupervised learning is used to optimize the model parameters, wherein the loss function includes at least pixel-level reconstruction loss, gradient difference loss, clustering compaction loss, and feature separation loss; and in terms of iterative update of network parameters, gradient descent is performed using the AdamW optimizer.

[0008] In one alternative embodiment of the first aspect, the encoder and decoder comprise multiple spatiotemporal Mamba blocks, each including a spatial VSS block and a temporal VSS block. The spatiotemporal Mamba blocks employ a two-stream serial modeling structure spanning both spatial and temporal dimensions, comprising: converting an input video block sequence into a first feature by performing a first-dimensional reshaping operation, fusing batch and temporal dimensions; sequentially normalizing the first feature and inputting it into the spatial VSS block for spatial feature extraction, and introducing a residual connection mechanism to fuse the original features with the spatial features, generating fused spatial features; converting the fused spatial features into a second feature with a temporal dimension as the sequence length by performing a second-dimensional reshaping operation; and performing normalization on the second feature and inputting it into the temporal VSS block to extract temporal dependent features, and enhancing key temporal dependent features through nonlinear mapping and channel recalibration mechanisms.

[0009] In one alternative embodiment of the first aspect, the spatial VSS block comprises: performing deformable convolution operations on the input first feature to extract deformable adaptive local spatial features and inputting the local spatial features into a two-dimensional state space model; performing serialization processing along multiple spatial scanning directions to generate global spatial features; based on the global spatial features, using the Silu activation function to obtain nonlinear expressive power and randomly discarding some network branches through random regularization; fusing the global spatial features after random regularization and the global spatial features after the Silu activation function element-wise to generate a first fused feature; and performing local spatial feature enhancement processing including normalization, channel projection, residual convolution, and re-normalization, extracting features again from the first fused feature and performing nonlinear mapping again through the Silu activation function, and then performing a second cross-layer residual connection to complete multi-level feature modeling and fusion of local and global to local in the spatial dimension, so as to generate a fused spatial feature that simultaneously contains local details and global structure.

[0010] In one alternative embodiment of the first aspect, the temporal VSS block comprises: performing a depthwise separable convolution operation on the input second feature to extract local temporal variation features between adjacent video frames and inputting the local temporal variation features into a one-dimensional state space model; modeling long-range temporal dependencies in the video sequence by unfolding the sequence along the time axis and utilizing an implicit state transition mechanism; sequentially performing normalization processing and feedforward neural network transformation on the output features of the one-dimensional state space model and introducing a temporal channel attention mechanism; dynamically allocating channel weights through feature recalibration; and adjusting the channel dimension of the processed output features through linear mapping and sequentially processing them through layer normalization and Silu activation functions, and then performing residual connections with the initial input second feature to achieve the fusion of short-term local and long-term global features in the time dimension, generating temporal dependency features.

[0011] In one optional embodiment of the first aspect, the step of generating a keyframe video sequence containing fine-grained anomaly categories by performing non-uniform sampling and keyframe compression processing in the time domain based on the anomaly spatiotemporal features includes: determining the time anchor frame where the anomaly event occurs and its neighboring time interval based on the anomaly spatiotemporal features, constructing an asymmetric sampling window around the time anchor frame in the time dimension, and obtaining dense sampling frames and sparse sampling frames, wherein a dense sampling strategy is used in the high-frequency time region near the anomaly anchor point, and a sparse sampling strategy is used in the low-frequency time region far from the anomaly anchor point; recombining the dense sampling frames and sparse sampling frames according to the time order to generate a compressed video sequence containing a preset number of keyframes; and performing uniform format conversion and tensor quantization processing on the compressed video sequence to generate a standardized keyframe video sequence suitable for subsequent bimodal classification model input.

[0012] In one optional embodiment of the first aspect, the step of inputting the keyframe video sequence into a contrastive learning bimodal frame classification model, calculating the matching relationship between the video and the anomaly category semantics through cross-modal alignment between visual features and text semantic features, generating probability distributions for each anomaly category, and generating video anomaly classification results based on the probability distributions, includes: employing a conditional cue learning text encoder to jointly encode preset scene description cue words with the anomaly spatiotemporal features, generating a dynamic context cue vector, and concatenating the dynamic context cue vector with the anomaly category description text to obtain high-dimensional text semantic features; normalizing the high-dimensional text semantic features and the multi-band visual features output by the video frame encoder and mapping them to a unified feature space, and calculating the cosine similarity between the two to generate a cross-modal similarity matrix; performing normalization mapping on the similarity matrix to convert it into probability distributions for each anomaly category; and outputting video anomaly classification results based on the probability distributions.

[0013] In one alternative embodiment of the first aspect, the video frame encoder comprises:

[0014] A two-dimensional discrete wavelet transform is performed on each frame of the keyframe video sequence to decompose the original image into a low-frequency approximate sub-band and multiple high-frequency detail sub-bands, wherein the low-frequency sub-bands represent the overall structural information of the image and the high-frequency sub-bands represent edge and texture detail information.

[0015] According to the time sampling strategy, cross-frequency domain filtering is performed on frames in different time regions, and the frequency domain features generated by cross-frequency domain filtering are fused to form multi-frequency features that simultaneously contain spatial structure information and motion detail information.

[0016] The multi-frequency features are processed by image block embedding and spatiotemporal location encoding, converted into serialized features, and input into a pre-trained clip image encoder to extract global low-frequency semantic features and local high-frequency semantic features, respectively; and,

[0017] The global low-frequency semantic features and local high-frequency semantic features are fused by a multilayer perceptron to generate unified multi-band visual features.

[0018] Secondly, this application provides a traffic video anomaly event detection device using the method according to any one of the first aspects, comprising: a data processing module for preprocessing the acquired traffic monitoring video stream to convert continuous video into a standardized video block sequence; a frame prediction module for constructing and training a spatiotemporal memory Mamba frame prediction model using unsupervised learning based on the video block sequence to model normal traffic behavior and calculate prediction error based on the difference between predicted frames and real frames, thereby achieving coarse-grained detection of anomalies and localization of anomaly segments through the prediction error, and generating anomaly spatiotemporal features; a video processing module for performing non-uniform sampling and keyframe compression processing in the time domain based on the anomaly spatiotemporal features to generate a keyframe video sequence containing fine-grained anomaly categories; and an anomaly detection module for inputting the keyframe video sequence into a bimodal frame classification model using contrastive learning, calculating the matching relationship between video and anomaly category semantics through cross-modal alignment between visual features and text semantic features, generating probability distributions for each anomaly category, and generating video anomaly classification results based on the probability distributions.

[0019] Thirdly, this application provides an electronic device, comprising: a processor adapted to execute a computer program; and a computer-readable storage medium storing a computer program, wherein when executed by the processor, the computer program implements the method described in any one of the first aspects.

[0020] Fourthly, this application provides a computer-readable storage medium for storing a computer program that, when run on a computer, causes the computer to perform the method described in any of the first aspects.

[0021] Fifthly, this application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the method described in any of the first aspects.

[0022] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0023] The accompanying drawings, which are incorporated herein and form part of this specification, illustrate one or more embodiments of the present application and, together with the description, serve to explain the principles of the present application and to enable those skilled in the art to make and use the present application.

[0024] Figure 1 This is a first flowchart illustrating an exemplary traffic anomaly detection method according to some embodiments of this application.

[0025] Figure 2This is a second flowchart illustrating an exemplary traffic anomaly detection method according to some embodiments of this application.

[0026] Figure 3 This is a flowchart illustrating an exemplary data preprocessing method according to some embodiments of this application.

[0027] Figure 4 This is a schematic diagram of a spatiotemporal memory Mamba network architecture based on an exemplary U-Net architecture according to some embodiments of this application.

[0028] Figure 5 This is a schematic diagram of the structure of an exemplary spatiotemporal Mamba module according to some embodiments of this application.

[0029] Figure 6 This is a schematic diagram of the structure of an exemplary spatial VSS block according to some embodiments of this application.

[0030] Figure 7 This is a schematic diagram of an exemplary SSM2D scanning method according to some embodiments of this application.

[0031] Figure 8 This is a schematic diagram of the structure of an exemplary time VSS block according to some embodiments of this application.

[0032] Figure 9 This is a schematic diagram of an exemplary SSM1D scanning method according to some embodiments of this application.

[0033] Figure 10 This is a flowchart illustrating an exemplary keyframe video sequence generation method according to some embodiments of this application.

[0034] Figure 11 This is a schematic diagram of an exemplary clip network framework employing wavelet transform according to some embodiments of this application.

[0035] Figure 12 This is a flowchart illustrating an exemplary method for generating video anomaly classification results according to some embodiments of this application.

[0036] Figure 13 This is a schematic diagram of the structure of an exemplary text encoder according to some embodiments of this application.

[0037] Figure 14 This is a schematic diagram of the structure of an exemplary video frame encoder according to some embodiments of this application.

[0038] Figure 15 This is a schematic diagram of the module connections of an exemplary detection device according to some embodiments of this application.

[0039] Figure 16 This is a schematic diagram of the structure of an exemplary electronic device according to some embodiments of this application. Detailed Implementation

[0040] To facilitate understanding of the technical solutions provided in this application, the relevant terms are explained below.

[0041] It should be noted that the terminology used in the implementation section of this application is only for explaining the embodiments of this application and is not intended to limit this application.

[0042] For example, the term "and / or" in this article simply describes the relationship between related objects, indicating that three relationships can exist. For instance, A and / or B can represent: A alone, A and B simultaneously, and B alone. The term "at least one" simply describes the combination relationship of listed objects, indicating that one or more can exist. For instance, at least one of the following: A, B, C can represent the following combinations: A alone, B alone, C alone, A and B simultaneously, A and C simultaneously, B and C simultaneously, and A, B, and C simultaneously. The term "multiple" refers to two or more. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0043] For example, the term "correspondence" can indicate a direct or indirect correspondence between two things, or an association between them, or a relationship of instruction and being instructed, configuration and being configured, etc. The term "instruction" can be direct, indirect, or indicate an association. For example, A instructing B can mean A directly instructs B, for example, B can be obtained through A; it can also mean A indirectly instructs B, for example, A instructs C, B can be obtained through C; or it can mean an association between A and B. The terms "predefined" or "preconfigured" can refer to pre-stored codes, tables, or other relevant information that can be used for instruction in the device, or it can refer to something agreed upon by a protocol. "Protocol" can refer to standard protocols in the field. The term "when..." can be interpreted as "if," "when," or "in response," etc. Similarly, depending on the context, the phrases "if determined" or "if detected (the condition or event of the statement)" can be interpreted as "when determined" or "in response to determined" or "when detected (the condition or event of the statement)" or "in response to detected (the condition or event of the statement)" and similar descriptions. The terms "first," "second," "third," "fourth," "A," "B," etc., are used to distinguish different objects, not to describe a specific order. The terms "includes" and "has," and any variations thereof, are intended to cover non-exclusive inclusion.

[0044] Currently, in practical applications of traffic anomaly detection for long-form videos of highways, existing technical solutions still face many serious challenges, including:

[0045] Firstly, in real-world traffic monitoring scenarios, abnormal events such as traffic accidents, road damage, and debris spills occur at extremely low frequencies, exhibiting a typical long-tail distribution. This results in a severe shortage of effectively labeled abnormal data required for model training. Existing deep learning models typically rely heavily on massive amounts of data, making it difficult to learn robust feature representations when abnormal data is scarce. This easily leads to overfitting, resulting in a persistently high false negative rate in practical applications. Given the constraint of extremely limited abnormal data, how to overcome the data bottleneck through methods such as few-sample learning and weakly supervised learning, extract generalizable abnormal features, and ensure that the monitoring system can accurately identify rare traffic events in complex environments over a long period is a core scientific problem that urgently needs to be solved in the current information transformation of the intelligent transportation field.

[0046] Secondly, traffic monitoring long videos, as non-stationary time series, have constantly changing data distribution in real-world scenarios, and the correlation between input and output will drift accordingly. In anomaly detection tasks, false anomalies at certain timestamps may not be real anomalies on a larger scale, leading to a large number of false alarms. In addition, long videos contain massive amounts of continuous frames and complex temporal information between events. How to quickly capture the long-term changes of traffic events from redundant information and solve the problem of computational time consumption in the detection process is a challenge to achieving accurate monitoring and real-time early warning.

[0047] Thirdly, highway monitoring scenarios are highly diverse and unpredictable, often leading to unique and unusual events such as road damage, illegal lane occupation, and animal crossings. Since these rare events often lack clear annotations in existing training datasets, traditional closed-domain models struggle to effectively understand and automatically detect them. The frequent occurrence of these unknown anomalies seriously affects the accuracy of traffic safety monitoring.

[0048] Therefore, to solve the above-mentioned technical problems, this application provides a traffic video abnormal event detection method 100. This detection method 100 aims to adopt a two-stage architecture, without relying on massive anomaly annotation data, and can achieve accurate discovery and qualitative classification of various rare traffic events in long videos of complex road networks while effectively reducing computation time. It mainly includes two parts: a frame prediction model using unsupervised learning and a bimodal frame classification model using contrastive learning. The frame prediction model uses a lightweight network to model normal traffic conditions, achieving initial screening of anomalies with extremely low computational cost; the bimodal frame classification model uses a visual-language neural network to make specific category determinations on the initially screened abnormal segments.

[0049] Figure 1The illustration shows a first flowchart of an exemplary traffic anomaly detection method according to some embodiments of this application; Figure 2 The diagram shows a second flowchart of an exemplary traffic anomaly detection method according to some embodiments of this application.

[0050] refer to Figure 1 and Figure 2 As shown. The detection method 100 includes at least the following steps 101 to 104.

[0051] 101: Perform data preprocessing on the acquired traffic monitoring video stream to convert continuous video into a standardized video block sequence. The preprocessing includes one or more of the following: decoding and frame extraction, spatial scaling, image normalization, temporal segmentation and cropping, and tensor encapsulation and output processing.

[0052] This data preprocessing is used to convert the original traffic monitoring video stream into a unified data format suitable for subsequent deep learning model input, thereby reducing the impact of data distribution differences on model training and inference and improving model processing efficiency.

[0053] Figure 3 A flowchart illustrating an exemplary data preprocessing method according to some embodiments of this application is shown.

[0054] refer to Figure 3 As shown. Specifically, in step 101, the acquired traffic monitoring video stream is preprocessed to convert the continuous video into a standardized video block sequence, including at least the following steps 101a to 101e.

[0055] 101a: The acquired traffic monitoring video stream (such as MP4 or AVI format) is decoded and continuous RGB image frames are extracted according to the preset frame rate. The original continuous dynamic video is transformed into an ordered temporal set of spatial static images, and an image temporal set is constructed.

[0056] Decoding refers to the process of restoring compressed video data to its original image frame data, which can be achieved using existing video decoding tools or libraries (such as FFmpeg, OpenCV, etc.). Frame rate refers to the number of video frames sampled per unit time. The preset frame rate value is set by the designer according to the actual application requirements, such as 10fps, 15fps, or 30fps, to balance computational efficiency and temporal resolution. The RGB image frame refers to color image data containing red, green, and blue channels, used to preserve complete visual information. Thus, the video data is converted from compressed format to a standard image sequence.

[0057] 101b: Perform a spatial scale unification operation on each frame in the temporal set of images, scaling the image resolution to a preset size to eliminate resolution differences between different video sources.

[0058] The uniformity of spatial scale refers to the uniform scaling of images of different sizes to make them have the same width and height. This scaling process can be achieved by methods such as bilinear interpolation, bicubic interpolation, or nearest neighbor interpolation. The preset size is set by the designer according to actual needs. In this application, the preset size is set to 224×224 pixels, which is consistent with the input size of mainstream deep learning models and is conducive to improving model compatibility and computational efficiency.

[0059] 101c: Normalize each frame in the scaled image time series to reduce color distribution differences caused by different acquisition devices and lighting conditions by unifying the pixel distribution range and color space characteristics.

[0060] The normalization process involves mapping pixel values ​​to a uniform numerical range (e.g., [0,1] or [-1,1]) and then standardizing them by subtracting the mean and dividing by the standard deviation. Normalization can be performed based on preset mean and variance parameters (e.g., statistical values ​​from the ImageNet dataset) to improve the convergence speed of model training. This effectively reduces the impact of factors such as changes in lighting and camera differences on model performance.

[0061] 101d: The normalized image frame is segmented along the time dimension and divided into multiple short video segments of fixed length. Each short video segment contains a preset number of consecutive frames and is segmented using a non-overlapping sliding window method, transforming the long video into multiple local spatiotemporal blocks that are easy for the model to process.

[0062] The non-overlapping sliding window refers to the fact that adjacent video segments do not share frame data, thereby reducing data redundancy. The preset number is set by the designer according to actual needs. In this application, the exemplary reference setting is that each short video segment contains 16 frames, that is, each video segment contains data of 16 consecutive time steps. Thus, the long video is divided into multiple local spatiotemporal units, which is beneficial for the subsequent model to model local dynamic features and significantly reduces the data scale of a single calculation.

[0063] 101e: Perform dimensionality reorganization and tensor encapsulation on each short video segment to generate a sequence of video blocks in multidimensional tensor form that meets the requirements of deep learning frameworks. Each video block sequence includes at least channel dimension, temporal dimension, and spatial dimension information.

[0064] Here, the tensor refers to a multidimensional array data structure used to represent the input data of the deep learning model; the dimensions of the video block sequence include at least the channel dimension (C), the time dimension (T), the spatial height (H), and the spatial width (W). In this application, the dimensions of each video block sequence are exemplarily set to [3, 16, 224, 224], where 3 represents the three RGB color channels, 16 represents the number of time frames, and 224×224 represents the spatial resolution; thus, a standardized video block sequence is generated and output as the input data for the subsequent spatiotemporal memory Mamba model.

[0065] 102: Based on the video block sequence, a spatiotemporal memory Mamba frame prediction model using unsupervised learning is constructed and trained to model normal traffic behavior and calculate the prediction error based on the difference between the predicted frame and the real frame. This prediction error is used to achieve coarse-grained detection of abnormal events and localization of abnormal segments, generating abnormal spatiotemporal features.

[0066] Among them, the abnormal spatiotemporal features refer to the video features corresponding to the high-abnormality score regions obtained by filtering through frame prediction errors, which include temporal location information and spatial feature representation.

[0067] Figure 4 A schematic diagram of a spatiotemporal memory Mamba network architecture of an exemplary U-Net architecture according to some embodiments of this application is shown.

[0068] refer to Figure 4 As shown. Specifically, in step 102, a spatiotemporal memory Mamba frame prediction model employing unsupervised learning is constructed and trained based on the video block sequence, including at least steps 102a to 102d.

[0069] 102a: Construct a spatiotemporal memory Mamba network with a symmetric encoder and decoder structure. The encoder is used to perform hierarchical feature compression and abstract representation on the input video block sequence to obtain high-level semantic features. The decoder is used to progressively upsample the compressed features to restore spatial resolution and output the predicted frame.

[0070] For example, this application uses a U-Net architecture spatiotemporal memory Mamba network to carry out coarse-grained detection of abnormal traffic behavior based on long videos, and learns and predicts the temporal and spatial features of continuous video frames. The model takes 16 consecutive video images as input, divides each input frame image into multiple non-overlapping image sub-blocks (e.g., 4×4 pixels) through image patch embedding, and maps each image block to 64-channel features. The features after patch embedding are then fed into the encoder.

[0071] The U-Net structure refers to a network structure in which features are fused between the encoder and decoder through skip connections to preserve shallow details. Mamba is a sequence modeling structure based on the State Space Model (SSM), which can model long-range dependencies while maintaining linear computational complexity, and is suitable for long sequence video modeling tasks.

[0072] 102b: Construct a normal behavior feature memory at the encoder output. By performing similarity retrieval and reconstruction on the input features, the ability to enhance the expression of normal pattern features and suppress the reconstruction of abnormal features is enhanced.

[0073] Specifically, the encoder consists of multiple stages (e.g., four stages), each composed of spatiotemporal Mamba blocks, which reduce spatial resolution and improve feature representation through patch merging. At the encoder output, a normal behavior feature memory bank is constructed to store typical feature patterns learned from normal traffic videos. Thus, the encoder output features are filtered and input into the feature memory bank to obtain reconstructed features, which are then residually fused with the original features before being sent to the decoder.

[0074] Here, the memory bank refers to a feature storage structure containing multiple feature vectors (memory items) used to represent the feature prototypes of normal behavior; the similarity retrieval refers to selecting the most similar feature by calculating the distance between the input feature and the memory item (e.g., Euclidean distance or cosine similarity); the feature reconstruction refers to reconstructing the input feature using the retrieved memory features, thereby enhancing the expression of normal patterns and suppressing abnormal features.

[0075] 102c: The difference between the predicted frame and the real frame is calculated as the prediction error and mapped to the anomaly score. When the anomaly score is greater than the preset threshold, it is determined that there is abnormal behavior at the corresponding time position, and the corresponding features are extracted as abnormal spatiotemporal features.

[0076] The decoder structure is symmetrical to the encoder. It gradually restores the feature space resolution through patch expanding and finally outputs the predicted frame. During the inference stage, the predicted frame is compared pixel by pixel with the real frame, the prediction error is calculated, and this prediction error is mapped to an anomaly score to complete the video anomaly detection. The prediction error is calculated using, but is not limited to, mean squared error (MSE), absolute error (MAE), or structural similarity index (SSIM). The anomaly score is a normalized or weighted result of the prediction error, used to represent the degree of anomaly. When the anomaly score is greater than a preset threshold, it is determined that there is abnormal behavior at the corresponding time position. Therefore, the features corresponding to the region with a high anomaly score are extracted as anomaly spatiotemporal features for subsequent anomaly classification processing.

[0077] 102d: During model training, a multi-constraint joint loss function based on unsupervised learning is used to optimize the model parameters. This loss function includes at least pixel-level reconstruction loss, gradient difference loss, clustering and compaction loss, and feature separation loss. In terms of iterative updates of network parameters, the AdamW optimizer is used to perform gradient descent.

[0078] For example, in the training of the spatiotemporal memory Mamba frame prediction model, unsupervised learning is performed using a self-built highway normal driving video dataset, and a multi-constraint joint continuous spatial regression and metric learning total loss function is adopted. This total loss function is: .

[0079] in For pixel-level reconstruction loss, mean square error is used as a regression constraint to minimize the absolute numerical difference between the predicted frame and the real frame in global pixel color and brightness. For gradient difference loss, an image spatial gradient-based method is used. or The norm is calculated to constrain the consistency between the predicted image and the real image in high-frequency structural features such as edges and textures. To achieve the clustering compact loss, it calculates the mean square error of the current query feature to the nearest neighbor memory key in the memory, thereby prompting the extracted normal spatiotemporal features to cluster tightly towards the normal prototype pattern in the memory. For feature separation loss, triplet margin loss is adopted, aiming to effectively separate the current feature from other unrelated memory items in the memory bank in the feature space, thereby ensuring the diversity of memory slot storage patterns. For iterative updates of network parameters, the AdamW optimizer is selected to perform gradient descent. While ensuring fast network convergence, its decoupled weight decay mechanism effectively suppresses the overfitting phenomenon that easily occurs in spatiotemporal memory Mamba networks in complex video feature spaces.

[0080] Unsupervised learning refers to training a model without relying on manual labeling, but rather through the inherent structure of the data itself. The training data is preferably highway surveillance video that does not contain abnormal events. Mean squared error (MSE) is the average of the squared differences between corresponding pixels, used to measure numerical deviation. Image gradient refers to the rate of change of pixels in space, usually calculated using the Sobel operator. L1 norm is the sum of absolute values, better preserving edge information. Memory bank refers to the feature set storing prototypes of normal behavioral characteristics. Nearest neighbors are determined by feature distance (such as Euclidean distance). Triples include anchor points, positive samples, and negative samples. The optimization objective is to shorten the distance between the Anchor and the Positive, while simultaneously increasing the distance between the Anchor and the Negative. AdamW is an improved adaptive gradient optimization algorithm that enhances generalization ability by decoupling weight decay and gradient updates. Its learning rate can be set to... The range and weight decay coefficient can be set to... .

[0081] Figure 5 A schematic diagram of the structure of an exemplary spatiotemporal Mamba module according to some embodiments of this application is shown.

[0082] refer to Figure 5 As shown. Specifically, the encoder and decoder contain multiple spatiotemporal Mamba blocks, each including a spatial VSS block and a temporal VSS block. The spatiotemporal Mamba block adopts a two-stream serial modeling structure across space and time, which includes: converting the input video block sequence into a first feature by performing a first-dimensional reshaping operation, fusing the batch dimension and the temporal dimension; performing normalization processing on the first feature sequentially and inputting it into the spatial VSS block for spatial feature extraction, and introducing a residual connection mechanism to achieve the fusion of the original feature and the spatial feature, generating a fused spatial feature; converting the fused spatial feature into a second feature with a temporal dimension as the sequence length by performing a second-dimensional reshaping operation; and performing normalization processing on the second feature and inputting it into the temporal VSS block to extract temporal dependency features, and enhancing key temporal dependency features through nonlinear mapping and channel recalibration mechanisms.

[0083] The VSS (Visual State Space) block is a visual modeling module that extends the state space model and is used to capture long-range dependencies in the spatial or temporal dimensions.

[0084] Specifically, the spatiotemporal Mamba block adopts a serially cascaded spatial and temporal dual-stream structure to decouple and extract spatiotemporal features from high-dimensional video stream data. The input video block sequence is first recombined into (B*T,H,W,C) through a dimension reshaping operation to fuse the batch and temporal dimensions, enabling the network to focus on independent modeling of spatial features within a single frame. The reshaped first feature is then normalized in the first layer to stabilize the data distribution, and then fed into the spatial VSS block to extract deep spatial dependencies. To enhance the flow of feature information and preserve the original representation, a global residual mechanism is introduced, which performs element-wise addition of the original input video block sequence and the output of the spatial VSS block, thereby achieving cross-layer fusion of spatial features. The fused spatial features are then passed through the second dimension reshaping unit to transform and flatten the tensor dimension into (B*H*W,T,C), thus exposing the temporal dimension as the sequence length. After normalization, it is coupled into the temporal VSS block to extract temporal dependencies.

[0085] Figure 6 A schematic diagram of the structure of an exemplary spatial VSS block according to some embodiments of this application is shown; Figure 7 A schematic diagram of an exemplary SSM2D scanning method according to some embodiments of this application is shown.

[0086] refer to Figure 6 and Figure 7 As shown. Specifically, the spatial VSS block includes: performing deformable convolution operations on the input first feature to extract deformable adaptive local spatial features and inputting the local spatial features into a two-dimensional state space model; performing serialization processing along multiple spatial scanning directions to generate global spatial features; based on the global spatial features, using the Silu activation function to obtain nonlinear expressive power and randomly discarding some network branches through random regularization; fusing the global spatial features after random regularization and the global spatial features after the Silu activation function element-wise to generate a first fused feature; and, through local spatial feature enhancement processing including normalization, channel projection, residual convolution, and re-normalization, performing feature extraction again on the first fused feature and nonlinear mapping again through the Silu activation function, and performing a second cross-layer residual connection to complete multi-level feature modeling and fusion of local and global to local in the spatial dimension, so as to generate a fused spatial feature that simultaneously contains local details and global structure.

[0087] Among them, deformable convolution is used to extract local spatial features that adapt to deformation; two-dimensional state-space model (SSM2D) models global spatial dependencies by scanning along multiple spatial directions (e.g., up, down, left, and right); nonlinear activation (Silu function) is used to enhance feature representation ability; random regularization (Dropout) is used to prevent overfitting; state-space model is a type of model that models sequential dependencies by recursively modeling hidden states, and has the ability to efficiently model long-range dependencies.

[0088] Specifically, in the spatial VSS block, deformable local spatial features are first extracted through deformable convolution operations. After obtaining the deformable local spatial features, the model enters the two-dimensional state space model (SSM2D), and the features are serialized along the four scan directions of the spatial grid (scan path reference). Figure 7 As shown in the diagram, long-range global spatial features are obtained. These global spatial features are then sequentially passed through the Silu activation function to acquire nonlinear expressive power, and some network branches are randomly discarded through a random regularization layer. The randomly regularized features are then element-wise added to the features normalized by the Silu activation function layer to obtain the first fused feature V1. V1 undergoes secondary extraction of local spatial features through normalization, channel projection, residual convolution feature extraction, and re-normalization structure local spatial feature enhancement operations. After another nonlinear mapping through the Silu activation function, a second cross-layer residual connection is executed, completing multi-level feature modeling and fusion in the spatial dimension, from local to global to local.

[0089] Figure 8 A schematic diagram of the structure of an exemplary time VSS block according to some embodiments of this application is shown; Figure 9 A schematic diagram of an exemplary SSM1D scanning method according to some embodiments of this application is shown.

[0090] refer to Figure 8 and Figure 9 As shown. Specifically, the temporal VSS block includes: performing a depthwise separable convolution operation on the input second feature to extract local temporal variation features between adjacent video frames and inputting these local temporal variation features into a one-dimensional state space model; modeling long-range temporal dependencies in the video sequence by unfolding the sequence along the time axis and utilizing an implicit state transition mechanism; sequentially performing normalization processing and feedforward neural network transformation on the output features of the one-dimensional state space model and introducing a temporal channel attention mechanism, dynamically allocating channel weights through feature recalibration; and adjusting the channel dimension of the processed output features through linear mapping and sequentially processing them through layer normalization and Silu activation functions, then performing a residual connection with the initial input second feature to achieve the fusion of short-term local and long-term global features in the temporal dimension, generating temporal dependency features.

[0091] Among them, depthwise separable convolution is used to extract short-term local temporal features; one-dimensional state-space model (SSM1D) is used to model long-range dependencies in the time dimension; feedforward neural network (FFN) is used to improve the non-linear representation of features; channel attention mechanism is used to adaptively adjust the importance of different feature channels; residual connection is used to fuse original features and extracted features.

[0092] Specifically, within the temporal VSS block, the input second feature is first fed into a depthwise separable convolutional layer to extract local temporal variation features between adjacent video frames. These local temporal variation features are then fed into a one-dimensional state-space model (SSM1D), which expands the features along the time axis and utilizes its inherent implicit state transition mechanism to model long-range global temporal dependencies between motion patterns and deep temporal associations in the video sequence. To further enhance the semantic abstraction level of the temporal features, layer normalization and a feedforward neural network are introduced sequentially to perform nonlinear feature transformations in the channel dimension. Based on this, a temporal channel attention mechanism is connected in series, dynamically allocating channel weights through feature recalibration, adaptively enhancing high-value channels related to key actions, and suppressing non-critical redundant channels. Finally, the features are processed by adjusting the channel dimension through a linear layer and then sequentially processed by layer normalization and the Silu activation function. A residual connection is then performed with the initial input second feature to achieve deep fusion of short-term local and long-term global features in the temporal dimension.

[0093] Therefore, the spatiotemporal memory Mamba frame prediction model can effectively detect and locate abnormal behavior by learning the spatiotemporal distribution characteristics of normal traffic behavior without the need for abnormal samples to participate in training, thereby reducing the dependence on labeled data.

[0094] 103: Based on the spatiotemporal characteristics of the anomaly, non-uniform sampling and keyframe compression in the time domain are performed to generate a keyframe video sequence containing fine-grained anomaly categories.

[0095] Specifically, by using non-uniform sampling in the temporal domain and keyframe compression, a representative set of keyframes is extracted from long video sequences to retain key dynamic information of abnormal events while reducing computational complexity, thereby providing high-quality input data for subsequent fine-grained anomaly classification.

[0096] Figure 10 A flowchart illustrating an exemplary keyframe video sequence generation method according to some embodiments of this application is shown.

[0097] refer to Figure 10As shown. Specifically, in step 103, based on the spatiotemporal characteristics of the anomaly, non-uniform sampling and keyframe compression processing in the time domain are performed to generate a keyframe video sequence containing fine-grained anomaly categories, including at least the following steps 103a to 103c.

[0098] 103a: Based on the spatiotemporal characteristics of the anomaly, determine the time anchor frame where the anomaly event occurs and its neighboring time interval, and construct an asymmetric sampling window around the time anchor frame in the time dimension to obtain dense sampling frames and sparse sampling frames. A dense sampling strategy is used in the high-frequency time region close to the anomaly anchor point, and a sparse sampling strategy is used in the low-frequency time region far from the anomaly anchor point.

[0099] Among them, the time anchor frame refers to the key time frame in which the anomaly score reaches a local maximum value or exceeds a preset threshold, and is used to characterize the core occurrence time of the anomaly event; the neighborhood time interval refers to the time range that extends forward and backward around the time anchor frame, and its length can be determined according to the duration of the anomaly or the preset window size (e.g., ±8 frames or ±1 second); the asymmetric sampling window refers to the time window structure that uses different sampling densities before and after the time anchor.

[0100] Specifically, in the high-frequency region of time close to the time anchor frame (i.e., the stage of anomaly occurrence and drastic change), a dense sampling strategy is adopted, such as frame-by-frame sampling or small step-size sampling (step size of 1 or 2); in the low-frequency region of time far from the time anchor frame (i.e. the background stage before and after anomaly occurrence), a sparse sampling strategy is adopted, such as interval sampling (step size of 4, 8 or larger); thereby achieving fine preservation of the key anomaly dynamic process while compressing redundant background information.

[0101] 103b: The densely sampled frames and sparsely sampled frames are recombined in chronological order to generate a compressed video sequence containing a preset number of keyframes. This reduces the data size while maintaining the semantic integrity of the anomaly by compressing the anomaly-related information in the original long video into a compact keyframe sequence.

[0102] Keyframes refer to representative frames that can effectively characterize changes in video content or abnormal behavior. The preset number is set by the designer according to the model input requirements. In this application, the preset number is exemplarily referred to as 8 frames, 16 frames, or 32 frames. When the number of sampled frames is greater than the preset number, keyframes can be filtered by uniform downsampling or based on importance scores. When the number of sampled frames is insufficient, it can be supplemented by repeated sampling or interpolation.

[0103] In actual implementation, to ensure consistency of time information, time position encoding or frame index information can be added to the sampled frames to preserve the original timing relationship.

[0104] 103c: Perform uniform format conversion and tensor quantization on the compressed video sequence to generate a standardized keyframe video sequence suitable for subsequent bimodal classification model input.

[0105] The format conversion includes image size unification (e.g., 224×224), color space unification (RGB format), and data type conversion (e.g., uint8 to float32); tensor quantization refers to organizing the keyframe sequence into a multi-dimensional array structure, such as forming a [C,T,H,W] or [T,C,H,W] format, which is exemplarily set to [T,C,H,W]=[16,3,224,224] in this application; after the compressed video sequence undergoes unified format conversion and tensor quantization, it is further normalized or standardized to match the subsequent model input requirements.

[0106] 104: Input the keyframe video sequence and use a contrastive learning bimodal frame classification model. Calculate the matching relationship between the video and the anomaly category semantics through cross-modal alignment between visual features and text semantic features, generate the probability distribution of each anomaly category, and generate the video anomaly classification result based on the probability distribution.

[0107] Figure 11 A schematic diagram of an exemplary clip network framework employing wavelet transform is shown, representing some embodiments of this application. Figure 12 The diagram shows a flowchart illustrating an exemplary method for generating video anomaly classification results according to some embodiments of this application.

[0108] refer to Figure 11 and Figure 12 As shown. Specifically, in step 104, the keyframe video sequence is input into a bimodal frame classification model using contrastive learning. The matching relationship between the video and the semantics of the anomaly category is calculated through cross-modal alignment between visual features and text semantic features. The probability distribution of each anomaly category is generated, and the video anomaly classification result is generated based on the probability distribution. This includes at least the following steps 104a to 104d.

[0109] Figure 13 A schematic diagram of the structure of an exemplary text encoder according to some embodiments of this application is shown.

[0110] refer to Figure 13 As shown in Figure 104a: A text encoder employing conditional cue learning jointly encodes preset scene description cue words with the spatiotemporal features of the anomaly to generate a dynamic context cue vector. This dynamic context cue vector is then concatenated with the anomaly category description text to obtain high-dimensional text semantic features.

[0111] Among them, conditional cue learning refers to dynamically enhancing the input text by introducing learnable context vectors, thereby improving the model's adaptability to specific tasks; prompts are textual descriptions used to guide the model to understand the semantics of the task, such as "traffic accident", "abnormal parking", "pedestrian intrusion", etc.

[0112] Specifically, in the text encoder stage, a conditional cue learning strategy is employed to construct learnable contextual cue vectors. Then, scene description cue words are coupled and vectorized with the extracted spatiotemporal features of 32 suspected anomalies, generating a set of dynamic, learnable, continuous contextual cue vectors, denoted as... Then, the feature vectors of the anomaly category hint words are concatenated to the continuous context hint vectors through vector concatenation, denoted as... , By acquiring high-dimensional text features through a pre-trained clip text encoder, and then providing instance-level semantic guidance for subsequent cross-modal alignment through end-to-end fine-tuning training.

[0113] 104b: After normalizing the high-dimensional text semantic features and the multi-band visual features output by the video frame encoder, map them to a unified feature space, and calculate the cosine similarity between the two to generate a cross-modal similarity matrix.

[0114] Figure 14 A schematic diagram of the structure of an exemplary video frame encoder according to some embodiments of this application is shown.

[0115] refer to Figure 14 As shown. In step 104b, the video frame encoder includes: performing a two-dimensional discrete wavelet transform on each frame of the keyframe video sequence to decompose the original image into a low-frequency approximate sub-band and multiple high-frequency detail sub-bands, wherein the low-frequency sub-band represents the overall structural information of the image, and the high-frequency sub-band represents edge and texture detail information; performing cross-frequency domain filtering on frames in different time regions according to a time sampling strategy and fusing the frequency domain features generated by the cross-frequency domain filtering to form multi-frequency features that simultaneously contain spatial structural information and motion detail information; performing image block embedding and spatiotemporal position encoding processing on the multi-frequency features to convert them into serialized features and inputting them into a pre-trained clip image encoder to extract global low-frequency semantic features and local high-frequency semantic features respectively; and fusing the global low-frequency semantic features and local high-frequency semantic features through a multilayer perceptron to generate unified multi-band visual features.

[0116] Among them, the low-frequency subband (LL) represents the overall structural information of the image; the high-frequency subbands (TLH, THL, THH) represent the detailed texture information in the horizontal, vertical, and diagonal directions, respectively; wavelet transform is a time-frequency analysis method that can decompose signals in both the spatial and frequency domains simultaneously, and is suitable for capturing multi-scale features; cross-frequency domain filtering refers to establishing a coupling relationship between the time dimension and the spatial frequency domain to achieve selective information retention; in cross-frequency domain filtering, for the high-frequency temporal region (frames near the anomaly), the high-frequency subband information is retained and the low-frequency subband is suppressed to highlight local motion and anomaly details; for the low-frequency temporal region (background frames), the low-frequency subband information is retained and the high-frequency subband is suppressed to enhance the semantic meaning of scene structure; image patch embedding refers to dividing the image into fixed-size blocks and mapping them to vectors; spatiotemporal positional encoding refers to introducing temporal and spatial positional information into features; multilayer perceptron is a nonlinear mapping structure composed of multiple fully connected networks, used for feature fusion and dimensional transformation.

[0117] Specifically, in the video frame encoding stage, the initially located abnormal frames are used as reference anchors. Asymmetric hierarchical sampling in the temporal dimension is then performed on the original long video sequence. This involves dense sampling in the high-frequency temporal domain (the adjacent intervals before and after the abnormal frame) and sparse sampling in the low-frequency temporal domain (random sampling within the outer 1-second interval), thereby efficiently compressing the redundant long video into a 46-frame key sequence. Then, for the extracted 46-frame video sequence (initial dimensions (B, C, H, W)), a two-dimensional discrete wavelet transform is introduced for spatial-frequency decoupling, resulting in one low-frequency approximate subband (TLL) and three high-frequency detail subbands (horizontal). (TLH, vertical THL, diagonal THH); based on this, cross-frequency domain filtering is implemented to filter out the spatial low-frequency features of high-frequency temporal frames (to focus on local motion details), and to filter out the spatial high-frequency features of low-frequency temporal frames (to preserve global background semantics); then, the processed multi-frequency features are sent to the image block embedding and spatiotemporal location coding module to obtain serialized features. These serialized features are fed into the pre-trained clip image encoder to extract global low-frequency semantic features and local high-frequency semantic features respectively. Finally, the multilayer perceptron module completes the deep fusion of multi-frequency features to generate unified multi-frequency visual features.

[0118] After acquiring the aforementioned textual semantic features and the fused multi-band visual features, to achieve accurate alignment between visual and linguistic modalities, the cosine similarity between the video visual features and the textual semantic features is calculated to measure their matching degree. The calculation formula is as follows:

[0119] ,

[0120] Where A is the visual feature vector extracted by the video frame encoder. B is the text semantic feature vector obtained by the text encoder from the category cue words. ; n is the dimension of the feature vector, which is exemplarily 512-dimensional in this application; i is the index number of the i-th dimension (element) in the vector.

[0121] The similarity matrix obtained by the above formula is then normalized by the Softmax activation function and transformed into a classification confidence score. This probability distribution reflects the probability that the current input video segment belongs to the normal mode or various specific abnormal categories, thereby achieving end-to-end fine-grained anomaly classification.

[0122] 104c: Perform normalization mapping on the similarity matrix to convert it into the probability distribution of each anomaly category.

[0123] Normalization can be achieved using the Softmax function, which maps any real number to a probability value between 0 and 1, and ensures that the sum of the probabilities of all categories is 1; the generated probability distribution represents the confidence level of the input video belonging to each anomaly category.

[0124] 104d: Output video anomaly classification results based on this probability distribution.

[0125] In some examples of this application, the model is built on the PyTorch deep learning framework and is accelerated at the hardware level by using at least one computing device equipped with multiple graphics processing units (GPUs), such as a hardware platform containing three NVIDIA GTX 1080Ti GPUs combined with the CUDA parallel computing architecture. In terms of the overall training strategy, a two-stage training paradigm from coarse to fine is adopted: first, the spatiotemporal memory Mamba network is pre-trained unsupervised, and then the clip anomaly classification network with wavelet transform is fine-tuned end-to-end.

[0126] Among them, PyTorch is an open-source deep learning framework that supports dynamic graph mechanisms, which facilitates the flexible construction and debugging of model structures; CUDA is a programming platform and execution model for GPU parallel computing, which can significantly improve the efficiency of large-scale tensor operations.

[0127] Therefore, in the training of a video anomaly classification network based on the clip architecture of wavelet transform, a joint fine-tuning mechanism for text prompts and visual representations for few-shot scenarios is proposed. This improves the depth alignment of multimodal features in a specific anomaly detection task space by improving the training of a small amount of anomaly data with video-level annotations. At the text encoder, scene description prompts, high-dimensional spatiotemporal features extracted by the prior spatiotemporal memory Mamba encoder, and anomaly category labels are used as prior guiding conditions and jointly injected into the conditional prompt learning module to construct dynamic contextual prompt vectors. At the image encoder, a parameter-efficient fine-tuning strategy is adopted, freezing most of the network weights in the bottom and middle layers of the pre-trained image encoder, and only opening and training its terminal linear mapping layer and external feature fusion layer. During training, the cross-modal similarity score between the multi-band visual features output by the visual encoder and the dynamic prompt word features is calculated, and Softmax is used to map it to classification probability and calculate cross-entropy loss with the real label. Finally, through backpropagation of error, the open parameters of the visual encoder and the learnable text prompt vector are updated synchronously to achieve accurate visual-language alignment.

[0128] Among them, few-shot learning refers to training a model with a limited number of samples; fine-tuning refers to adjusting the parameters of a pre-trained model to adapt it to a specific task; frozen parameters refer to not updating its gradient during training; and cross-entropy loss is used to measure the difference between the predicted probability distribution and the true label.

[0129] Finally, the two-stage prediction results of the model are quantitatively compared with the real labels, and evaluation indicators such as the overall accuracy are calculated to assess the generalization ability and practical application effectiveness of the framework in long-term traffic video anomaly detection and classification tasks.

[0130] Therefore, the traffic video abnormal event detection method 100 of this application has the following beneficial effects:

[0131] To address the severe shortage of abnormal data samples in real traffic monitoring scenarios, this application employs the spatiotemporal memory Mamba for unsupervised training on a large number of normal driving videos on highways, establishing a prior for the normal driving distribution. It also combines a multi-constraint joint loss function to widen the decision boundary between the normal pattern prototype and the abnormal reconstruction features, overcoming the tendency of traditional deep learning models to overfit due to the scarcity of abnormal samples under long-tailed data distributions.

[0132] To address the issues of time-consuming detection and variable data structures caused by the dynamic evolution of long-term videos, this application proposes a two-stage video anomaly detection mechanism. In the first stage, a spatiotemporal memory state space model is used to perform rapid coarse-grained scanning and detection on massive long video streams to determine whether there are anomalies in the long video segments and accurately locate the core anchor frames where the anomalies occur. In the second stage, the long video is compressed into 46-frame short videos for fine-grained classification through pre-selection of spatiotemporal features.

[0133] This application addresses the lack of ability to identify unknown anomaly categories (such as illegal road occupancy, animal crossings, and rare events like scattered debris) caused by the extremely open nature of traffic scenarios. In the second stage, a clip network using wavelet transform is introduced for fine-grained classification of video anomalies. By leveraging the zero-shot and few-shot generalization and transfer learning capabilities of a large-scale visual-language model (VLM), rare anomaly categories can be automatically discovered and accurately classified in fine-grained terms using rich open-domain semantic prior knowledge from the text, without the need to predefine rare anomaly categories in the training set.

[0134] The preferred embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the embodiments described above. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the specific embodiments described above can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application.

[0135] It should also be understood that, in the various method embodiments of this application, the order of the processes mentioned above does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0136] thus, Figure 15 A schematic diagram of the module connections of an exemplary detection device according to some embodiments of this application is shown.

[0137] refer to Figure 15As shown, the detection device 200 of this application may include: a data processing module 201, used to preprocess the acquired traffic monitoring video stream, converting continuous video into a standardized video block sequence, wherein the preprocessing includes one or more of decoding and frame extraction, spatial scaling, image normalization, temporal segmentation and cropping, and tensor encapsulation and output processing; and a frame prediction module 202, used to construct and train a spatiotemporal memory Mamba frame prediction model using unsupervised learning based on the video block sequence, to model normal traffic behavior and calculate the prediction error based on the difference between the predicted frame and the real frame, and to use the prediction error... The system achieves coarse-grained detection of abnormal events and localization of abnormal segments, generating abnormal spatiotemporal features; video processing module 203 is used to perform non-uniform sampling and keyframe compression processing in the time domain based on the abnormal spatiotemporal features to generate a keyframe video sequence containing fine-grained abnormal categories; and an anomaly detection module 204 is used to input the keyframe video sequence into a bimodal frame classification model using contrastive learning, calculate the matching relationship between video and anomaly category semantics through cross-modal alignment between visual features and text semantic features, generate the probability distribution of each anomaly category, and generate video anomaly classification results based on the probability distribution.

[0138] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, they will not be repeated here. Specifically, the detection device 200 can correspond to the corresponding subject in the detection method 100 of the embodiments of this application, and each unit in the detection device 200 is for implementing the corresponding process in the detection method 100. For the sake of brevity, they will not be repeated here.

[0139] It should also be understood that the various units in the detection device 200 involved in the embodiments of this application are based on logical functional division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. Furthermore, these functions can also be implemented with the assistance of one or more other units. For example, some or all of the detection device 200 may be merged into one or more additional units. As another example, some units(s) in the detection device 200 can be further divided into multiple functionally smaller units to achieve the same operation without affecting the technical effects of the embodiments of this application. Furthermore, the detection device 200 may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0140] It should also be understood that the terms "module" or "unit" used in the embodiments of this application refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0141] For example, the detection device 200 involved in the embodiments of this application, and the method of the embodiments of this application, can be constructed and implemented by running a computer program (including program code) capable of executing the steps involved in the corresponding method on a general-purpose computing device including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable storage medium and loaded into an electronic device through the computer-readable storage medium. The computer program is used to implement the corresponding method of the embodiments of this application. In other words, the units mentioned above can be implemented in hardware, in software instructions, or in a combination of hardware and software. Specifically, the steps of the method embodiments in the embodiments of this application can be completed by the integrated logic circuits of the hardware in the processor and / or by the instructions in software. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software in the decoding processor. Optionally, the software can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The software in the memory can be run by a processor to perform the steps described in the method embodiments above.

[0142] Figure 16 A schematic diagram of the structure of an exemplary electronic device according to some embodiments of this application is shown.

[0143] refer to Figure 16As shown, the electronic device 300 includes at least a processor 310 and a computer-readable storage medium 320. The processor 310 and the computer-readable storage medium 320 can be connected via a bus or other means. The computer-readable storage medium 320 stores a computer program 321, which includes computer instructions. The processor 310 executes the computer instructions stored in the computer-readable storage medium 320. The processor 310 is the computing and control core of the electronic device 300, and is suitable for implementing one or more computer instructions, specifically for loading and executing one or more computer instructions to achieve a corresponding method flow or function.

[0144] As an example, processor 310 may also be referred to as a central processing unit (CPU). Processor 310 may include, but is not limited to: general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete component gate or transistor logic devices, discrete hardware components, etc.

[0145] As an example, the computer-readable storage medium 320 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device; optionally, it may also be at least one computer-readable storage medium located remotely from the aforementioned processor 310. Specifically, the computer-readable storage medium 320 includes, but is not limited to, volatile memory and / or non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0146] refer to Figure 16 As shown, the electronic device 300 may also include a transceiver 330.

[0147] The processor 310 can control the transceiver 330 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 330 may include a transmitter and a receiver. The transceiver 330 may further include antennas, and the number of antennas may be one or more.

[0148] It should be understood that the various components in the electronic device 300 are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus. It is worth noting that the electronic device 300 can be any type of electronic device with data processing capabilities; the computer-readable storage medium 320 stores first computer instructions; the processor 310 loads and executes the first computer instructions stored in the computer-readable storage medium 320 to implement the corresponding steps in the method embodiments of this application; in specific implementations, the first computer instructions in the computer-readable storage medium 320 are loaded and executed by the processor 310, and to avoid repetition, this will not be described further here.

[0149] According to another aspect of this application, embodiments of this application provide a chip. This chip can be an integrated circuit chip with signal processing capabilities, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The chip can also be referred to as a system-on-a-chip (SoC), system-on-a-chip (SoC), chip system, or system-on-chip, etc. This chip can be applied to various electronic devices capable of mounting chips, enabling the device with the chip mounted to execute the corresponding steps in the methods or logic block diagrams disclosed in the embodiments of this application. For example, the chip may be suitable for implementing one or more computer instructions, specifically suitable for loading and executing one or more computer instructions to achieve a corresponding method flow or corresponding function.

[0150] According to another aspect of this application, embodiments of this application provide a computer-readable storage medium (Memory). This computer-readable storage medium is a computer's memory device used to store programs and data. It is understood that the computer-readable storage medium here may include both built-in storage media within the computer and, of course, extended storage media supported by the computer. The computer-readable storage medium provides storage space that stores the operating system of an electronic device. This storage space contains computer instructions suitable for loading and execution by a processor. When these computer instructions are read and executed by the processor of the computer device, they cause the computer device to perform the corresponding steps in the methods or logic diagrams disclosed in the embodiments of this application.

[0151] According to another aspect of this application, embodiments of this application provide a computer program product or computer program. The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform corresponding steps in the methods or logic block diagrams disclosed in the embodiments of this application. In other words, when the solutions provided in this application are implemented using software, they can be implemented in whole or in part as a computer program product or computer program. The computer program product or computer program includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes of the embodiments of this application are run or the functions of the embodiments of this application are implemented.

[0152] It is worth noting that the computer involved in this application can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions involved in this application can be stored in a computer-readable storage medium, or can be transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0153] Those skilled in the art will recognize that the units and process steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. In other words, those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of protection of this application.

[0154] Finally, it should be noted that the above content is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the basic idea of ​​this application, and they should also be considered as the content disclosed in this application.

Claims

1. A method for detecting abnormal events in traffic videos, characterized in that, The method includes: The collected traffic monitoring video stream is preprocessed to convert continuous video into a standardized video block sequence, wherein the video block sequence includes at least channel dimension, time dimension and spatial dimension information; Based on the video block sequence, a spatiotemporal memory Mamba frame prediction model using unsupervised learning is constructed and trained to model normal traffic behavior and calculate the prediction error based on the difference between the predicted frame and the real frame. The prediction error is used to achieve coarse-grained detection of abnormal events and localization of abnormal segments, and to generate abnormal spatiotemporal features. Based on the aforementioned spatiotemporal characteristics of the anomalies, non-uniform sampling and keyframe compression in the time domain are performed to generate a keyframe video sequence containing fine-grained anomaly categories; and... The keyframe video sequence is input into a bimodal frame classification model using contrastive learning. The matching relationship between the video and the anomaly category semantics is calculated through cross-modal alignment between visual features and text semantic features. The probability distribution of each anomaly category is generated, and the video anomaly classification result is generated based on the probability distribution.

2. The method according to claim 1, characterized in that, The process of preprocessing the acquired traffic monitoring video stream, converting continuous video into a standardized sequence of video blocks, includes: The collected traffic monitoring video stream is decoded and continuous RGB image frames are extracted according to a preset frame rate to construct an image time series set. A spatial scale unification operation is performed on each frame of the image time series to scale the image resolution to a preset size in order to eliminate the resolution difference between different video sources; Normalization is performed on each frame of the scaled image time series to reduce color distribution differences caused by different acquisition devices and lighting conditions by unifying the pixel distribution range and color space characteristics. Each normalized frame is segmented along the time dimension into multiple fixed-length short video segments. Each short video segment contains a preset number of consecutive frames and is divided using a non-overlapping sliding window method; and, Each short video segment undergoes dimensionality reorganization and tensor encapsulation to generate a video block sequence in multidimensional tensor form that meets the requirements of deep learning frameworks. Each video block sequence includes at least channel dimension, temporal dimension, and spatial dimension information.

3. The method according to claim 1 or 2, characterized in that, The step of constructing and training a spatiotemporal memory Mamba frame prediction model using unsupervised learning based on the video block sequence includes: A spatiotemporal memory Mamba network with a symmetric encoder and decoder structure is constructed. The encoder performs hierarchical feature compression and abstract representation on the input video block sequence, while the decoder progressively restores the spatial resolution and generates prediction frames. A normal behavior feature memory is constructed at the encoder output. Similarity retrieval and reconstruction of input features are performed to enhance the representation of normal pattern features and suppress the reconstruction of abnormal features. Furthermore, the difference between the predicted frame and the real frame is calculated as a prediction error, which is then mapped to an anomaly score. When the anomaly score exceeds a preset threshold, abnormal behavior is determined to exist at the corresponding time location, and the corresponding features are extracted as anomalous spatiotemporal features. During model training, a multi-constraint joint loss function based on unsupervised learning is used to optimize the model parameters. The loss function includes at least pixel-level reconstruction loss, gradient difference loss, clustering compaction loss, and feature separation loss. Furthermore, for iterative updates of network parameters, gradient descent is performed using the AdamW optimizer.

4. The method according to claim 3, characterized in that, The encoder and decoder comprise multiple spatiotemporal Mamba blocks, each including a spatial VSS block and a temporal VSS block. The spatiotemporal Mamba blocks employ a two-stream serial modeling structure spanning both space and time, comprising: The input video block sequence is transformed into a first feature that fuses the batch and time dimensions through a first-dimensional reshaping operation. The first feature is normalized sequentially and input into the spatial VSS block for spatial feature extraction. A residual connection mechanism is introduced to achieve the fusion of the original feature and the spatial feature, generating the fused spatial feature. The fused spatial features are transformed into a second feature with a time-series length through a second-dimensional reshaping operation; and... The second feature is normalized and input into a time VSS block to extract temporal dependent features, and key temporal dependent features are enhanced through nonlinear mapping and channel recalibration mechanisms.

5. The method according to claim 4, characterized in that, The space VSS block includes: A deformable convolution operation is performed on the first input feature to extract deformable adaptive local spatial features, and the local spatial features are input into a two-dimensional state space model. The global spatial features are generated by serialization along multiple spatial scanning directions. Based on the global spatial features, the Silu activation function is applied to obtain nonlinear expressive power, and some network branches are randomly discarded through random regularization. The global spatial features after random regularization and those after the Silu activation function are then fused element-wise to generate a first fused feature; and... By performing local spatial feature enhancement processing including normalization, channel projection, residual convolution and re-normalization, the first fused feature is extracted again and then nonlinearly mapped by the Silu activation function. A second cross-layer residual connection is then performed to complete multi-level feature modeling and fusion in the spatial dimension, from local to global to local, so as to generate spatial features that contain both local details and global structure.

6. The method according to claim 4 or 5, characterized in that, The time VSS block mentioned therein includes: A depthwise separable convolution operation is performed on the second input feature to extract local temporal variation features between adjacent video frames. The local temporal variation features are then input into a one-dimensional state space model. The sequence is unfolded along the time axis and the long-range temporal dependencies in the video sequence are modeled using an implicit state transition mechanism. The output features of the one-dimensional state-space model are sequentially normalized and transformed using a feedforward neural network, and a temporal channel attention mechanism is introduced to dynamically allocate channel weights through feature recalibration; and, The processed output features are adjusted for channel dimensions through linear mapping and then processed sequentially through layer normalization and Silu activation functions. Finally, they are residually connected with the second feature of the initial input to achieve the fusion of short-term local and long-term global features in the time dimension, generating time-dependent features.

7. The method according to claim 6, characterized in that, The step of generating a keyframe video sequence containing fine-grained anomaly categories by performing non-uniform sampling and keyframe compression processing in the time domain based on the aforementioned spatiotemporal features includes: Based on the abnormal spatiotemporal characteristics, the time anchor frame of the abnormal event and its neighboring time interval are determined, and an asymmetric sampling window is constructed around the time anchor frame in the time dimension to obtain dense sampling frames and sparse sampling frames. A dense sampling strategy is used in the high-frequency time region close to the abnormal anchor point, and a sparse sampling strategy is used in the low-frequency time region far from the abnormal anchor point. The densely sampled frames and sparsely sampled frames are reassembled in chronological order to generate a compressed video sequence containing a preset number of keyframes; and, The compressed video sequence is subjected to uniform format conversion and tensor quantization to generate a standardized keyframe video sequence suitable for subsequent bimodal classification model input.

8. The method according to claim 1 or 7, characterized in that, The process of inputting the keyframe video sequence into a bimodal frame classification model employing contrastive learning, calculating the matching relationship between video and anomaly category semantics through cross-modal alignment between visual features and textual semantic features, generating probability distributions for each anomaly category, and generating video anomaly classification results based on the probability distributions includes: A text encoder employing conditional cue learning jointly encodes preset scene description cue words with the abnormal spatiotemporal features to generate a dynamic context cue vector, and then concatenates the dynamic context cue vector with the abnormal category description text to obtain high-dimensional text semantic features. The high-dimensional text semantic features and the multi-band visual features output by the video frame encoder are normalized and mapped to a unified feature space, and the cosine similarity between the two is calculated to generate a cross-modal similarity matrix. The similarity matrix is ​​normalized and mapped to convert it into a probability distribution for each anomaly category; and, The video anomaly classification results are output based on the probability distribution.

9. The method according to claim 8, characterized in that, in, The video frame encoder includes: A two-dimensional discrete wavelet transform is performed on each frame of the keyframe video sequence to decompose the original image into a low-frequency approximate sub-band and multiple high-frequency detail sub-bands, wherein the low-frequency sub-bands represent the overall structural information of the image and the high-frequency sub-bands represent edge and texture detail information. According to the time sampling strategy, cross-frequency domain filtering is performed on frames in different time regions, and the frequency domain features generated by cross-frequency domain filtering are fused to form multi-frequency features that simultaneously contain spatial structure information and motion detail information. The multi-frequency features are processed by image block embedding and spatiotemporal location encoding, converted into serialized features, and input into a pre-trained clip image encoder to extract global low-frequency semantic features and local high-frequency semantic features, respectively; and, The global low-frequency semantic features and local high-frequency semantic features are fused by a multilayer perceptron to generate unified multi-band visual features.

10. A traffic video anomaly event detection device for implementing the method of any one of claims 1-9, characterized in that, include: The data processing module is used to preprocess the collected traffic monitoring video stream, converting continuous video into a standardized video block sequence, wherein the video block sequence includes at least channel dimension, time dimension and spatial dimension information; The frame prediction module is used to construct and train a spatiotemporal memory Mamba frame prediction model using unsupervised learning based on the video block sequence, so as to model normal traffic behavior and calculate the prediction error based on the difference between the predicted frame and the real frame. The prediction error is used to realize coarse-grained detection of abnormal events and abnormal segment localization, and generate abnormal spatiotemporal features. The video processing module is configured to perform non-uniform sampling and keyframe compression processing in the time domain based on the aforementioned abnormal spatiotemporal characteristics, generating a keyframe video sequence containing fine-grained anomaly categories; and, The anomaly detection module is used to input the keyframe video sequence into a bimodal frame classification model using contrastive learning, calculate the matching relationship between the video and the semantics of the anomaly category through cross-modal alignment between visual features and text semantic features, generate the probability distribution of each anomaly category, and generate the video anomaly classification result based on the probability distribution.

Images