A weakly supervised video anomaly detection method based on prompt learning knowledge enhancement

Abstract

The application discloses a weakly supervised video anomaly detection method based on prompt learning knowledge enhancement, and belongs to the technical field of video intelligent analysis. A video side is given an abnormal scene video, video sequence features and audio sequence features are obtained through a feature extraction network, then a relatively complete feature aggregation network which has been trained is input to perform multi-modal feature aggregation, and an abnormal score is obtained through a score prediction network. In the text representation step based on prompt learning, an abnormal video label is used to construct a prompt template through a knowledge graph, normal labels are used to perform semantic expansion through a plurality of learnable parameters, and cross-modal alignment is performed with the video side, so that the video side features are close to different normal semantics. External information is introduced to perform knowledge enhancement, the video positive anomaly boundary is learned, the detection performance is improved, and finally multi-task joint optimization is performed through different loss functions to position an abnormal video segment.

Description

Technical Field

[0001] This invention belongs to the field of video intelligent analysis technology, and relates to a weakly supervised video anomaly detection method based on cue-based knowledge enhancement. Background Technology

[0002] Video anomaly detection is a crucial area of ​​intelligent video analytics. Its purpose is to identify abnormal behaviors and events in video and accurately pinpoint their timeframe. In recent years, surveillance cameras have been widely used, which is of significant practical importance for combating crime and maintaining social stability. However, effectively monitoring this massive amount of video data and maintaining a healthy online and offline environment has become a challenging task. Due to the diversity of abnormal events, video anomaly detection technology shows broad application prospects in various fields such as intelligent surveillance and security, traffic management, industrial production, and online content supervision, while also facing considerable challenges.

[0003] With the groundbreaking advancements of deep learning in computer vision, deep learning-based video anomaly detection technology has become a core paradigm. By constructing deep neural network models, complex spatiotemporal features can be automatically learned, significantly improving the accuracy and generalization ability of anomaly detection. One classification method is based on the degree of supervision, dividing it into unsupervised learning, semi-supervised learning, weakly supervised learning, and fully supervised learning. Fully supervised learning methods for video anomaly detection require explicitly labeling each frame or segment of the video as "normal" or "abnormal," which is time-consuming, labor-intensive, and overly reliant on labeling; therefore, research in this area is relatively limited. Unsupervised methods do not require any labeling, and their advantage is that detection can be continuously trained without human intervention, but this also leads to a decrease in performance. Semi-supervised methods rely on a single normal sample to construct the representation space during the training phase, and during testing, samples deviating from the normal pattern are considered anomalies. This type of method leads to poor adaptability to unseen samples, increased false trigger rates, and higher false positive rates. In contrast, video anomaly detection algorithms based on weakly supervised learning jointly model discriminative features of normal and anomalous patterns during the training phase. They only require weak video-level annotations as supervision signals to drive the model to learn the potential distribution differences between normal and anomalous behaviors, and achieve frame-level anomaly event localization during the testing phase. Compared to other methods, they offer advantages such as low cost, high performance, and wide applicability. Therefore, this invention employs a weakly supervised approach.

[0004] In semantic space, normal and abnormal events exhibit significant intra-class variance, while some abnormal and normal events show relatively small inter-class variance. For example, explosions and fighting, both belonging to abnormal behaviors, have vastly different visual representations; crossing the road when the traffic light is green and crossing when the traffic light is red have very similar feature representations, yet one is abnormal and the other normal. Furthermore, the number of normal scenes is countless, and different normal scenes also exhibit feature differences. Meanwhile, in the datasets commonly used for weakly supervised video anomaly detection tasks, the number of abnormal video segments is far less than that of normal segments, and when there is a high degree of similarity between normal and abnormal segments, it poses a greater challenge to improving the robustness of the model. In the weakly supervised video anomaly detection framework, the supervision signal for normal video samples (labeled 0) has high reliability because it only contains normal video segments; while the supervision signal for abnormal video samples (labeled 1) contains inherent noise, as such videos essentially contain both normal and abnormal segments. This labeling paradigm leads to two key problems: first, normal segments in abnormal videos can introduce misleading supervision signals; second, existing methods overemphasize anomaly feature mining while neglecting normal pattern modeling. Learning normal patterns is just as valuable as learning anomalous knowledge for constructing a discriminative feature space. Introducing anomalous knowledge while simultaneously learning normal video features can optimize feature representation, enabling the model to extract more discriminative features.

[0005] While existing methods consciously utilize normal video features, they lack utilization of textual modalities. However, CLIP, with its unprecedented performance in many downstream vision-linguistic tasks, leverages its powerful text representation capabilities to better align visual and linguistic features. These methods represent normal text as a single vector, but real-world normal scenes are complex and diverse. Blindly clustering them into one category loses boundary information between different normal scenes, hindering the learning of normal scenes and even introducing more noise. Bringing all normal visual features close to a single normal semantic center may increase the learning difficulty, as normal scenes have multiple semantic centers, each focusing on different aspects, but lacking clear label classifications like anomalies. Therefore, a simple and effective method for learning normal scenes is currently lacking. Summary of the Invention

[0006] This invention addresses the problems of existing technologies by providing a weakly supervised video anomaly detection method based on cue-based knowledge enhancement.

[0007] A weakly supervised video anomaly detection method based on cue-based knowledge enhancement includes the following steps: feature extraction and aggregation; score prediction; text representation based on cue-based learning; cross-modal alignment; and multi-task joint optimization.

[0008] The process also includes the following steps: First, given an abnormal scene video, the video and audio sequence features are obtained through a feature extraction network. Then, these features are input into a well-trained feature aggregation network for multimodal feature aggregation, and an abnormality score is obtained through a score prediction network. Based on the text representation step of cue learning, the abnormal video labels are used to construct cue templates through a knowledge graph. The normal labels are semantically expanded through multiple learnable parameters and aligned with the video side across modalities, so that the video side features move closer to different normal semantics. Knowledge enhancement is performed by introducing external information to learn the boundaries between positive and abnormal video features and improve detection performance. Finally, multi-task joint optimization is performed through different loss functions to locate abnormal video segments.

[0009] The advantages of this invention are: it proposes a learnable multi-head normal prompt semantic generation mechanism, which obtains normal text features at different semantic centers through the CLIP encoder, and selects the normal text feature with the highest similarity to each normal visual feature for cross-modal alignment, clustering complex normal scenes into different semantic centers, which can better learn normal features, thereby promoting the boundary discrimination of abnormal features and improving the model detection performance.

[0010] Experiments demonstrate the effectiveness of this invention on the XD-Violence and UCF-Crime datasets. On the XD-Violence dataset, it achieves an average precision of 86.85%, with the false alarm rate decreasing from 0.1% in existing methods to 0.01%. On the UCF-Crime dataset, it achieves an accuracy of 87.75%, representing a 0.99% improvement over existing methods. This demonstrates that this invention can leverage cue learning to enhance the network's ability to represent normal scenes, thereby improving the robustness of the model for imbalanced tasks. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. As shown in the figures:

[0012] Figure 1 This is a flowchart of a weakly supervised video anomaly detection method according to an embodiment of the present invention;

[0013] Figure 2 This is a logical schematic diagram of a weakly supervised video anomaly detection method according to an embodiment of the present invention;

[0014] Figure 3 This is a schematic diagram illustrating the expansion of anomaly tags according to an embodiment of the present invention;

[0015] Figure 4 This is a schematic diagram illustrating different semantic centers in abnormal video detection.

[0016] Figure 5(a) is one of the detection results of the XD-Violence dataset according to an embodiment of the present invention (Fast.Five.2011);

[0017] Figure 5(b) is a second image showing the detection results of the XD-Violence dataset according to an embodiment of the present invention (Jason.Bourne.2016);

[0018] Figure 5(c) shows three images of the detection results of the XD-Violence dataset according to an embodiment of the present invention (v = 2WkuPNLfl5s);

[0019] Figure 5(d) shows four images of the detection results of the XD-Violence dataset according to an embodiment of the present invention (v=Ehh8ZdlMMj4);

[0020] Figure 5(e) shows five images illustrating the detection results of the XD-Violence dataset according to an embodiment of the present invention (The.Hurt.Locker.2008);

[0021] Figure 5(f) shows six images illustrating the detection results of the XD-Violence dataset according to an embodiment of the present invention (Be.with.You.2018);

[0022] Figure 6(a) is one of the detection results of the UCF-Crime dataset according to an embodiment of the present invention (Burglary079_x264);

[0023] Figure 6(b) is a second image showing the detection results of the UCF-Crime dataset according to an embodiment of the present invention (Stealing079_x264);

[0024] Figure 6(c) shows the detection results of the UCF-Crime dataset according to an embodiment of the present invention (Robbery137_x264);

[0025] Figure 6(d) is a fourth image showing the detection results of the UCF-Crime dataset according to an embodiment of the present invention (Explosion008_x264);

[0026] Figure 6(e) shows the detection results of the UCF-Crime dataset according to an embodiment of the present invention (Normal_Videos_904_x264);

[0027] Figure 6(f) is the sixth image (Normal_Videos_189_x264) showing the detection results of the UCF-Crime dataset according to an embodiment of the present invention.

[0028] In all the accompanying drawings, the same reference numerals indicate similar or corresponding features or functions. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Example 1: As Figure 1 , Figure 2 , Figure 3 , Figure 4 As shown in Figures 5(a), 5(b), 5(c), 5(d), 5(e), 5(f), 6(a), 6(b), 6(c), 6(d), 6(e), and 6(f), a weakly supervised video anomaly detection method based on cue-based learning knowledge enhancement expands simple labeled text based on cue-based learning and introduces external knowledge for knowledge enhancement, thereby better learning positive anomaly boundary information and achieving more accurate localization.

[0031] A weakly supervised video anomaly detection method based on cue-based knowledge reinforcement learning includes:

[0032] In the feature extraction and aggregation steps, non-overlapping video segments of different lengths are selected using a sliding window and input into a pre-trained l3D network to obtain video feature sequences. To address the issue of large differences in video duration, the input length is standardized through temporal sampling to achieve batch training (insufficient lengths are padded with zeros in the time dimension). If audio features are present, they are divided into overlapping segments, with each segment aligned with the end of its corresponding video segment. These segments are then input into a pre-trained VGGish network to obtain corresponding audio feature sequences. The extracted feature sequences are then input into the audio-visual feature aggregation step. The correlation matrix is ​​reused to model the global and local RGB information of the video segments, while complementary audio information is added. Three different learnable weights are introduced to dynamically fuse the three elements, and the best-performing one is selected as the final method. After residual connections and layer normalization, the output is the final multimodal feature, which is then fed into a multilayer perceptron to output fused features with high-level semantics.

[0033] In the score prediction step, the fused features are input into a multilayer perceptron for dimensionality reduction, and causal convolution is used followed by a Sigmoid activation function to obtain the anomaly score of the video sequence.

[0034] Based on the prompt-based text representation step, this method generates anomaly confidence scores using predicted anomaly scores to perform subject-background separation on the fused features. Anomaly labels are then input into the ConceptNet knowledge graph for augmentation and fed in parallel into a pre-trained CLIP network to extract anomaly text representations. Normal labels are fed into the same CLIP network to obtain initial embedding vectors. Learnable parameters are added before and after these embedding vectors, along with positional encoding, before being fed into the CLIP network to encode text features, resulting in normal text feature vectors. The number of vectors and the length of the learnable parameters can be adjusted according to different datasets, resulting in multiple normal text representations. The similarity scores between the normal text representations and the subject or background features are calculated, and the one with the highest score is selected as the final normal text representation. This method can better learn normal features, bringing them closer to the nearest center, thus improving the model's detection performance.

[0035] The cross-modal alignment step aligns the positive anomalous text representation with the fused features after subject-background separation. For anomalous videos, the cosine similarity between environmental features and normal text representation, as well as the cosine similarity between video subject features and anomalous text representation, is calculated. For normal videos, only the cosine similarity between normal text representation and video subject features is calculated.

[0036] The multi-task joint optimization steps involve: for the anomaly detection task, multi-instance learning is performed on the input anomaly scores to construct a binary cross-entropy loss function for anomaly prediction; for the cross-modal alignment task, the similarity matrix between the input subject or background features and the text representation, along with the corresponding original positive anomaly binary labels from the video, is used to construct a label distribution matrix. The similarity matrix is ​​then normalized and logarithmized to obtain the predicted distribution matrix, and the KL divergence loss between the two distribution matrices is calculated. The two tasks are then jointly optimized through a weighted summation of the loss functions.

[0037] Training is completed by optimizing the learnable parameters in the feature aggregation step, score prediction step, and normal text representation using the backpropagation algorithm until both loss functions converge. Testing is then performed using only the multi-instance learning loss.

[0038] Backpropagation is one of the core algorithms in deep learning used to train neural networks. It minimizes the loss function by calculating the gradients of the loss function with respect to the network weights and using these gradients to update the weights. The derivation of backpropagation is based on the chain rule, which calculates the local gradient of each layer and propagates it to the previous layer to obtain the gradient of the input layer.

[0039] Example 2: As Figure 1 , Figure 2 , Figure 3 , Figure 4 As shown in Figures 5(a), 5(b), 5(c), 5(d), 5(e), 5(f), 6(a), 6(b), 6(c), 6(d), 6(e), and 6(f), a weakly supervised video anomaly detection method based on cue-based knowledge enhancement is proposed. Addressing the problems existing in current weakly supervised video anomaly detection methods, this invention uses some pre-trained models from existing work as the basis for data preprocessing and designs a video anomaly detection model based on cue-based knowledge enhancement.

[0040] This invention first obtains fused features through feature extraction and aggregation, generates anomaly scores using a classifier, and then uses these scores to perform subject-background separation on the fused features. This is then cross-modal aligned with positive anomalous text obtained from a cue-based learning-based text representation step. Considering the success of cue-based learning in visual understanding, introducing external semantic knowledge helps the model better understand visual scenes. In the text representation step, for anomalous labels, they are input into the ConceptNet knowledge graph to obtain richer anomalous expressions to construct cue templates. For normal labels, a learnable multi-head normal cue semantic generation mechanism is introduced. Learnable parameters of different lengths are set, and normal text features with different semantic centers are obtained through a CLIP encoder. Each normal visual feature selects the normal text feature with the highest similarity for cross-modal alignment, clustering complex normal scenes into different semantic centers. This allows for better learning of normal features, compensating for the inability of a single normal label to adequately represent complex scenes, thereby promoting boundary discrimination of anomalous features and improving model detection performance. Finally, a multi-task joint optimization approach was adopted for parameter optimization. The KL divergence loss and multi-instance loss were used as constraints, and the model parameters were updated iteratively by backpropagating gradients to obtain a detection model that accurately locates anomalies.

[0041] The video anomaly detection method provided by this invention aims to alleviate the problems of large intra-class variance, small inter-class variance, and uneven distribution of positive and negative samples in positive / abnormal videos, and enriches the knowledge acquisition path of the model by semantically expanding video tags based on prompting learning.

[0042] like Figure 1 and Figure 2 As shown in the figure, the video anomaly detection method provided in this embodiment mainly includes the following steps:

[0043] In the feature extraction and aggregation steps, for video features, a sliding window is used to divide the video V into non-overlapping segments v. i If there is not enough window at the end of the video, it will be discarded.

[0044] The RGB features of these fragments are extracted using a pre-trained I3D network, denoted as... Where s represents the initial video features, V represents the video features, i represents the i-th segment, D represents the video feature dimension, and the corresponding video RGB feature is represented as X. s V ∈R N×D , where N is the sequence length.

[0045] Simultaneously, a unified input length Γ is achieved through temporal sampling. When the number of video sequences N≥Γ, Γ segments are randomly and consecutively sampled from the sequence. If N≤Γ, zero-padding is performed in the temporal dimension to the length Γ to achieve batch training. For audio features, they are divided into overlapping 960ms segments, where each segment is aligned with the end of the video segment, and the boundaries are zero-padding aligned.

[0046] Audio features, denoted as X, are extracted using a VGGish network pre-trained on the AudioSet. A ∈R N×D′ , where A represents audio and D' represents the feature dimension of audio.

[0047] Feature aggregation uses global-local temporal modeling to model RGB features. First, RGB feature X... s V ∈R N×D The mimicking attention mechanism maps to query vectors, key vectors, and value vectors through different linear layers. The matrix obtained by multiplying the query vector and the key vector is then normalized by Softmax to obtain the global attention map A1. The local attention map A2 is further processed by local masking.

[0048] Since the local timestamps of audio and video are difficult to synchronize strictly (such as audio-visual delay), local masks may introduce misalignment interference. Therefore, the same method is used to obtain the audio global attention map A3.

[0049] The attention map is then multiplied with the value vector to obtain the global and local video features and the global audio feature X. g X l X a .

[0050] Finally, adaptive weights are used to adjust the response amplitudes of the three modal branches, and the fused audio-visual global-local features X are obtained by weighted summation. ag .

[0051] The adaptive weights are fused using three different methods: α,β = 1 - α, γ, α,β, γ = 1 - α - β, and α,β, γ. The method with the best performance is selected during training.

[0052] Fusion Feature X ag The calculation formula is as follows:

[0053] X ag =α·X g +β·X l +γ·X a , α, β, γ∈(0,1)

[0054] Where ag represents the average value, g represents the global value, l represents the local value, a represents the audio value, and α, β, and γ are adaptive weighting parameters.

[0055] The fused features are processed through an activation function and a layer normalization function R-Layemorm(·), and then combined with the video RGB features X. s V The residual connections output the final multimodal feature X. m :

[0056]

[0057] Where X m In this context, 'm' represents multimodal.

[0058] The fused multimodal features are then passed through a multilayer perceptron (MLP) to obtain a higher-level feature representation X. e :

[0059] X e =MLP(X m )

[0060] MLP=Dropout(GELU(Convld(·)))

[0061] MLP is a multilayer perceptron containing a one-dimensional convolutional layer Conv1d, an activation function GELU, and a dropout layer, which can randomly drop neurons to prevent the model from overfitting. e represents a higher level of output.

[0062] The score prediction step, after obtaining the aggregated audio and video features X e By inputting the same multilayer perceptron (MLP) for feature dimensionality reduction, a more complex nonlinear feature representation X is obtained. v , where v represents the final feature of the video.

[0063] The causal convolutional (conv) method is used for prediction, and the final anomaly score S is output after passing through the activation function σ, as shown in the following formula:

[0064] X v =MLP(X e )

[0065] S=σ(conv(X v ))

[0066] The text representation step based on prompting learning involves selecting highly semantically related core concepts from the ConceptNet knowledge base for anomaly labels based on the semantic association characteristics of the anomaly detection task, and ensuring through manual verification that the final retained concepts are semantically consistent with the target anomaly category.

[0067] Each anomaly category c corresponds to multiple expanded concepts {p1, p2, p3, K, p n}, where n is the number of expanded concepts, which are used as cue inputs to the CLIP model's text encoder to obtain anomalous text features T. c ={t1, t2, K, t n} Calculate the mean of the abnormal text features for each category, and use it as the semantic feature for that category.

[0068]

[0069]

[0070] For normal labels, such as Figure 4 As shown, the normal scenes in the video are quite broad and may cover multiple semantic centers. It's not feasible to simply align the video with a single semantic focus, as this could lead to misjudgments. Therefore, multiple learnable normal text labels are designed to mitigate this issue. First, the label 'Normal' is fed into a pre-trained CLIP model for text encoding to obtain the initial embedding vector e. init ∈R d :

[0071] e init =CLIP text (′Normal′)

[0072] Where d is the embedding dimension of CLIP, init represents the initial embedding, and CLIP text This represents text embedding encoding. The learnable parameters are divided into prefix parameters and suffix parameters, which are added before and after the embedding vector, respectively, along with positional encoding E. pos The text features are fed into the CLIP network to encode the text features, resulting in a normal text feature vector T. i e A normal text feature vector represents the semantic center of a normal scene. Since the within-class variance of normal scenes varies in different datasets, the number of vectors and the length of the parameters can be adjusted to adapt to different normal scenes, as shown in the following formula:

[0073] E i =Concat(P ipre e init P i pos t)+E pos

[0074] T i e =CLIP text-encoder (E i )

[0075] Among them, P i pre For the prefix parameter, P i post The suffix parameter, i, represents the i-th normal text feature in the design, with lengths l1, l2, and CLIP respectively. text-encoder For text encoders.

[0076] Since anomalous events in real-world scenes often occupy only a local area of ​​the image or a brief temporal segment, in order to better learn scene information in the image, the score S output by the score prediction step is used to generate the video anomaly confidence A. sub The value ranges from 0 to 1, and is then combined with the audio / video global-local feature aggregation feature X. e Multiplying them together yields the enhanced video subject feature V. sub For environmental characteristics V sur Then, using 1-S as the normal confidence level, the same exponential e-transform is performed to suppress video background interference through normalization.

[0077]

[0078] Where λ is a preset scaling factor, sub represents the subject, sur represents the environment, k represents the segment number in a video, and S k Includes the outlier score representing the k-th segment of the video.

[0079] Then, the main feature V of the video was... sub and video environment features V sur Cosine similarity is calculated between the text feature vector and the normal text feature vector, and the text feature vector with the highest similarity is selected as the normal text representation of the video segment. For normal videos, the similarity between the video subject features and the text feature vector is calculated; for abnormal videos, the similarity between the video environment features and the text feature vector is also calculated. i :

[0080]

[0081] Where V = {V sub ∪V surFinally, the best-matching text feature is selected as the normal text for the current segment.

[0082] Semantic center, i.e., similarity CosineSim i Text features corresponding to the maximum value i represents the number of normal text features set:

[0083]

[0084] By matching different text features to cluster semantic centers of different scenarios, the model can dynamically capture fine-grained features of different normal scenarios, avoiding insufficient representation caused by a single feature.

[0085] The cross-modal alignment step aligns the selected normal text features, expanded abnormal text features, and video features based on cue-learning text knowledge representation across modalities. Specifically, it brings the normal video subject features, abnormal video environment features, and normal text features closer together. Distance between and abnormal text The distance between the main features of the abnormal video and the normal text should be increased, but the distance between the main features of the abnormal video and the normal text should be increased so that the model can better learn the feature representation of the normal text, thereby clarifying the boundary between the abnormal and the normal.

[0086] That is, for abnormal videos, maximize the environmental feature V. sur Cosine similarity to normal text and video subject features V sub Cosine similarity to abnormal text.

[0087] For normal videos, only the cosine similarity between the normal text and the main features of the video is maximized:

[0088]

[0089] Where V = {V sub ∪V sur},

[0090] τ is a learnable temperature coefficient used to scale the influence of cosine similarity on the Softmax probability distribution. As τ decreases, the probability distribution tends to become sharper, enhancing the model's confidence in high-similarity positive samples. v2t indicates that video features are moving closer to the text side.

[0091] The multi-task joint optimization step of the video anomaly detection model based on cue-based knowledge enhancement involves two tasks. For the anomaly detection task, multiple instances are used to learn from the input anomaly score, and a binary cross-entropy loss function L for anomaly prediction is constructed. mil .

[0092] In weakly supervised scenarios, anomaly detection only has video-level labels during the training phase. i However, frame-level prediction scores are needed during testing. Therefore, a loss based on multi-instance learning is applied as the basic objective function. In the multi-instance learning paradigm, each video is a packet containing non-overlapping video segments. Videos containing at least one anomalous segment are called positive packets, and videos containing no anomalous segments are called negative packets. The goal of this task is to train the network to increase the distance between negative and positive packets, and to widen the distance between the largest positive packet value and the largest negative packet value, so as to achieve effective discrimination and localization of anomalous segments. First, the obtained scores S are sorted in descending order, and the average of the scores of the first k segments is taken as the anomalous score pre of the video. i :

[0093]

[0094] in, It is the final video anomaly score. This is a ground truth label at the video level, where Num represents the number of videos. For abnormal videos... Where q is the scale factor of the sampled video; while for normal video, k = 1.

[0095] For cross-modal alignment tasks, after calculating the cosine similarity score of a text or video feature with dimension (Length, Dim), a matrix Sin& with dimension (Length, Length) is obtained. Therefore, a sample label matrix gt with the same dimension needs to be generated.

[0096] The specific process is as follows:

[0097] Each video feature has its original video-level tag. i To align with the cross-modal alignment task, labels are generated for each video subject or environmental feature. That is, for each video sample, if y i =0, which indicates a normal video; therefore, extract its main feature V. sub and will Add to labels; if y i =1, indicating an abnormal video, then extract its main feature V. sub and will Add to labels. Simultaneously extract its environmental features V. sur Add to label The corresponding cross-modal aligned label matrix gt(i,j) is set according to the following rules:

[0098]

[0099] Finally, Sim& learns the distribution of ground truth (gt) to achieve cross-modal text-video alignment. The Softmax function is used to transform the similarity matrix Sim& into a probability distribution, and its logarithmic value is calculated as the prediction distribution. Since the probability values ​​are usually small, direct processing may lead to numerical underflow. Performing logarithmic calculation transforms them into larger negative numbers, which can more clearly capture the differences between different segments, improving the model's stability and accuracy.

[0100] The formula is as follows:

[0101] preds = Softmax((CosineSim&)

[0102] p & =log(preds)

[0103] In order to match the predicted distribution p & The target distribution and the target value are aligned in terms of their magnitudes, as follows:

[0104] target = Softmax(gt × scale)

[0105] Where scale is an adjustable parameter for the label.

[0106] Finally, the KL divergence loss is used to calculate the loss L between the two. align :

[0107]

[0108] The final loss function is the multi-instance learning loss L. mil and KL divergence L align Weighted summation:

[0109] L=ζL mil +ρL align

[0110] Here, ζ and ρ are adjustable loss weights. After calculating the loss, the model performs backpropagation to compute the gradient and uses the Adam optimizer to update the parameters to achieve multi-task joint optimization.

[0111] This invention was tested on two large public datasets, XD-Violence and UCF-Crime. The XD-Violence dataset contains 4754 untrimmed videos, totaling 217 hours in length. These video clips come from movies, YouTube, and other sources, covering a variety of complex scenes. The dataset includes six types of violence: abuse, car accidents, explosions, fights, riots, and shootings. Each video clip is accompanied by an audio signal and a weak label. The UCF-Crime dataset is a large dataset for anomaly detection in video surveillance systems. It contains 1900 untrimmed real-world surveillance videos totaling 128 hours, covering 13 types of anomalous activities, including abuse, arrests, and arson. Following a general evaluation paradigm, appropriate evaluation metrics were selected for the characteristics of different datasets: for the UCF-Crime dataset, the Area Under the Curve (AUC) was used to evaluate model performance; for the XD-Violence dataset, Average Precision (AP) was used as the evaluation metric, with ROC or AP values ​​being proportional to model performance. To address the extreme class imbalance between abnormal and normal events in real-world scenarios, a false positive rate with a threshold of 0.5 is introduced as a supplementary evaluation dimension. This false positive rate is also known as the False Alarm Rate (FAR). The FAR measures the probability that a model misclassifies a normal event as an abnormal one; therefore, a lower FAR is better.

[0112] Tables 1 and 2 show ablation experiments with different numbers of normal cue heads on two datasets.

[0113] Table 1. Experimental results on the XD-Violence dataset with different numbers of cue heads.

[0114]

[0115] Note: Different numbers of cue heads represent different amounts of semantic space. Each cue head can learn parameters of (3, 3), and the same applies below.

[0116] Table 2 Experimental results for different numbers of alert heads on the UCF-Crime dataset.

[0117]

[0118] It can be seen that the performance is best when the number of prompts is 3. Subsequently, the model performance deteriorates as the number of normal prompts increases, and the false alarm rate also increases. This shows that although normal scenarios are complex, if the semantics are too scattered, it will hinder the model from learning the boundaries between positive and abnormal, which will be counterproductive and cause misjudgment.

[0119] Tables 3 and 4 respectively verify the impact of the length of learnable parameters in three normal heads containing learnable parameters on model performance on two datasets. The tables show that optimal performance is achieved when the learnable parameter length is 3. The overall model performance initially increases and then decreases with increasing parameter length. This is because when the number of learnable parameters exceeds the range that the model can dynamically learn, too many parameters introduce noise, hindering the representation of normal label semantics.

[0120] Table 3 Ablation experiments with different learnable parameter lengths on the XD-Violence dataset.

[0121]

[0122] Table 4 Ablation experiments with different learnable parameter lengths on the UCF-Crime dataset.

[0123]

[0124] To visually represent the performance of this method on existing datasets, the performance of the model before and after the improvement was compared on two commonly used datasets, as shown in Tables 5 and 6.

[0125] The model of this invention has an average accuracy of 86.85%, and the false alarm rate has been reduced from 0.1% of existing models to 0.01%.

[0126] Table 5 Comparison of results between the method of the present invention and existing methods on the XD-Violence dataset.

[0127]

[0128] This invention achieves a performance of 87.75% on the UCF-Crime dataset, which is 0.99% better than existing inventions, and the false alarm rate is reduced from the initial 0.1% to 0.07%. This demonstrates that this invention can enhance the network's ability to represent normal scenes by utilizing cue learning, thereby improving the robustness of the model for imbalanced tasks.

[0129] Table 6 Comparison of results between the method of the present invention and existing methods on the UCF-Crime dataset.

[0130]

[0131] Furthermore, to demonstrate the effectiveness of the steps proposed in this invention, the final detection results obtained by the prior art and this invention are visualized, as shown in Figures 5(a), 5(b), 5(c), 5(d), 5(e), 5(f), 6(a), 6(b), 6(c), 6(d), 6(e), and 6(f). The horizontal axis represents the number of frames, and the vertical axis represents the predicted score. The orange portion represents the ground truth label, and the curve represents the predicted score. The upper curve represents the result of the prior art, and the lower curve represents the prediction result of this invention. Figures 5(a), 5(b), 5(c), 5(d), 5(e), and 5(f) are from the XD-Violence dataset, and Figures 6(a), 6(b), 6(c), 6(d), 6(e), and 6(f) are from the UCF-Crime dataset.

[0132] As can be seen, for abnormal videos in the XD-Violence dataset, the method of this invention is superior to existing inventions in distinguishing normal and abnormal boundaries, and the score difference between normal and abnormal is more significant, indicating that the model of this invention can accurately locate the time of abnormality.

[0133] For normal videos, compared to the original model in the baseline model, the prediction score of this invention is lower and smoother. As shown in Figures 5(d) and 5(e), when the video is a complex scene similar to an anomaly, the prior art produces perturbations, while this invention is more robust and can perform anomaly detection more stably. This demonstrates that this invention, through cue-based knowledge enhancement, can improve the model's understanding of anomaly scenes and promote the improvement of model detection performance.

[0134] Similarly, on the UCF-Crime dataset, as shown in Figures 6(a) and 6(d), the present invention can more accurately locate the time period of the anomaly and has a larger prediction gap between normal and anomalies.

[0135] As shown in Figures 6(e) and 6(f), in normal videos, the present invention effectively mitigates false alarms of anomalies and ensures that normal segments have low prediction values. This proves that learnable normal heads can guide the model to better learn the differences between positive and abnormal features through external knowledge and achieve robust anomaly detection.

[0136] Example 3: As Figure 1 , Figure 2 , Figure 3 , Figure 4 As shown in Figures 5(a), 5(b), 5(c), 5(d), 5(e), 5(f), 6(a), 6(b), 6(c), 6(d), 6(e), and 6(f), a weakly supervised video anomaly detection method based on cue-based knowledge enhancement includes the following steps:

[0137] The feature extraction and aggregation steps involve inputting a video segment, selecting non-overlapping video segments through a sliding window, and inputting these segments into a pre-trained deep learning network to obtain a video feature sequence, which is then truncated to a fixed length. If audio features are present, they are divided into overlapping segments, with each segment aligned with the end of its corresponding video segment. These segments are then input into a pre-trained deep learning audio feature extraction network to obtain the corresponding audio feature sequence. The extracted feature sequences are then input into the audio-visual feature aggregation step to model the global and local RGB information of the video segments, while introducing complementary audio information. Since the local timestamps of audio and video are difficult to synchronize strictly (e.g., audio-visual delay), local masks may introduce misalignment interference, and some datasets exhibit persistent anomalies across segments (e.g., continuous fighting, continuous shooting). Global audio guidance can enhance the model's perception of overall anomaly patterns. Therefore, only global audio features are used for fusion, with learnable weight parameters used for dynamic fusion. Finally, after residual connections and layer normalization, the output is the final multimodal feature, which is then processed by a multilayer perceptron to obtain fused features with high-level semantics.

[0138] The score prediction step involves dimensionality reduction of the fused features and the use of one-dimensional convolution to obtain the anomaly score of the video sequence.

[0139] The text representation step based on cue learning uses predicted anomaly scores to generate anomaly confidence levels for subject-background separation of fused features, and augments both anomaly and normal labels in different ways. Specifically for normal text, addressing the issue of normal scenes having multiple semantic centers but only one label, a learnable multi-head normal cue is designed, corresponding to multiple normal semantic centers. During training, normal features are positioned closer to the nearest center to better learn normal features and obtain a text representation with richer knowledge.

[0140] The cross-modal alignment step aligns the positive abnormal text representation with the fused features after subject-context separation across modalities, and calculates the cosine similarity between the context-subject features and the text representation.

[0141] The multi-task joint optimization steps involve: for the anomaly detection task, multi-instance learning is performed on the input anomaly scores to construct a loss function for anomaly prediction; for the cross-modal alignment task, a similarity matrix between the input subject or background features and the text representation, along with the corresponding original positive anomaly binary labels from the video, is used to construct a label distribution matrix. The similarity matrix is ​​then normalized and logarithmized to obtain a predicted distribution matrix, and the loss function between the two distribution matrices is calculated. Finally, the two tasks are jointly optimized by weighted summation of the loss functions.

[0142] Training is completed by optimizing the learnable parameters in the feature aggregation step, score prediction step, and normal text representation using the backpropagation algorithm until both loss functions converge. Testing is then performed using only the multi-instance learning loss.

[0143] The feature extraction and aggregation steps use a pre-trained I3D network to extract video features and a pre-trained VGGish network to extract audio features.

[0144] The feature extraction and aggregation steps utilize an attention mechanism to calculate the correlation matrix between feature segments, resulting in global and local attention maps. These attention maps are then multiplied by the features and weighted summed, fusing global video features, global audio features, and local video features. The weights are learnable parameters α, β, γ ∈ (0, 1). Different fusion methods are compared to analyze the relationships between different modalities. After aggregation, the features are processed through residual connections and layer normalization to output the final multimodal features. Finally, a multilayer perceptron is used to obtain fused features with high-level semantics.

[0145] The score prediction step involves reducing the dimensionality of the fused features using a multilayer perceptron, followed by a special one-dimensional convolution, namely causal convolution, and then inputting it into the Sigmoid activation function to obtain the final anomaly score.

[0146] The text representation step based on cue learning uses anomaly scores to generate video anomaly or normal confidence levels, which are then multiplied with the fused features obtained from the feature aggregation step to obtain enhanced video subject or environment features. Cue learning is then used to augment the anomaly and normal labels respectively.

[0147] The abnormal and normal labels are augmented using cue learning. The abnormal labels are input into the ConceptNet knowledge graph for augmentation and fed into a pre-trained CLIP network to extract abnormal text representations. The normal labels are fed into the same CLIP network to obtain initial embedding vectors. Learnable parameters are added before and after the embedding vectors, and positional encoding is added. The vectors are then fed into the CLIP network to encode text features, resulting in normal text feature vectors. The number of vectors and the length of the learnable parameters can be adjusted according to different datasets to obtain multiple normal text representations. The similarity scores between the normal text representations and the subject or background features are calculated, and the one with the highest score is selected as the final normal text representation.

[0148] The cross-modal alignment step calculates the cosine similarity between environmental features and normal text representations, as well as the cosine similarity between video subject features and abnormal text representations, for anomalous videos. For normal videos, it only calculates the cosine similarity between normal text representations and video subject features.

[0149] The multi-task joint optimization steps use the binary cross-entropy loss function for multi-instance learning and the KL divergence loss for cross-modal alignment tasks.

[0150] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A weakly supervised video anomaly detection method based on cue-based knowledge enhancement, characterized in that, Includes the following steps: Feature extraction and aggregation steps; Score prediction steps; Text representation steps based on cue-based learning; Cross-modal alignment steps; Multi-task joint optimization steps; The feature extraction and aggregation steps include the following: Given an abnormal scene video, a feature extraction network obtains video sequence features and audio sequence features. These are then input into a well-trained feature aggregation network for multimodal feature aggregation, followed by an anomaly score prediction network. A text representation step based on cue learning constructs cue templates for abnormal video labels using a knowledge graph. Normal labels undergo semantic expansion using multiple learnable parameters and are cross-modal aligned with the video features, aligning them towards different normal semantics. External information is introduced for knowledge enhancement, learning the boundaries between positive and negative video features to improve detection performance. Finally, multi-task joint optimization using different loss functions is performed to locate abnormal video segments. The text representation steps based on cue learning include the following steps: generating anomaly confidence through predicted anomaly scores to perform subject-background separation on fused features; expanding the anomaly and normal labels in different ways; designing learnable multi-head normal cue corresponding to multiple normal semantic centers; during training, moving normal features closer to the nearest center to better learn normal features and obtain a text representation with richer knowledge.

2. The weakly supervised video anomaly detection method based on cue-based knowledge enhancement according to claim 1, characterized in that, The feature extraction and aggregation steps include the following steps: non-overlapping video segments are selected through a sliding window, input into a pre-trained network to obtain video feature sequences, which are then truncated to a fixed length. If audio features are present, they are divided into overlapping segments, aligned with the ends of the corresponding video segments, and input into a pre-trained audio feature extraction network to obtain corresponding audio feature sequences. The extracted feature sequences are then input into the audio-visual feature aggregation step to model the global and local RGB information of the video segments, while introducing complementary audio information. After passing through a ReLU activation function and layer normalization, and then through residual connections, the final multimodal features are output, and a multilayer perceptron is used to obtain fused features with high-level semantics.

3. The weakly supervised video anomaly detection method based on cue-based knowledge enhancement according to claim 1, characterized in that, The score prediction step includes the following steps: inputting the fused features back into a multilayer perceptron for dimensionality reduction, and using causal convolution followed by a Sigmoid activation function to obtain the anomaly score of the video sequence.

4. The weakly supervised video anomaly detection method based on cue-based knowledge enhancement according to claim 1, characterized in that, The cross-modal alignment step includes the following steps: performing cross-modal alignment between positive anomalous text representations and the fused features after subject-background separation; for anomalous videos, calculating the cosine similarity between environmental features and normal text representations, as well as between video subject features and anomalous text representations; for normal videos, only calculating the cosine similarity between normal text representations and video subject features.

5. The weakly supervised video anomaly detection method based on cue-based knowledge enhancement according to claim 1, characterized in that, The multi-task joint optimization step includes the following steps: design two different loss functions for the anomaly detection task and the cross-modal alignment task, and jointly optimize the two tasks by weighted summation of the loss functions. The backpropagation algorithm is used to optimize the learnable parameters in the feature aggregation step, the score prediction step, and the normal text representation to make the two loss functions converge and complete the training.

6. The weakly supervised video anomaly detection method based on cue-based knowledge enhancement according to claim 2, characterized in that, The feature extraction and aggregation steps include the following: For video features, a sliding window is used to aggregate the video... Divide into non-overlapping segments If the window at the end of the video is insufficient, it will be discarded. The RGB features of these fragments are extracted using a pre-trained I3D network, denoted as... R represents the real number space, where s represents the initial video features, V represents the video features, i represents the i-th segment, D represents the video feature dimension, and the corresponding video RGB features are represented as follows: Where N is the sequence length, Simultaneously, the input length is unified through timing sampling. When the number of video sequences When that happens, random continuous sampling is performed from the sequence. A segment, if Zero-padding to length in the time dimension To achieve batch training, audio features are divided into overlapping 960 ms segments, each aligned with the end of a video segment, with zero-padding alignment at the boundaries. Audio features are extracted using a VGGish network pre-trained on the AudioSet, denoted as . Where A represents audio and D' represents the feature dimension of the audio. Feature aggregation uses global-local temporal modeling to model RGB features. First, RGB features... The mimicking attention mechanism maps query vectors, key vectors, and value vectors through different linear layers, and then normalizes the matrix obtained by multiplying the query vector and key vectors using Softmax to obtain the global attention map. Local attention map We also need to perform local masking and obtain the audio global attention map in the same way. , Then, the attention map is multiplied by the value vector to obtain global and local video features, and global video features. Local video features Global audio features , Finally, adaptive weights are used to adjust the response amplitudes of the three modal branches, and the fused audio-visual global-local features are obtained by weighted summation. , Adaptive weights are fused using three different methods: , and Three different weight ratios are available; you can choose one during training. Fusion features The calculation formula is as follows: Where ag represents the average value, g represents the global value, l represents the local value, and a represents the audio value. These are adaptive weight parameters. The fused features are processed by activation functions and layer normalization functions. Then, the video RGB features are analyzed. The residual connections are used to output the final multimodal features. : in In this context, 'm' represents multimodal. These are the aforementioned fusion features. The fused multimodal features are then processed through a multilayer perceptron (MLP) to obtain a higher-level feature representation. : MLP is a multilayer perceptron containing a one-dimensional convolutional layer Conv1d, an activation function GELU, and a dropout layer, which can randomly drop neurons to prevent the model from overfitting. Here, the subscript e represents a higher level output.

7. The weakly supervised video anomaly detection method based on cue-based knowledge enhancement according to claim 2, characterized in that, The score prediction step includes the following steps: after obtaining the aggregated audio and video features By inputting the same multilayer perceptron (MLP) for feature dimensionality reduction, more complex nonlinear feature representations are obtained. Where v represents the final feature of the video, Causal convolution (conv) is used for prediction, followed by activation function. Output the final anomaly score The formula is as follows: In the text representation step based on prompt-based learning, for anomaly labels, based on the semantic relevance characteristics of the anomaly detection task, highly semantically related core concepts are selected from the ConceptNet knowledge base. Manual verification is then used to ensure that the final retained concepts are semantically consistent with the target anomaly category. Each anomaly category Corresponding to multiple expanded concepts Where n is the number of expanded concepts, which are used as cue inputs to the CLIP model's text encoder to obtain anomalous text features. Calculate the mean of the abnormal text features for each category, and use it as the semantic feature for that category. : For normal labels, the label 'Normal' is fed into the pre-trained CLIP model for text encoding to obtain the initial embedding vector. : Where d is the embedding dimension of CLIP, init represents the initial embedding, and CLIP text This represents text embedding encoding, where learnable parameters are divided into prefix parameters and suffix parameters, which are added before and after the embedding vector, respectively, along with positional encoding E. pos The text features are fed into the CLIP network to encode the text features, resulting in a normal text feature vector. A normal text feature vector represents the semantic center of a normal scene. Since the within-class variance of normal scenes varies in different datasets, the number of vectors and the length of the parameters can be adjusted to adapt to different normal scenes, as shown in the following formula: in, For prefix parameters, The suffix parameter, i, represents the i-th normal text feature in the design, and its length is respectively... CLIP text-encoder For text encoders, The video anomaly confidence score is generated using the anomaly score S output from the score prediction step. The value ranges from 0 to 1, and is then combined with the multimodal high-level features obtained above. Multiply to obtain the enhanced main features of the video. Regarding environmental characteristics Then use The same exponential e-transform is applied to the normal confidence level, thereby suppressing background interference in the video through normalization: in The preset scaling factor is used, where sub represents the subject, sur represents the environment, k represents the segment number in the video, and S... k Includes the outlier score representing the k-th segment of the video; Then, the main features of the video will be... and video environment features The cosine similarity is calculated between the text feature vector and the normal text feature vector. The text feature vector with the highest similarity is selected as the normal text representation of the video segment. For normal videos, the similarity between the video subject features and the text feature vector is calculated. For abnormal videos, the similarity between the video environment features and the text feature vector is also calculated. : in Finally, the most matching text feature is selected as the normal semantic center of the current segment, i.e., the similarity Sim. i Text features corresponding to the maximum value , where i represents the number of normal text features set: Semantic centers for different scenarios are clustered by matching different text features.

8. The weakly supervised video anomaly detection method based on cue-based knowledge enhancement according to claim 7, characterized in that, The cross-modal alignment step includes the following steps: cross-modal alignment of normal text features, augmented anomalous text features, and video features selected from the text knowledge representation based on cue learning; for anomalous videos, maximizing environmental features. Cosine similarity to normal text and video subject features Cosine similarity to anomalous text For normal videos, only the cosine similarity between the normal text and the main features of the video is maximized: in , τ is a learnable temperature coefficient used to scale the influence of cosine similarity on the Softmax probability distribution. When τ decreases, the probability distribution tends to become sharper, enhancing the model's confidence in high-similarity positive samples. v2t indicates that video features are closer to the text side.

Images