Weak supervision video anomaly detection method and system based on prompt learning

Abstract

The invention provides a weak supervision video anomaly detection method and system based on prompt learning, and belongs to the technical field of abnormal event detection based on computer vision, and the method comprises the steps: obtaining to-be-processed video data; and processing the acquired to-be-processed video data by using a pre-trained anomaly detection model to obtain a specific classification result of the abnormal events in the video. According to the invention, a video local and global adaptive time modeling module is introduced to capture local and global dependency relationships at the same time, and the relationship between the demand of detailed time modeling and the calculation efficiency is balanced; by utilizing an external knowledge base, the distinguishing capability of the model on different categories is improved; according to the method, a text-video comparison loss function is designed, the similarity of a correctly matched text-video pair is enhanced, the similarity of wrong matching is reduced, and too high similarity of a negative sample is effectively inhibited, so that the distinguishing capability of the model is improved, the matching of the text and the video is more accurate, and the video and text alignment capability of the model is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of abnormal event detection based on computer vision, and in particular to a weakly supervised video anomaly detection method and system based on prompt learning. Background Art

[0002] Video anomaly detection analyzes video data to automatically identify and detect unusual behavior or events, and has widespread application in the security field. By improving detection accuracy and robustness, this technology not only enhances the effectiveness of surveillance systems and reduces false alarm rates, but also significantly reduces labor costs, thereby improving overall surveillance efficiency and generating positive economic benefits.

[0003] Early approaches to video anomaly detection primarily relied on semi-supervised methods. These methods, such as those based on future frame prediction, utilize only normal videos to learn normal patterns and identify abnormal events as those that deviate from these patterns. However, because these methods primarily rely on normal videos for training, they struggle to generalize to unseen abnormal patterns, have limited use of abnormal videos, and are prone to high false positive rates. Consequently, in practical applications, the detection effectiveness of semi-supervised methods is often limited.

[0004] In recent years, weakly supervised video anomaly detection has attracted more and more attention because it includes both normal and abnormal videos in training, thereby achieving more discriminative representation learning. It also has the advantages of only requiring video-level annotations to indicate the existence of anomalies but not requiring precise time information and having fewer annotation requirements. It has become the mainstream direction of current video anomaly detection research.

[0005] Existing methods for weakly supervised video anomaly detection can be primarily categorized as those based on multi-instance learning and those based on visual-text alignment. First, the principle underlying multi-instance learning methods is that the most anomalous segment in a normal video should be more normal than segments in an abnormal video. Based on this principle, the model distinguishes the features of normal and abnormal regions, gradually mining all potential normal and abnormal regions. This results in a continuous increase in the anomaly confidence of abnormal regions, while a continuous decrease in the anomaly confidence of normal regions, thereby improving detection performance. Second, with the rise of visual-text alignment learning methods, the large-scale pre-trained model CLIP has demonstrated excellent cross-modal representation capabilities, particularly in building deep connections between vision and language. Building task-specific models based on CLIP has become an emerging and rapidly developing research direction, demonstrating potential for broad application in various visual understanding tasks. By combining multimodal information from vision and text, semantic understanding and feature modeling in video anomaly detection can be further enriched, addressing the limitations of traditional unimodal visual approaches.

[0006] Therefore, to effectively apply the knowledge learned by CLIP to the more complex weakly supervised video anomaly detection task, researchers proposed VadCLIP. This directly leverages the frozen CLIP model and employs a dual-branch architecture. The video branch uses CLIP's image encoder as the backbone to extract video features. It then performs temporal modeling through the proposed local-global temporal adapter, directly utilizing visual features for coarse-grained binary classification, specifically detecting whether the video contains anomalies. The text branch uses CLIP's text encoder to encode text labels into class embeddings and combines them with visual cues for fine-grained classification, specifically detecting specific anomaly categories. VadCLIP leverages the pre-trained vision-language model CLIP, adapting to weakly supervised video anomaly detection tasks without additional fine-tuning, significantly improving detection accuracy and robustness.

[0007] In summary, in weakly supervised video anomaly detection, effectively modeling the temporal dynamics of abnormal events and understanding the contextual information in video clips are crucial. Although many existing methods have explored various temporal relationship modeling methods, such as graph convolutional networks and self-attention mechanisms, each method has its limitations: graph convolution only considers local node aggregations and may lose the broader context of the anomaly, while self-attention mechanisms, while capturing global correlations, may introduce irrelevant noise. To overcome these shortcomings, researchers have proposed networks that integrate local and global context through parallel or serial structures. Although these methods enhance context modeling by mitigating long-range noise interference, they increase computational complexity and the number of parameters, which may lead to overfitting and hinder practical deployment. On the other hand, although CLIP already has strong cross-modal alignment capabilities, its text embedding may still be limited in fine-grained distinction and inter-class separability. Summary of the Invention

[0008] The purpose of the present invention is to provide a weakly supervised video anomaly detection method and system based on prompt learning to solve at least one technical problem existing in the above background technology.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] In a first aspect, the present invention provides a weakly supervised video anomaly detection method based on prompt learning, comprising:

[0011] Obtain video data to be processed;

[0012] The acquired video data to be processed is processed using a pre-trained anomaly detection model to obtain specific classification results of abnormal events in the video; wherein, the anomaly detection model includes a video classification branch and a text alignment branch; in the video classification branch, the frozen CLIP image encoder is first used to extract video features, and then the temporal information is obtained through local and global adaptive temporal modeling of the video to achieve local-global context adaptive fusion, and binary classification is performed after two layers of MLP to obtain the classification of video abnormality or normality; in the text alignment branch, the semantic representation of the original word embedding is enriched by the vocabulary database WordNet, and the external semantic information is fused with the original word embedding by weighted averaging, and the anomalies in the video are used as video-level prompts to assist the text branch; finally, the matching similarity between the text embedding and the frame-level visual features is calculated to obtain the alignment mapping, thereby achieving the classification of specific anomalies.

[0013] As a further limitation of the first aspect of the present invention, in the video classification branch, the segment features X are projected into the latent space through different linear layers, the similarity matrix G is obtained by calculating the inner product, and the global attention map M is generated by applying softmax normalization. g , re-weight the projected segment features through the attention map to obtain the global context feature X g ; Local context calibration is achieved by reusing the similarity matrix in the mask window; through the attention map M l Reweight the projected segment features to obtain the local calibration feature X l , global context features and local calibration features are adaptively fused in local-global context, dynamically balancing the importance of global temporal patterns and local subtle differences; dynamic position encoding is introduced to model the relative distance of fragments, and dynamic position encoding is embedded in the similarity matrix G as a position prior; a two-layer MLP is used for feature dimensionality reduction, and each Conv1D layer is followed by a GELU activation and dropout operation; a causal convolutional layer is used to predict fragment-level anomaly scores.

[0014] As a further limitation of the first aspect of the present invention, in the text alignment branch, the categories are semantically enhanced using the vocabulary database WordNet, so that CLIP can better understand different ways of expressing events; first, synonyms or related concepts of the category label are obtained to expand the semantic scope of the category; then, the semantically enhanced category is converted into a category token using CLIP's Tokenizer; second, the enhanced category token is substituted into the Prompt structure, and a learnable prompt containing m context tokens is added to form a complete text input sequence; wherein the category token t initis placed in the middle of the text input sequence; finally, CLIP’s text encoder receives the text input sequence as input and generates the final category embedding.

[0015] As a further limitation of the first aspect of the present invention, visual context is used to improve class text embeddings to make concise text labels more accurate, and visual embeddings in abnormal clips obtained from the video classification branch are used as video-level cues to assist the text alignment branch; the matching similarities between all class embeddings and frame-level visual features are calculated to obtain an alignment map; for each row, the top K similarities are selected and their average is calculated to measure the degree of alignment between the video and the current class; then a vector S is obtained, which represents the similarity between this video and all categories; finally, the alignment loss L is calculated by cross entropy nce , by optimizing so that the matching score of the correct category is the highest.

[0016] As a further limitation of the first aspect of the present invention, the text-video contrast loss function L cl for:

[0017]

[0018] Among them, Y + Represents the set of all positive sample pairs, N represents the number of texts, M represents the number of videos, and Y ij Indicates whether the i-th text and the j-th video match, P ij Represents the softmax probability of the i-th text and the j-th video.

[0019] According to the correlation score between each frame and the text embedding, the frame embeddings of the video are weighted and summed to generate a weighted feature representation of the entire video, and finally the final score of each text-video match is obtained.

[0020] As a further limitation of the first aspect of the present invention, a contrast loss is introduced to distinguish the normal class embedding from the other abnormal class embeddings. First, the cosine similarity between the normal class embedding and the other abnormal class embeddings is calculated, and then the contrast loss L is calculated. clt for:

[0021]

[0022] Among them, t n is the normal class embedding, t aj is the j-th anomaly class embedding, T represents the transposition operation, and ||·||2 represents the L2 norm.

[0023] In a second aspect, the present invention provides a weakly supervised video anomaly detection system based on prompt learning, comprising:

[0024] An acquisition module, used to acquire video data to be processed;

[0025] The processing module is used to process the acquired video data to be processed using a pre-trained anomaly detection model to obtain a specific classification result of abnormal events in the video; wherein, the anomaly detection model includes a video classification branch and a text alignment branch; in the video classification branch, first, a frozen CLIP image encoder is used to extract video features, and then temporal information is obtained through local and global adaptive time modeling of the video to achieve local-global context adaptive fusion, and binary classification is performed after two layers of MLP to obtain a classification of video abnormality or normality; in the text alignment branch, the semantic representation of the original word embedding is enriched using the vocabulary database WordNet, and external semantic information is fused with the original word embedding by weighted averaging, and the anomalies in the video are used as video-level prompts to assist the text branch; finally, the matching similarity between the text embedding and the frame-level visual features is calculated to obtain an alignment mapping, thereby achieving the classification of specific anomalies.

[0026] In a third aspect, the present invention provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the weakly supervised video anomaly detection method based on prompt learning as described in the first aspect is implemented.

[0027] In a fourth aspect, the present invention provides a computer device comprising a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the weakly supervised video anomaly detection method based on prompt learning as described in the first aspect.

[0028] In a fifth aspect, the present invention provides an electronic device comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the weakly supervised video anomaly detection method based on prompt learning as described in the first aspect.

[0029] Explanation of terms:

[0030] Video Anomaly Detection (VAD): A key research area in computer vision and machine learning, VAD aims to automatically analyze video content to identify and detect a variety of unusual events or behaviors, such as intrusions, traffic accidents, violent conflicts, and equipment failures. Its core goal is to build intelligent systems that can autonomously identify anomalies in video streams, thereby improving the efficiency and security of surveillance systems. This field has broad potential for practical applications.

[0031] Abnormal events: These are events that deviate from the norm and are unusual or out of the ordinary compared to expected or normal behavior. Abnormal events are rare and often have negative consequences.

[0032] Weakly supervised video anomaly detection (VAD) aims to detect abnormal events at the frame level using only video-level annotations. During the training phase, both normal and abnormal samples are provided, clearly defining the anomaly and providing clear guidance for the detection algorithm, but the exact location labels of the anomaly in the video are unknown.

[0033] The beneficial effects of the present invention include: introducing a video local and global adaptive temporal modeling module to simultaneously capture local and global dependencies, thereby balancing the need for detailed temporal modeling with computational efficiency; utilizing an external knowledge base, namely WordNet, to enrich text embedding, thereby improving the model's ability to discriminate between different categories, and enabling CLIP to better understand different expressions and improve generalization capabilities; designing a text-video contrast loss function to enhance the similarity of correctly matched text-video pairs while reducing the similarity of incorrect matches, effectively suppressing excessive similarity of negative samples, thereby improving the model's ability to distinguish, making text and video matching more accurate, and enhancing the model's ability to align videos and texts.

[0034] Additional advantages of the present invention will be more clearly given in the following description or learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 This is a functional block diagram of the weakly supervised video anomaly detection model based on prompt learning described in an embodiment of the present invention.

[0037] Figure 2 This is a structural diagram of the video local and global adaptive temporal modeling module according to an embodiment of the present invention.

[0038] Figure 3 The anomaly scoring results for the UCF-Crime and XD-Violence datasets, as described in an embodiment of the present invention, are visualized in the figure. The red curve represents the anomaly score for each frame of the video, and the orange area indicates the frame sequence interval with anomalies. DETAILED DESCRIPTION

[0039] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention.

[0040] Those skilled in the art will understand that unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs.

[0041] It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood to have a meaning consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless as defined herein.

[0042] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a," "an," "said," and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or groups thereof.

[0043] In the description of this specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Those skilled in the art may combine and integrate different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless otherwise contradictory.

[0044] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings. However, the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0045] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0046] Example 1

[0047] In this embodiment 1, a weakly supervised video anomaly detection system based on prompt learning is first provided, comprising: an acquisition module for acquiring video data to be processed; a processing module for processing the acquired video data to be processed using a pre-trained anomaly detection model to obtain a specific classification result of abnormal events in the video; wherein the anomaly detection model includes a video classification branch and a text alignment branch; in the video classification branch, video features are first extracted using a frozen CLIP image encoder; then, temporal information is obtained through local and global adaptive temporal modeling of the video to achieve local-global context adaptive fusion; binary classification is performed after passing through two layers of MLP to obtain a classification of the video as abnormal or normal; in the text alignment branch, the semantic representation of the original word embedding is enriched using the vocabulary database WordNet, external semantic information is fused with the original word embedding through weighted averaging, and anomalies in the video are used as video-level prompts to assist the text branch; finally, the matching similarity between the text embedding and the frame-level visual features is calculated to obtain an alignment mapping, thereby achieving classification of specific anomalies.

[0048] In this embodiment, the above-mentioned system is used to implement a weakly supervised video anomaly detection method based on prompt learning, including: using an acquisition module to acquire video data to be processed; using a processing module to process the acquired video data to be processed based on a pre-trained anomaly detection model to obtain a specific classification result of abnormal events in the video. Wherein, the anomaly detection model includes a video classification branch and a text alignment branch; in the video classification branch, firstly, a frozen CLIP image encoder is used to extract video features, and then temporal information is obtained through local and global adaptive time modeling of the video to achieve local-global context adaptive fusion, and binary classification is performed after two layers of MLP to obtain a classification of video abnormality or normal; in the text alignment branch, the semantic representation of the original word embedding is enriched using the vocabulary database WordNet, and the external semantic information is fused with the original word embedding by weighted averaging, and the anomaly in the video is used as a video-level prompt to assist the text branch; finally, the matching similarity between the text embedding and the frame-level visual features is calculated to obtain an alignment mapping, thereby achieving the classification of specific anomalies.

[0049] In the video classification branch, the segment feature X is projected into the latent space through different linear layers, the similarity matrix G is obtained by calculating the inner product, and the softmax normalization is applied to generate the global attention map M g , re-weight the projected segment features through the attention map to obtain the global context feature X g ; Local context calibration is achieved by reusing the similarity matrix in the mask window; through the attention map M l Reweight the projected segment features to obtain the local calibration feature X l, global context features and local calibration features are adaptively fused in local-global context, dynamically balancing the importance of global temporal patterns and local subtle differences; dynamic position encoding is introduced to model the relative distance of fragments, and dynamic position encoding is embedded in the similarity matrix G as a position prior; a two-layer MLP is used for feature dimensionality reduction, and each Conv1D layer is followed by a GELU activation and dropout operation; a causal convolutional layer is used to predict fragment-level anomaly scores.

[0050] Specifically, in the video branch, the local and global adaptive temporal modeling modules are used to capture the contextual information of the video. This primarily utilizes a similarity matrix and an adaptive fusion mechanism to simultaneously model global and local dependencies. The module employs a learnable strategy to dynamically balance the importance of global temporal patterns with local detail differences. It also introduces a distance adjustment matrix to assign higher weights to closer elements to improve temporal modeling. Specifically, the segment features x are first projected into the latent space through different linear layers. The similarity matrix G is then calculated by computing the inner product, which can be expressed as:

[0051] G=L q (X)·L k (X) T

[0052] Among them, L q , L k are two different linear layers, and T represents the transpose operation.

[0053] Then, softmax normalization is applied to generate the global attention map M g , which can be expressed as:

[0054]

[0055] Where W is the hidden dimension in the latent space.

[0056] Second, the projected segment features are reweighted by the attention map to obtain the global context feature X g , which can be expressed as:

[0057] X g =M g ·L v (X)

[0058] Among them, L v is a linear layer.

[0059] Although the above method helps to model the global context, it inevitably introduces long-range noise. To this end, we implement local context calibration by reusing the similarity matrix in the mask window, which can be expressed as:

[0060]

[0061] where ω is the masked window size and T is the maximum length of the input sequence. The i-th segment only interacts with its neighborhood of window size ω. The window is lower bounded by the earliest traceable observation moment and upper bounded by the maximum length of the sequence.

[0062] Similarly, through the attention map M l Reweighting the projected segment features can obtain the local calibration feature X l , which can effectively capture slight changes and achieve feature enhancement in local neighborhoods. Subsequently, we use a learnable approach instead of direct concatenation or average pooling to achieve local-global context adaptive fusion, enabling the model to dynamically balance the importance of global temporal patterns and local subtle differences. The fusion process is described as follows:

[0063] X c =α·X g +(1-α)·X l

[0064] X d =LN(X+L h (Norm(X c )))

[0065] Where α and 1-α represent the global weight and local weight of context fusion respectively, and Norm represents the combination of power normalization and L2 normalization. Then, a linear layer L is used. h , residual connection and layer normalization LN to obtain context features X d .

[0066] In addition, considering the importance of position information, we introduce dynamic position encoding to model the relative distance of fragments, as follows:

[0067] D=exp(-|γ(ij) 2 +β|)

[0068] Here, i and j represent the absolute positions of the two segments, and γ and β are learnable weights and bias terms.

[0069] Specifically, dynamic position encoding is embedded as a position prior into the similarity matrix G, i.e., G←G+D, to avoid affecting the original feature distribution. Compared to fixed position encoding, dynamic position encoding is adaptive and can adapt to different video lengths. Its Gaussian-like kernel mechanism effectively suppresses the influence of long-range noise, emphasizes the importance of relationships between neighboring segments, and enhances sensitivity to nonlinear patterns.

[0070] Secondly, in order to obtain high-level semantic representation, a two-layer MLP is used to reduce the feature dimension. Each Conv1D layer is followed by a GELU activation and dropout operation. This process is represented as follows

[0071] X e =Dropout(GELU(Conv1D(X d )))

[0072] X f =Dropout(GELU(Conv1D(X e )))

[0073] Subsequently, a causal convolutional layer is used to predict segment-level anomaly scores, which considers both current and historical observations to obtain more reliable results. The classifier is of the form:

[0074] F=σ(L t (X f ))

[0075] Among them L t is a causal convolutional layer with kernel size t, σ is the sigmoid function, F i is the anomaly score of the ith segment.

[0076] The Top-K mechanism is used to select K high anomaly confidence scores in abnormal and normal videos as video-level predictions, and then the classification loss L is calculated using the binary cross entropy between the video-level predictions and the true values. bce .

[0077] In the text alignment branch, the WordNet database is used to semantically enhance the categories, enabling CLIP to better understand different ways of expressing events. First, synonyms or related concepts of the category label are obtained to expand the semantic scope of the category. Then, the semantically enhanced category is converted into a category token using CLIP's Tokenizer. Second, the enhanced category token is substituted into the Prompt structure, and a learnable prompt containing m context tokens is added to form a complete text input sequence. Among them, the category token t init is placed in the middle of the text input sequence; finally, CLIP’s text encoder receives the text input sequence as input and generates the final category embedding.

[0078] Use visual context to improve class text embeddings to make concise text labels more accurate, and use the visual embeddings in the abnormal clips obtained from the video classification branch as video-level cues to assist the text alignment branch; calculate the matching similarity between all class embeddings and frame-level visual features to obtain an alignment map; for each row, select the top K similarities and calculate their average to measure the degree of alignment between the video and the current class; then obtain a vector S that represents the similarity between this video and all categories; finally, calculate the alignment loss L through cross entropy nce , by optimizing so that the matching score of the correct category is the highest.

[0079] Example 2

[0080] In this embodiment 2, a weakly supervised video anomaly detection method based on prompt learning is proposed, which aims to achieve efficient context modeling and enhance semantic identifiability based on the large-scale pre-trained model CLIP, thereby improving the coarse-grained and fine-grained anomaly detection capabilities. The specific model structure is as follows Figure 1 shown.

[0081] First, inspired by the pioneering work VadCLIP, a two-branch framework is introduced: a coarse-grained video classification branch and a fine-grained video-text alignment branch. In the video branch, video features are first extracted using the frozen CLIP image encoder. Then, temporal information is obtained through local and global adaptive temporal modeling of the video, achieving local-global context adaptive fusion. Binary classification is performed after two layers of MLP to classify the video as abnormal or normal. In the text branch, the semantic representation of the original word embedding is enriched using the vocabulary database WordNet. External semantic information is fused with the original word embedding through weighted averaging, and anomalies in the video are used as video-level cues to assist the text branch. Finally, the matching similarity between the text embedding and the frame-level visual features is calculated to obtain an alignment mapping, thereby achieving classification of specific anomalies.

[0082] In the video branch, the local and global adaptive temporal modeling modules are used to capture the contextual information of the video. This primarily utilizes a similarity matrix and an adaptive fusion mechanism to simultaneously model global and local dependencies. The module employs a learnable strategy to dynamically balance the importance of global temporal patterns with local detail differences. It also introduces a distance adjustment matrix to assign higher weights to closer elements to improve temporal modeling. Specifically, the segment features X are first projected into the latent space through different linear layers. The similarity matrix G is then calculated by computing the inner product, which can be expressed as:

[0083] G=L q (X)·L k (X) T

[0084] Among them, L q, L k are two different linear layers, and T represents the transpose operation.

[0085] Then, softmax normalization is applied to generate the global attention map M g , which can be expressed as:

[0086]

[0087] Where W is the hidden dimension in the latent space.

[0088] Second, the projected segment features are reweighted by the attention map to obtain the global context feature X g , which can be expressed as:

[0089] X g =M g ·L v (X)

[0090] Among them, L v is a linear layer.

[0091] Although the above method helps to model the global context, it inevitably introduces long-range noise. To this end, local context calibration is achieved by repeatedly using the similarity matrix in the mask window, which can be expressed as:

[0092]

[0093] where ω is the masked window size and T is the maximum length of the input sequence. The i-th segment only interacts with its neighborhood of window size ω. The window is lower bounded by the earliest traceable observation moment and upper bounded by the maximum length of the sequence.

[0094] Similarly, through the attention map M l Reweighting the projected segment features can obtain the local calibration feature X l , which can effectively capture slight changes and achieve feature enhancement in local neighborhoods. Subsequently, a learnable approach is adopted to achieve local-global context adaptive fusion instead of direct concatenation or average pooling, enabling the model to dynamically balance the importance of global temporal patterns and local subtle differences.

[0095] The fusion process is described as follows:

[0096] X c =α·X g +(1-α)·X l

[0097] X d =LN(X+L h (Norm(X c )))

[0098] Where α and 1-α represent the global weight and local weight of context fusion respectively, and Norm represents the combination of power normalization and L2 normalization. Then, a linear layer L is used. h , residual connection and layer normalization LN to obtain context features X d .

[0099] In addition, considering the importance of position information, we introduce dynamic position encoding to model the relative distance of fragments, as follows:

[0100] D=exp(-|γ(ij) 2 +β|)

[0101] Here, i and j represent the absolute positions of the two segments, and γ and β are learnable weights and bias terms.

[0102] Specifically, dynamic position encoding is embedded as a position prior into the similarity matrix G, i.e., G←G+D, to avoid affecting the original feature distribution. Compared to fixed position encoding, dynamic position encoding is adaptive and can adapt to different video lengths. Its Gaussian-like kernel mechanism effectively suppresses the influence of long-range noise, emphasizes the importance of relationships between neighboring segments, and enhances sensitivity to nonlinear patterns.

[0103] Secondly, in order to obtain high-level semantic representation, a two-layer MLP is used to reduce the feature dimension. Each Conv1D layer is followed by a GELU activation and dropout operation. This process is represented as follows

[0104] X e =Dropout(GELU(Conv1D(X d )))

[0105] X f =Dropout(GELU(Conv1D(X e )))

[0106] Subsequently, a causal convolutional layer is used to predict segment-level anomaly scores, which considers both current and historical observations to obtain more reliable results. The classifier is of the form:

[0107] F=σ(L t (X f ))

[0108] Among them L t is a causal convolutional layer with kernel size t, σ is the sigmoid function, F i is the anomaly score of the ith segment.

[0109] The Top-K mechanism is used to select K high anomaly confidence scores in abnormal and normal videos as video-level predictions, and then the classification loss L is calculated using the binary cross entropy between the video-level predictions and the true values. bce .

[0110] In the text branch, text labels are words or phrases, which are too concise and fail to fully summarize the complexity of abnormal events. To leverage the robust transferability of learning text embeddings, we draw inspiration from CoOp and add learnable prompts to the original class embeddings. Specifically, in the CoOp approach, hand-crafted prompts such as "A photo of..." serve as fixed templates. However, this example employs learnable prompts, rather than fixed sentences. Instead, the model learns the optimal prompt format. During training, these learnable tokens are continuously adjusted, allowing the model to acquire contextual information that best suits the category. Based on this, this example further utilizes the WordNet lexical database to semantically enhance categories, enabling CLIP to better understand diverse event representations. First, synonyms or related concepts of category labels are obtained to expand the semantic scope of the categories and improve the model's adaptability to diverse representations. For example, training data may only contain the category label "explosion," while people in the real world may use "burst" or "detonation" to describe the same event. Introducing external knowledge can reduce matching errors caused by such terminology discrepancies. Then, the semantically enhanced categories are converted into category tags using CLIP’s Tokenizer, denoted as:

[0111] t init =Tokenizer(Enhance Label)

[0112] Secondly, the enhanced category token is substituted into the Prompt structure, and a learnable prompt {l1,...,l m}, forming a complete text input sequence:

[0113] t p ={l1,...,t init ,...l m}

[0114] Category label t init is placed in the middle of the sequence. p Position embedding is added to provide position information. Finally, the text encoder of CLIP receives t p As input, and generate the final category embedding

[0115] To further improve the representation ability of text labels for abnormal events, visual context is used to refine the class text embeddings to make the concise text labels more accurate, and the visual embeddings in the abnormal clips obtained from the video branch are used as video-level cues to assist the text branch.

[0116] Finally, the matching similarities between all class embeddings and frame-level visual features are calculated to obtain the alignment map N. For each row, the top K similarities are selected and their average is calculated to measure the degree of alignment between the video and the current class. Then a vector S = {s1,...s n}, which represents the similarity between this video and all categories. Calculate multi-class prediction:

[0117]

[0118] where p i is the prediction about the i-th category, τ is the temperature hyperparameter, that is, to determine which category the video is most likely to belong to. Finally, the alignment loss L is calculated by cross entropy nce , by optimizing to maximize the matching score of the correct category, thus improving the classification effect.

[0119] A text-video contrast loss function L is designed cl , expressed as:

[0120]

[0121] Based on the relevance score of each frame to the query (text embedding), the video frame embeddings are weighted and summed to generate a weighted feature representation of the entire video, ultimately obtaining the final score for each text-video match. This can highlight frames that are more relevant to the query, reduce the interference of irrelevant frames on the overall video features, improve matching accuracy, increase the similarity of correct text-video pairs, and reduce the similarity of incorrect matches.

[0122] Finally, a contrast loss is introduced to slightly separate the normal class embedding and the other abnormal class embedding. First, the cosine similarity between the normal class embedding and the other abnormal class embedding is calculated, and then the contrast loss L is calculated. clt , which can be expressed as:

[0123]

[0124] where t n is the normal class embedding, t aj is the j-th abnormal class embedding.

[0125] Finally, the overall objective function of the model can be expressed as:

[0126] L=L bce +Lnce +L cl +λL clt .

[0127] The above-mentioned anomaly detection model proposed in this embodiment can be trained and tested on a single RTX 4090 GPU.

[0128] Example 3

[0129] This embodiment 3 provides a non-transitory computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method for weakly supervised video anomaly detection based on prompt learning is implemented. The method includes:

[0130] Obtain video data to be processed;

[0131] The acquired video data to be processed is processed using a pre-trained anomaly detection model to obtain specific classification results of abnormal events in the video; wherein, the anomaly detection model includes a video classification branch and a text alignment branch; in the video classification branch, the frozen CLIP image encoder is first used to extract video features, and then the temporal information is obtained through local and global adaptive temporal modeling of the video to achieve local-global context adaptive fusion, and binary classification is performed after two layers of MLP to obtain the classification of video abnormality or normality; in the text alignment branch, the semantic representation of the original word embedding is enriched by the vocabulary database WordNet, and the external semantic information is fused with the original word embedding by weighted averaging, and the anomalies in the video are used as video-level prompts to assist the text branch; finally, the matching similarity between the text embedding and the frame-level visual features is calculated to obtain the alignment mapping, thereby achieving the classification of specific anomalies.

[0132] Example 4

[0133] This embodiment 4 provides a computer device including a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the above-described method for weakly supervised video anomaly detection based on prompt learning, the method comprising:

[0134] Obtain video data to be processed;

[0135] The acquired video data to be processed is processed using a pre-trained anomaly detection model to obtain specific classification results of abnormal events in the video; wherein, the anomaly detection model includes a video classification branch and a text alignment branch; in the video classification branch, the frozen CLIP image encoder is first used to extract video features, and then the temporal information is obtained through local and global adaptive temporal modeling of the video to achieve local-global context adaptive fusion, and binary classification is performed after two layers of MLP to obtain the classification of video abnormality or normality; in the text alignment branch, the semantic representation of the original word embedding is enriched by the vocabulary database WordNet, and the external semantic information is fused with the original word embedding by weighted averaging, and the anomalies in the video are used as video-level prompts to assist the text branch; finally, the matching similarity between the text embedding and the frame-level visual features is calculated to obtain the alignment mapping, thereby achieving the classification of specific anomalies.

[0136] Example 5

[0137] This embodiment 5 provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the above-described method for weakly supervised video anomaly detection based on prompt learning. The method includes:

[0138] Obtain video data to be processed;

[0139] The acquired video data to be processed is processed using a pre-trained anomaly detection model to obtain specific classification results of abnormal events in the video; wherein, the anomaly detection model includes a video classification branch and a text alignment branch; in the video classification branch, the frozen CLIP image encoder is first used to extract video features, and then the temporal information is obtained through local and global adaptive temporal modeling of the video to achieve local-global context adaptive fusion, and binary classification is performed after two layers of MLP to obtain the classification of video abnormality or normality; in the text alignment branch, the semantic representation of the original word embedding is enriched by the vocabulary database WordNet, and the external semantic information is fused with the original word embedding by weighted averaging, and the anomalies in the video are used as video-level prompts to assist the text branch; finally, the matching similarity between the text embedding and the frame-level visual features is calculated to obtain the alignment mapping, thereby achieving the classification of specific anomalies.

[0140] In summary, the method and system for unsupervised video anomaly detection based on prompt learning described in the embodiments of the present invention, for the first time, introduces a video local and global adaptive temporal modeling module into a dual-branch framework, namely the coarse-grained video classification branch and the fine-grained video text alignment branch. This is superior to methods that integrate local and global context networks through parallel or serial structures, while minimizing parameters and computational load. For the first time, semantic enhancement is performed by combining the vocabulary database WordNet, optimizing the representation of category tokens based on the Prompt mechanism, enabling CLIP to better understand different expressions and improve generalization capabilities. A text-video contrast loss module is designed to ensure that correct text-video pairs have higher similarity, reduce the similarity of incorrect matches, reduce the impact of irrelevant frames, and achieve more accurate video anomaly detection.

[0141] The VadCLIP method uses the proposed local-global temporal adapter for temporal modeling in the video branch, but its use of a local and global serial structure introduces additional parameters and computational requirements. The video local and global adaptive temporal modeling module introduced in the present invention captures local and global dependencies simultaneously by reusing the similarity matrix, minimizing parameters and computational load, and efficiently performing temporal modeling. In the text branch, the present invention combines the vocabulary database WordNet for semantic enhancement, improving CLIP's ability to understand different expressions and making CLIP's text representation more semantically rich and generalizable. A text-video comparison loss function is also designed to give correct text-video pairs a higher similarity, reduce the similarity of incorrect matches, and reduce the impact of irrelevant frames, thereby improving the model's ability to distinguish and making the matching of text and video more accurate.

[0142] The present invention utilizes the strong correlation ability of CLIP between vision and language, and combines it with the vocabulary database WordNet for semantic enhancement to achieve effective transfer of pre-trained knowledge to downstream tasks. This method accurately captures abnormal information through efficient temporal modeling, and designs a text-video contrast loss function to optimize the cross-modal matching effect, thereby improving the accuracy of anomaly detection. Weakly supervised video anomaly detection based on spatio-temporal prompts (Spatio-Temporal Prompts, STPrompt) decomposes the weakly supervised video anomaly detection task into temporal anomaly detection and spatial anomaly localization. A spatial attention aggregation mechanism is designed in STPrompt to filter out irrelevant background in temporal anomaly detection. In addition, a training-free anomaly localization method based on large language models (LLMs) is introduced to obtain fine-grained text prompts for spatial anomaly localization. This method can also improve the accuracy of abnormal event detection.

[0143] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0144] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0145] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0146] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0147] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solutions disclosed in the present invention without the need for creative work should be included in the scope of protection of the present invention.

Claims

1. A weakly supervised video anomaly detection method based on prompt learning, characterized in that include: Obtain video data to be processed; The acquired video data to be processed is processed using a pre-trained anomaly detection model to obtain specific classification results of abnormal events in the video; wherein, the anomaly detection model includes a video classification branch and a text alignment branch; in the video classification branch, the frozen CLIP image encoder is first used to extract video features, and then the temporal information is obtained through local and global adaptive temporal modeling of the video to achieve local-global context adaptive fusion, and binary classification is performed after two layers of MLP to obtain the classification of video abnormality or normality; in the text alignment branch, the semantic representation of the original word embedding is enriched by the vocabulary database WordNet, and the external semantic information is fused with the original word embedding by weighted averaging, and the anomalies in the video are used as video-level prompts to assist the text branch; finally, the matching similarity between the text embedding and the frame-level visual features is calculated to obtain the alignment mapping, thereby achieving the classification of specific anomalies.

2. The method for weakly supervised video anomaly detection based on prompt learning according to claim 1, characterized in that In the video classification branch, the segment features X are projected into the latent space through different linear layers, the similarity matrix G is obtained by calculating the inner product, and the softmax normalization is applied to generate the global attention map M g , re-weight the projected segment features through the attention map to obtain the global context feature X g ; Local context calibration is achieved by reusing the similarity matrix in the mask window; through the attention map M l Reweight the projected segment features to obtain the local calibration feature X l , global context features and local calibration features are adaptively fused in local-global context, dynamically balancing the importance of global temporal patterns and local subtle differences; dynamic position encoding is introduced to model the relative distance of fragments, and dynamic position encoding is embedded in the similarity matrix G as a position prior; a two-layer MLP is used for feature dimensionality reduction, and each Conv1D layer is followed by a GELU activation and dropout operation; a causal convolutional layer is used to predict fragment-level anomaly scores.

3. The method for weakly supervised video anomaly detection based on prompt learning according to claim 1, characterized in that In the text alignment branch, the WordNet database is used to semantically enhance the categories, enabling CLIP to better understand different ways of expressing events. First, synonyms or related concepts of the category label are obtained to expand the semantic scope of the category. Then, the semantically enhanced category is converted into a category token using CLIP's Tokenizer. Second, the enhanced category token is substituted into the Prompt structure, and a learnable prompt containing m context tokens is added to form a complete text input sequence. Among them, the category token t init is placed in the middle of the text input sequence; finally, CLIP’s text encoder receives the text input sequence as input and generates the final category embedding.

4. The method for weakly supervised video anomaly detection based on prompt learning according to claim 3, characterized in that Leveraging visual context to refine class text embeddings and make concise text labels more accurate, we use the visual embeddings in the anomaly segments obtained from the video classification branch as video-level cues to assist the text alignment branch. Calculate the matching similarity between all class embeddings and frame-level visual features to obtain an alignment map; for each row, select the top K similarities and calculate their average to measure the degree of alignment between the video and the current class; then obtain a vector S, which represents the similarity between this video and all categories; finally, calculate the alignment loss L by cross entropy nce , by optimizing so that the matching score of the correct category is the highest.

5. The method for weakly supervised video anomaly detection based on prompt learning according to claim 4, characterized in that Text-video contrast loss function L cl for: Among them, Y + Represents the set of all positive sample pairs, N represents the number of texts, M represents the number of videos, and Y ij Indicates whether the i-th text and the j-th video match, P ij Represents the softmax probability of the i-th text and the j-th video. According to the correlation score between each frame and the text embedding, the frame embeddings of the video are weighted and summed to generate a weighted feature representation of the entire video, and finally the final score of each text-video match is obtained.

6. The method for weakly supervised video anomaly detection based on prompt learning according to claim 4, characterized in that A contrast loss is introduced to distinguish the normal class embedding from the other abnormal class embeddings. First, the cosine similarity between the normal class embedding and the other abnormal class embeddings is calculated, and then the contrast loss L is calculated. clt for: Among them, t n is the normal class embedding, t aj is the j-th anomaly class embedding, T represents the transposition operation, and ||·||2 represents the L2 norm.

7. A weakly supervised video anomaly detection system based on prompt learning, characterized in that include: An acquisition module, used to acquire video data to be processed; The processing module is used to process the acquired video data to be processed using a pre-trained anomaly detection model to obtain a specific classification result of abnormal events in the video; wherein, the anomaly detection model includes a video classification branch and a text alignment branch; in the video classification branch, first, a frozen CLIP image encoder is used to extract video features, and then temporal information is obtained through local and global adaptive time modeling of the video to achieve local-global context adaptive fusion, and binary classification is performed after two layers of MLP to obtain a classification of video abnormality or normality; in the text alignment branch, the semantic representation of the original word embedding is enriched using the vocabulary database WordNet, and external semantic information is fused with the original word embedding by weighted averaging, and the anomalies in the video are used as video-level prompts to assist the text branch; finally, the matching similarity between the text embedding and the frame-level visual features is calculated to obtain an alignment mapping, thereby achieving the classification of specific anomalies.

8. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by the processor, the weakly supervised video anomaly detection method based on prompt learning is implemented as described in any one of claims 1-6.

9. A computer device, characterized in that: It includes a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the weakly supervised video anomaly detection method based on prompt learning as described in any one of claims 1 to 6.

10. An electronic device, characterized in that: include: A processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute instructions for implementing the weakly supervised video anomaly detection method based on prompt learning as described in any one of claims 1 to 6.

Images