Prototype pseudo-instance enhanced video anomaly detection method and system

Abstract

The present invention proposes a prototype pseudo-instance enhanced video anomaly detection method and system, which includes: extracting features from a video to obtain deep features, and obtaining a feature sequence based on the deep features; inputting the feature sequence into a prototype interaction layer, calculating cosine similarity using a normality prototype set; obtaining enhanced features using the cosine similarity; calculating anomaly scores using a classifier; screening the video using the anomaly scores to obtain an index set of extreme instances; based on the index set of extreme instances, performing low-dimensional mapping on the enhanced features corresponding to the extreme instances to obtain a deep feature embedding representation; constructing a loss function based on the deep feature embedding representation; training a model in combination with the loss function; and obtaining detection results using the trained model. The present invention dynamically generates feature context vectors using a learnable normality prototype set and an attention mechanism, embedding structured normality knowledge into the feature representation, and establishing a robust normality benchmark.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and artificial intelligence technology, and in particular to a prototype pseudo-instance enhanced video anomaly detection method and system. Background Art

[0002] Computer vision, a core branch of artificial intelligence, is dedicated to using algorithms to perceive, analyze, and understand image and video data. Its technological development has undergone a transformation from traditional image processing to intelligent processing driven by deep learning. Early computer vision relied primarily on handcrafted features (such as SIFT and HOG) and statistical learning methods, completing basic tasks such as object recognition and image classification through edge detection and feature matching.

[0003] Existing methods, such as MIL-based RTFM and contrastive learning CLAWS, improve performance through ranking loss or clustering, but fail to address the problems of structured embedding of normal prototypes and contrast enhancement of extreme instances, resulting in high model complexity and limited generalization capability. Specifically, normal segments dominate in abnormal videos, making it difficult for the model to learn fine-grained abnormal features; the lack of explicit modeling of normal patterns leads to a fuzzy anomaly detection benchmark; and the lack of targeted optimization of boundary instances (such as the highest / lowest scoring segments) affects positioning accuracy. Summary of the Invention

[0004] In view of the above situation, the main purpose of the present invention is to propose a prototype pseudo-instance enhanced video anomaly detection method and system to solve the above technical problems.

[0005] The present invention proposes a prototype pseudo-instance enhanced video anomaly detection method, which comprises the following steps:

[0006] Step 1: Input the video into the video anomaly detection model, perform time segmentation and feature extraction on the video in sequence to obtain deep features, and obtain a feature sequence based on the deep features;

[0007] Step 2: Input the feature sequence into the prototype interaction layer and calculate the cosine similarity through the normal prototype set;

[0008] Attention weights are generated based on cosine similarity;

[0009] Get the normal context vector through attention weights;

[0010] After linear transformation of the normal context vector, residual connection is performed with the deep feature to obtain enhanced features;

[0011] Step 3: Based on the enhanced features, the anomaly score is calculated by the classifier;

[0012] The videos are filtered by anomaly scores to obtain anomaly sets and normal sets respectively. The anomaly sets and normal sets are merged to obtain an index set of extreme instances.

[0013] Step 4: Based on the index set of extreme instances, the enhanced features corresponding to the extreme instances are low-dimensionally mapped to obtain a deep feature embedding representation;

[0014] Constructing a total supervision contrast loss based on deep feature embedding representation;

[0015] The video anomaly detection model is trained by combining the total supervised contrast loss and the multi-instance learning loss to obtain a trained video anomaly detection model.

[0016] Step 5: Input the video into the trained video anomaly detection model and output the video-level anomaly classification results and instance-level anomaly localization scores.

[0017] The present invention also proposes a prototype pseudo-instance enhanced video anomaly detection system, the system comprising:

[0018] Feature extraction module, used to:

[0019] The video is input into the video anomaly detection model, and the video is sequentially segmented and feature extracted to obtain deep features, and a feature sequence is obtained based on the deep features;

[0020] Enhanced feature modules for:

[0021] The feature sequence is input into the prototype interaction layer, and the cosine similarity is calculated through the normal prototype set;

[0022] Attention weights are generated based on cosine similarity;

[0023] Get the normal context vector through attention weights;

[0024] After linear transformation of the normal context vector, residual connection is performed with the deep feature to obtain enhanced features;

[0025] Screening module for:

[0026] Based on the enhanced features, the anomaly score is calculated by the classifier;

[0027] The videos are filtered by anomaly scores to obtain anomaly sets and normal sets respectively. The anomaly sets and normal sets are merged to obtain an index set of extreme instances.

[0028] Training modules for:

[0029] Based on the index set of extreme instances, the enhanced features corresponding to the extreme instances are mapped into low dimensions to obtain a deep feature embedding representation;

[0030] Constructing a total supervision contrast loss based on deep feature embedding representation;

[0031] The video anomaly detection model is trained by combining the total supervised contrast loss and the multi-instance learning loss to obtain a trained video anomaly detection model.

[0032] Result output module, used for:

[0033] The video is input into the trained video anomaly detection model, which outputs the video-level anomaly classification results and instance-level anomaly localization scores.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. This paper dynamically generates feature context vectors through a learnable normality prototype set and an attention mechanism, embeds structured normality knowledge into feature representations, and establishes a robust normality benchmark.

[0036] 2. Based on the model's predicted scores, it selects pseudo-positive / negative samples with extreme scores. It then uses a targeted contrast loss to enforce feature separation of difficult instances and sharpen the decision boundary. These two modules work together to enhance both the ability to perceive normal patterns and detect anomalies during the feature learning phase.

[0037] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flowchart of a prototype pseudo-instance enhanced video anomaly detection method proposed by the present invention;

[0039] Figure 2 This is a framework diagram of a prototype pseudo-instance enhanced video anomaly detection method proposed in the present invention;

[0040] Figure 3 This is a schematic diagram of the overall framework of a prototype pseudo-instance enhanced video anomaly detection system proposed in this invention. DETAILED DESCRIPTION

[0041] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0042] These and other aspects of the embodiments of the present invention will become clear with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0043] See also Figure 1 and Figure 2 The embodiment of the present invention proposes a prototype pseudo-instance enhanced video anomaly detection method, which includes the following steps:

[0044] Step 1: Input the video into the video anomaly detection model, perform time segmentation and feature extraction on the video in sequence to obtain deep features, and obtain a feature sequence based on the deep features.

[0045] Specifically, in this step, after the video is input into the video anomaly detection model, the video is evenly divided into a certain number of non-overlapping segments, each segment contains 16 frames and the time span is 1 second;

[0046] A 10-crop spatial enhancement is performed on each clip: five 224×224 pixel regions are cropped from the four corners and the center of the video frame and horizontally flipped, generating a total of 10 enhanced samples.

[0047] Step 2: Input the feature sequence into the prototype interaction layer and calculate the cosine similarity through the normal prototype set;

[0048] Attention weights are generated based on cosine similarity;

[0049] Get the normal context vector through attention weights;

[0050] After linear transformation of the normal context vector, residual connection is performed with the deep feature to obtain enhanced features;

[0051] In step 2, the feature sequence is input into the prototype interaction layer, and the cosine similarity is calculated through the normal prototype set. The relationship between the corresponding process is:

[0052] ;

[0053] in, represents the cosine similarity, Indicates the deep features, Indicates the Characteristics of a normal prototype, represents the L2 norm, represents transpose;

[0054] The attention weight is generated based on the cosine similarity, and the relationship between the corresponding process is:

[0055] ;

[0056] in, Indicates the The deep feature pair The feature attention weight of the normal prototype, represents the temperature coefficient, The total number of features representing the normality prototype;

[0057] The normal context vector is obtained by the attention weight, and the relationship between the corresponding process is:

[0058] ;

[0059] in, represents the normality context vector;

[0060] After linear transformation of the normal context vector, residual connection is performed with the deep feature to obtain the enhanced feature. The relationship between the corresponding process is:

[0061] ;

[0062] in, Indicates enhanced features; represents the learnable weight matrix, and ; represents the learnable bias vector, and .

[0063] Specifically, in this step, the normal prototype set is composed of five normal prototype vector sets, and each normal prototype vector has a dimension of 512 and obeys the normal distribution (0, 0.02).

[0064] Step 3: Based on the enhanced features, the anomaly score is calculated by the classifier;

[0065] The videos are filtered by anomaly scores to obtain anomaly sets and normal sets respectively. The anomaly sets and normal sets are merged to obtain an index set of extreme instances.

[0066] In step 3, based on the enhanced features, the anomaly score is calculated by the classifier. The relationship between the corresponding process is:

[0067] ;

[0068] in, represents the anomaly score, Indicates that it is processed by the activation function. Indicates processing through a two-layer multilayer perceptron classifier.

[0069] Specifically, in this step, extreme instances are selected based on the anomaly score:

[0070] Select a certain number of indexes of the highest-scoring instances to form an anomaly set;

[0071] Select a certain number of indexes of the lowest-scoring instances to form a normal set;

[0072] The abnormal set and the normal set are aggregated into an index set of extreme instances.

[0073] Step 4: Based on the index set of extreme instances, the enhanced features corresponding to the extreme instances are low-dimensionally mapped to obtain a deep feature embedding representation;

[0074] Constructing a total supervision contrast loss based on deep feature embedding representation;

[0075] The video anomaly detection model is trained by combining the total supervised contrast loss and the multi-instance learning loss to obtain a trained video anomaly detection model.

[0076] In step 4, based on the index set of extreme instances, the enhanced features corresponding to the extreme instances are low-dimensionally mapped to obtain a deep feature embedding representation. The relationship between the corresponding process is:

[0077] ;

[0078] in, Indicates the The embedded representation of deep features, Indicates processing by the projection head, represents the learnable projection weight matrix, represents the learnable projection bias vector;

[0079] Constructing the total supervised contrast loss based on the deep feature embedding representation, the specific steps are as follows:

[0080] The index set of other instances is obtained through deep feature embedding representation. The relationship between the corresponding process is:

[0081] ;

[0082] in, Represents the index collection of other instances, represents the index of the anchor instance in the set of extreme instances, Indicates the index of other instances in the extreme instance set except the current anchor instance, Indicates the index of the current anchor instance, represents the pseudo-labeling operation of extracting instances;

[0083] The dynamic weight of the negative sample is obtained by deep feature embedding representation. The relationship between the corresponding process is:

[0084] ;

[0085] in, represents the dynamic weight of negative samples, Indicates the negative samples, Indicates the negative samples, represents the temporary index of the negative sample, represents the weight temperature coefficient;

[0086] Through the index set of other instances and the dynamic weight of negative samples, the supervised contrast loss is obtained. The relationship between the process is:

[0087] ;

[0088] in, Indicates the The supervised contrast loss for each instance, represents feature decoupling regularization, Represents the dynamic temperature coefficient that is adaptively adjusted based on the characteristic differences of the sample pairs, and Both represent the results of structural decoupling of features. Represents the projection characteristics of other extreme instances, Represents the weight coefficient of the feature decoupling regularization term;

[0089] All supervised contrast losses are summed up to get the total supervised contrast loss. The relationship between the corresponding process is:

[0090] ;

[0091] in, represents the total supervised contrast loss, represents the set of extreme instances;

[0092] The video anomaly detection model is trained by combining the total supervised contrast loss and the multi-instance learning loss to obtain the trained video anomaly detection model, where the total loss is:

[0093] ;

[0094] in, represents the total loss, represents the uncertainty parameter related to the multi-instance learning task, represents the uncertainty parameter of the total supervision comparison task, represents the multi-instance learning loss, represents the basic mixing coefficient, represents the regularization coefficient, represents the prototype regularization term.

[0095] Step 5: Input the video into the trained video anomaly detection model and output the video-level anomaly classification results and instance-level anomaly localization scores.

[0096] To verify the effectiveness of this invention, we conducted comprehensive tests on two major datasets: SH-AUC and UCF-ADC. SH-AUC contains 437 videos from 13 different scenarios, covering common abnormal events; UCF-ADC contains 1,900 long videos, covering more complex abnormal events, ensuring the reliability of the model's generalization ability assessment.

[0097] Comparison with Existing Technologies: Experiments were conducted on two commonly used weakly supervised video anomaly detection datasets, SH-AUC and UCF-ADC, using AUC (Area Under the ROC Curve) as the evaluation metric to measure the model's anomaly detection performance. Comparisons were also conducted with several recently published state-of-the-art methods, which utilize different features (C3D, I3D, ViT) and techniques. As shown in Tables 1 and 2, our proposed method achieves or approaches the state-of-the-art performance, while also offering significant advantages in terms of lightweightness.

[0098] Table 1 Comparison with existing technology

[0099]

[0100] Table 2 Lightweight comparison

[0101]

[0102] To verify the effectiveness of the proposed (SND-VAD) prototype interaction layer and supervised contrast (PIDE) loss and the synergy between them, four model configurations are also compared: Baseline (only ViT features), Baseline + Protolnteract, Baseline + PIDE and full SND-VAD;

[0103] Tables 3 and 4 show that adding either the prototype interaction layer or the PIDE loss alone significantly improves performance. The full SND-VAD model achieves the best performance, indicating a synergistic effect between the two components.

[0104] Table 3 Ablation comparison

[0105]

[0106] Table 4 Number of normality prototypes k

[0107]

[0108] See also Figure 3 , an embodiment of the present invention further provides a prototype pseudo-instance enhanced video anomaly detection system, the system comprising:

[0109] Feature extraction module, used to:

[0110] The video is input into the video anomaly detection model, and the video is sequentially segmented and feature extracted to obtain deep features, and a feature sequence is obtained based on the deep features;

[0111] Enhanced feature modules for:

[0112] The feature sequence is input into the prototype interaction layer, and the cosine similarity is calculated through the normal prototype set;

[0113] Attention weights are generated based on cosine similarity;

[0114] Get the normal context vector through attention weights;

[0115] After linear transformation of the normal context vector, residual connection is performed with the deep feature to obtain enhanced features;

[0116] Screening module for:

[0117] Based on the enhanced features, the anomaly score is calculated by the classifier;

[0118] The videos are filtered by anomaly scores to obtain anomaly sets and normal sets respectively. The anomaly sets and normal sets are merged to obtain an index set of extreme instances.

[0119] Training modules for:

[0120] Based on the index set of extreme instances, the enhanced features corresponding to the extreme instances are mapped into low dimensions to obtain a deep feature embedding representation;

[0121] Constructing a total supervision contrast loss based on deep feature embedding representation;

[0122] The video anomaly detection model is trained by combining the total supervised contrast loss and the multi-instance learning loss to obtain a trained video anomaly detection model.

[0123] Result output module, used for:

[0124] The video is input into the trained video anomaly detection model, which outputs the video-level anomaly classification results and instance-level anomaly localization scores.

[0125] It should be understood that various components of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0126] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0127] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A prototype pseudo-instance enhanced video anomaly detection method, characterized in that The method comprises the following steps: Step 1: Input the video into the video anomaly detection model, perform time segmentation and feature extraction on the video in sequence to obtain deep features, and obtain a feature sequence based on the deep features; Step 2: Input the feature sequence into the prototype interaction layer and calculate the cosine similarity through the normal prototype set; Attention weights are generated based on cosine similarity; Get the normal context vector through attention weights; After linear transformation of the normal context vector, a residual connection is performed with the deep feature to obtain the enhanced feature; Step 3: Based on the enhanced features, the anomaly score is calculated by the classifier; The videos are filtered by anomaly scores to obtain anomaly sets and normal sets respectively. The anomaly sets and normal sets are merged to obtain an index set of extreme instances. Step 4: Based on the index set of extreme instances, the enhanced features corresponding to the extreme instances are low-dimensionally mapped to obtain a deep feature embedding representation; Constructing a total supervision contrast loss based on deep feature embedding representation; The video anomaly detection model is trained by combining the total supervised contrast loss and the multi-instance learning loss to obtain a trained video anomaly detection model. Step 5: Input the video into the trained video anomaly detection model and output the video-level anomaly classification results and instance-level anomaly localization scores; In step 2, the normal context vector is obtained by the attention weight, and the relationship between the corresponding process is: ； in, represents the normality context vector, Indicates the The deep feature pair The feature attention weight of the normal prototype, Indicates the Characteristics of a normal prototype, The total number of features representing the normality prototype; After linear transformation of the normal context vector, residual connection is performed with the deep feature to obtain the enhanced feature. The relationship between the corresponding process is: ； in, Represents enhanced features, Indicates the deep features, represents the learnable weight matrix, represents the learnable bias vector; In step 4, the total supervised contrast loss is constructed based on the deep feature embedding representation. The specific steps are as follows: The index set of other instances is obtained through deep feature embedding representation. The relationship between the corresponding process is: ； in, Represents the index collection of other instances, represents the index of the anchor instance in the set of extreme instances, Indicates the index of other instances in the extreme instance set except the current anchor instance, Indicates the index of the current anchor instance, represents the pseudo-labeling operation of extracting instances; The dynamic weight of the negative sample is obtained by deep feature embedding. The corresponding relationship is: ； in, represents the dynamic weight of negative samples, Indicates the negative samples, Indicates the negative samples, represents the temporary index of the negative sample, represents the weight temperature coefficient, Indicates the Embedded representation of deep features; Through the index set of other instances and the dynamic weight of negative samples, the supervised contrast loss is obtained. The relationship between the process is: ； in, Indicates the The supervised contrast loss for each instance, represents feature decoupling regularization, Represents the dynamic temperature coefficient that is adaptively adjusted based on the characteristic differences of the sample pairs, and Both represent the results of structural decoupling of features. Represents the projection characteristics of other extreme instances, Represents the weight coefficient of the feature decoupling regularization term; All supervised contrast losses are summed up to get the total supervised contrast loss. The relationship between the corresponding process is: ； in, represents the total supervised contrast loss, represents the set of extreme instances; The video anomaly detection model is trained by combining the total supervised contrast loss and the multi-instance learning loss to obtain the trained video anomaly detection model, where the total loss is: ； in, represents the total loss, represents the uncertainty parameter related to the multi-instance learning task, represents the uncertainty parameter of the total supervision comparison task, represents the multi-instance learning loss, represents the basic mixing coefficient, represents the regularization coefficient, represents the prototype regularization term.

2. The prototype pseudo-instance enhanced video anomaly detection method according to claim 1, characterized in that In step 2, the feature sequence is input into the prototype interaction layer, and the cosine similarity is calculated by the normal prototype set. The relationship between the corresponding process is: ； in, represents the cosine similarity, represents the L2 norm, Indicates transpose.

3. The prototype pseudo-instance enhanced video anomaly detection method according to claim 2, characterized in that: In step 2, the attention weight is generated based on the cosine similarity, and the relationship between the corresponding process is: ； in, Represents the temperature coefficient.

4. The prototype pseudo-instance enhanced video anomaly detection method according to claim 1, characterized in that In step 3, based on the enhanced features, the anomaly score is calculated by the classifier, and the relationship between the corresponding process is: ； in, represents the anomaly score, Indicates that it is processed by the activation function. Indicates processing through a two-layer multilayer perceptron classifier.

5. The prototype pseudo-instance enhanced video anomaly detection method according to claim 4, characterized in that: In step 4, based on the index set of extreme instances, the enhanced features corresponding to the extreme instances are low-dimensionally mapped to obtain a deep feature embedding representation. The relationship between the corresponding process is: ； in, Indicates processing by the projection head, represents the learnable projection weight matrix, represents the learnable projection bias vector.

6. A prototype pseudo-instance enhanced video anomaly detection system, characterized by: The system applies any one of claims 1 to 5 of the prototype pseudo-instance enhanced video anomaly detection method, and the system comprises: Feature extraction module, used to: The video is input into the video anomaly detection model, and the video is sequentially segmented and feature extracted to obtain deep features, and a feature sequence is obtained based on the deep features; Enhanced feature modules for: The feature sequence is input into the prototype interaction layer, and the cosine similarity is calculated through the normal prototype set; Attention weights are generated based on cosine similarity; Get the normal context vector through attention weights; After linear transformation of the normal context vector, residual connection is performed with the deep feature to obtain enhanced features; Screening module for: Based on the enhanced features, the anomaly score is calculated by the classifier; The videos are filtered by anomaly scores to obtain anomaly sets and normal sets respectively. The anomaly sets and normal sets are merged to obtain an index set of extreme instances. Training modules for: Based on the index set of extreme instances, the enhanced features corresponding to the extreme instances are mapped into low dimensions to obtain a deep feature embedding representation; Constructing a total supervision contrast loss based on deep feature embedding representation; The video anomaly detection model is trained by combining the total supervised contrast loss and the multi-instance learning loss to obtain a trained video anomaly detection model. Result output module, used for: The video is input into the trained video anomaly detection model, which outputs the video-level anomaly classification results and instance-level anomaly localization scores.

Images