Weakly supervised video anomaly detection method and system based on prototype orthogonality

Abstract

The application provides a weakly supervised video anomaly detection method and system based on prototype orthogonality. The method filters redundant information in time series and captures key dynamics by inputting a visual feature sequence into a selective state space model to output high-value time sequence features. A learnable prototype codebook containing a normal category prototype codebook and an abnormal category prototype codebook is given, and an end-to-end training is performed by using a multi-instance learning framework of a video level label. Prototype orthogonality constraints are applied during the training process. After the training is completed, only the distance between the high-value time sequence features and the nearest prototype in the abnormal category prototype codebook is needed, and the video anomaly score is obtained according to the distance size, so that the video anomaly detection is realized. The application introduces a global geometric constraint of "prototype orthogonality", and combines a computationally efficient selective state space model (S3M), so that the extreme lightweight of the model is realized while the high detection accuracy is ensured.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision, and in particular to a weakly supervised video anomaly detection method and system based on prototype orthogonality. Background Technology

[0002] Currently, the technical methods used for video anomaly detection can be mainly divided into the following three categories:

[0003] (1) Reconstruction-based method: learn only the normal video pattern and identify anomalies through high reconstruction error. Its advantage is that no abnormal samples are required.

[0004] (2) Classification-based methods: Directly classify normal and abnormal, with high detection accuracy, but heavily rely on costly frame-by-frame annotation (full supervision).

[0005] (3) Weakly supervised methods: To reduce annotation costs, training is performed using only video-level labels, which is the current mainstream research approach. This method typically combines techniques such as multiple instance learning (MIL) and attention mechanisms.

[0006] The two options are as follows:

[0007] (1) Cloud-based large-scale model solution: After the monitoring robot collects video, it transmits the data to a cloud server for analysis by a large-scale visual model. Although this solution has high accuracy, it has the following drawbacks: ① High latency: The back-and-forth data transmission leads to untimely response, which cannot meet the real-time alarm requirements of emergency situations; ② High cost: Continuous cloud services and API calls are expensive, especially for large-scale robot clusters; ③ Privacy and security risks: Uploading monitoring data to the cloud increases the risk of privacy leaks.

[0008] (2) Traditional localization solutions: To address cloud-based issues, some solutions attempt to deploy the robot locally. However, traditional video analysis models (such as those based on 3D CNN or Transformer) require massive computation and have numerous model parameters, placing extremely high demands on the robot's onboard computing units (such as GPU / NPU), resulting in: ① soaring hardware costs; ② enormous power consumption, severely shortening the robot's battery life; ③ severe overheating, affecting the stability and lifespan of the robot's hardware.

[0009] Despite the progress made in existing technologies, the following profound technical shortcomings and bottlenecks still exist:

[0010] (1) Inefficient temporal modeling: When processing long videos containing a lot of redundant information, existing models face a dilemma: traditional models (such as RNN) are difficult to capture long-range dependencies, while new models (such as Transformer) have good performance but high computational complexity and are not suitable for real-time detection.

[0011] (2) The serious “representation bottleneck” problem: “Anomalies” themselves contain a variety of visually distinct patterns (such as fires and fights). Existing methods usually force these multimodal anomalies to be mapped to a single feature region, which leads to serious confusion between different anomalies and between normal and abnormal features, fundamentally limiting the detection accuracy.

[0012] (3) Limitations of reasoning logic: Existing methods rely too much on "semantic recognition" (such as identifying objects), while their ability to detect many anomalies that only manifest as dynamic pattern changes (such as stealth and abnormal equipment vibration) is very weak.

[0013] (4) Existing technologies are caught in a dilemma in the field of robot monitoring: cloud solutions are costly and have high latency; traditional local solutions have high power consumption and demanding computing power requirements, making it difficult to achieve lightweight edge deployment. Summary of the Invention

[0014] In view of the above, the main objective of this invention is to propose a weakly supervised video anomaly detection method and system based on prototype orthogonality to solve the aforementioned technical problems.

[0015] This invention proposes a weakly supervised video anomaly detection method based on prototype orthogonality, the method comprising the following steps:

[0016] Step 1: Given a video stream as the training set, use a pre-trained visual encoder to extract features and obtain a visual feature sequence.

[0017] Step 2: Input the visual feature sequence into the selective state space model to filter redundant information in the time series and capture key dynamics, and output high-value time series features.

[0018] Step 3: Given a learnable prototype codebook containing prototype codebooks of normal categories and anomaly categories, the pre-trained visual encoder, the selective state space model, and the learnable prototype codebook constitute a video anomaly detection model.

[0019] Step 4: Based on high-value temporal features and learnable prototype codebooks, the video anomaly detection model is trained end-to-end using a video-level labeled multi-instance learning framework. Prototype orthogonality constraints are applied during training to force the feature subspaces formed by the normal category prototype codebook and the anomaly category prototype codebook to be mutually orthogonal, thus obtaining the trained video anomaly detection model.

[0020] Step 5: Input the video to be detected into the trained video anomaly detection model, obtain the high-value temporal features of the video to be detected, calculate the distance between the high-value temporal features of the video to be detected and the nearest prototype in the anomaly category prototype codebook, and obtain the video anomaly score based on the distance to achieve video anomaly detection.

[0021] This invention also proposes a weakly supervised video anomaly detection system based on prototype orthogonality, wherein the system applies the weakly supervised video anomaly detection method based on prototype orthogonality as described above, and the system includes:

[0022] The feature extraction module is used for:

[0023] Given a video stream as the training set, a visual feature sequence is obtained by using a pre-trained visual encoder for feature extraction.

[0024] The timing processing module is used for:

[0025] Visual feature sequences are input into a selective state-space model to filter redundant information in the time series and capture key dynamics, outputting high-value time series features.

[0026] The multimodal representation module is used for:

[0027] Given a learnable prototype codebook containing prototype codebooks of normal categories and anomaly categories, a pre-trained visual encoder, a selective state-space model, and the learnable prototype codebook constitute a video anomaly detection model.

[0028] The constraint optimization module is used for:

[0029] Based on high-value temporal features and learnable prototype codebooks, a video-level labeled multi-instance learning framework is used to train the video anomaly detection model end-to-end. During the training process, prototype orthogonality constraints are applied to force the feature subspaces formed by the normal category prototype codebook and the anomaly category prototype codebook to be mutually orthogonal, thus obtaining a well-trained video anomaly detection model.

[0030] The anomaly scoring module is used for:

[0031] The required video is input into the trained video anomaly detection model to obtain the high-value temporal features of the required video, and the distance between the high-value temporal features of the required video and the nearest prototype in the anomaly category prototype codebook is calculated. Based on the distance, the video anomaly score is obtained to achieve video anomaly detection.

[0032] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0033] (1) Superior performance: By introducing a fundamental global geometric constraint—prototype orthogonality constraint—a representation space with a clear structure and global separation of normal and abnormal features is constructed. Combined with optimal transmission codebook learning for accurate capture of anomaly multimodality, the model achieves state-of-the-art (SOTA) performance on complex datasets such as UCF-Crime, demonstrating extremely high detection accuracy and robustness. This invention has achieved SOTA detection accuracy on multiple authoritative public datasets such as UCF-Crime and ShanghaiTech.

[0034] (2) Traditional methods treat all visually distinct "anomalies" as a single concept, forcibly mapping them to the same feature region, resulting in a severe "representation bottleneck" where the model cannot learn a pure anomaly representation. This invention introduces a dynamic codebook learning paradigm based on Optimal Transport (OT) into the field of video anomaly detection for the first time, explicitly modeling the multimodality of anomalies. It allows the model to learn specific, decoupled prototype representations for multiple different anomaly patterns, greatly enhancing the model's expressive power and generalization ability. At the same time, through a distribution-level matching method, it provides a solid foundation for applying the higher-level structural constraint of "prototype orthogonality".

[0035] (3) Computationally efficient: The S3M module used in the front end has linear time complexity, which fundamentally solves the "computing power explosion" problem of architectures such as Transformer when processing long videos. The entire framework has few parameters and the model size is only at the megabyte (MB) level, which is much smaller than the mainstream models that are often tens or hundreds of MB. Compared with the Transformer-based method, it greatly reduces the computation and memory overhead when processing long videos of thousands of frames, making it suitable for real-time application deployment.

[0036] (4) Strong robustness: Traditional methods passively learn a fuzzy normal-abnormal decision boundary, resulting in a chaotic feature space structure, severe overlap between normal and abnormal feature distributions, and poor model robustness. This invention no longer passively "searches" for boundaries, but actively "builds" the structure. This invention proposes and introduces a powerful global geometric constraint—prototype orthogonality—for the first time. The prototype orthogonality design makes the feature representations learned by the model have better structure and separability, fundamentally solving the "fuzzy boundary" problem. It ensures that the normal and abnormal features learned by the model occupy structurally separated and unrelated regions in the global representation space. This makes the decision boundary between the two types of features naturally clear and stable, greatly improving the model's discriminative ability and robustness to complex scenarios, effectively solving the problem of abnormal multimodality, and improving the model's robustness to complex and diverse real-world abnormal scenarios.

[0037] (5) Weak supervision capability: The training process is based on a multi-instance learning framework with video-level labels. Training can be completed with only video-level labels, which greatly reduces the cost of data labeling and has strong practical value.

[0038] (6) Ideal for edge deployment: Existing high-precision models (especially those based on Transformer) have high computational complexity and large model size, making them difficult to deploy on edge devices (such as monitoring robots) that are limited in computing power and sensitive to power consumption.

[0039] This invention is the first to introduce the emerging and computationally efficient Selective State-Space Model (S3M) into the field of video anomaly detection as a front-end temporal processor, combined with a lightweight codebook learning module in the back-end. S3M processes long video sequences with linear time complexity, efficiently filtering temporal redundancy. The back-end codebook learning is also far more lightweight than complex classification heads or Transformer decoders.

[0040] The innovative architecture of this invention enables the entire framework to maintain state-of-the-art detection accuracy while possessing extremely low computational cost and a very small model size. It perfectly resolves the contradiction between "high accuracy" and "low power consumption," making it possible to lightweight high-performance video anomaly detection models and deploy them on edge devices such as local surveillance robots, greatly expanding the application scenarios and commercial value of the technology.

[0041] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0042] Figure 1 This is an architecture diagram of a weakly supervised video anomaly detection method based on prototype orthogonality proposed in this invention;

[0043] Figure 2 This is a schematic diagram illustrating the working principle of the Selective State Space Module (S3M).

[0044] Figure 3 A schematic diagram illustrating the principle of Optimal Transmission (OT) codebook learning;

[0045] Figure 4 This is a schematic diagram of the structure of a weakly supervised video anomaly detection system based on prototype orthogonality proposed in this invention. Detailed Implementation

[0046] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0047] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0048] Please see Figure 1 This embodiment provides a weakly supervised video anomaly detection method based on prototype orthogonality, the method comprising the following steps:

[0049] Step 1: Given a video stream as the training set, use a pre-trained visual encoder to extract features and obtain a visual feature sequence.

[0050] This step is the core starting point of the entire framework, and its task is to transform human-readable visual fragments into machine-understandable, structured mathematical language. This process can be broken down into two levels: vectorizing individual fragments and combining all vectors into a sequence.

[0051] First, vectorization of individual segments: A video stream is received. The video stream is uniformly divided into T non-overlapping segments. For the segment at time point t, it is fed into a powerful pre-trained visual encoder (such as CLIP ViT-B / 16). This encoder acts like a "visual translator," condensing the complex pixel information of the segment into an information-dense mathematical object—a feature vector. This process can be precisely described by the following formula:

[0052] ;

[0053] in, This represents the visual feature vector generated at the t-th time step (i.e., the t-th video segment); Represents a D-dimensional real vector space;

[0054] Next, the sequence is constructed: the above vectorization operation is performed on each video segment (from the 1st to the Tth), resulting in a series of... The system then combines these independent feature vectors into a complete feature sequence, strictly following their temporal order in the original video. The final construction process is defined by the following core formula:

[0055] ;

[0056] in, This indicates the final video feature sequence that is constructed and used to represent the entire video. It is no longer a single vector, but a set of vectors. This represents the temporal feature vector of each segment arranged in chronological order (t from 1 to T), where T is the total number of segments into which the video is divided.

[0057] Through the process described by the formula for video feature sequences, the originally unstructured video data is successfully transformed into a structured mathematical object containing rich spatiotemporal information—the video feature sequence F. The video feature sequence F is the direct and sole input for subsequent time-series dynamic analysis by the S3M module.

[0058] Step 2: Input the visual feature sequence into the selective state space model to filter redundant information in the time series and capture key dynamics, and output high-value time series features.

[0059] Please see Figure 2 This step aims to clarify the theoretical basis of the Selective State Space Model (S3M module) and relate it to the specific application scenario (video feature processing) of this embodiment.

[0060] The revolutionary aspect of S3M lies in its parameterization of the A, B, and C matrices, as well as the discretized step size Δ, making them dependent on the input at each time step. This endows the model with powerful selectivity: when faced with redundant normal segments, the model learns to generate a smaller B value to "ignore" the input; when a crucial anomalous clue is detected, it generates a larger B value to "lock in" and update its internal state. In this way, S3M can efficiently filter out temporal redundancy in linear time complexity, outputting a high-value feature sequence that has undergone deep context encoding and selective filtering. .

[0061] The core workflow of the S3M module is to apply a selective mechanism and iteratively solve the problem. It processes each feature in the input sequence one by one through an iterative loop from t=1 to T, generating the final high-value output sequence. At each time step t of the loop, the system precisely executes the following series of interconnected internal operations:

[0062] (1) Dynamic parameter generation:

[0063] The visual feature vector currently being processed Three parallel miniature neural networks were fed in. The goal is to use the core functions of these three networks to "examine" the current segment. The content is dynamically calculated, and a set of state transition parameters tailored to this specific moment is determined: discretization step size. Input matrix and output matrix The formula is as follows:

[0064] ;

[0065] in, This represents the dynamic step size parameter obtained by projecting the current segment content through the neural network at each time step; This represents the dynamic input matrix obtained at each time step by projecting the content of the current segment through the neural network; This represents the dynamic output matrix obtained at each time step by projecting the content of the current segment through the neural network; This represents three parallel micro-neural networks; This indicates the Softplus activation function.

[0066] The system uses a discretized version of the state update equation. To calculate the new hidden state This calculation incorporates three pieces of information: ① the hidden state from the previous time step. ② Input of the current time step ③ The input matrix, dynamically generated at the current moment, determines the importance of the input. .in, Represents the discretized state matrix. Represents the discretized input matrix. , , This represents the learnable continuous-time parameter matrix.

[0067] (2) Generate output features:

[0068] The system then applies the output equation It uses the recently updated "memory". and the customized output matrix at the current moment To calculate the high-value features of a video segment at time step t. After the loop has traversed all T time steps, all generated... They will be combined to form the final high-value feature sequence. And pass it to subsequent modules.

[0069] Step 3: Given a learnable prototype codebook containing prototype codebooks of normal categories and anomaly categories, the pre-trained visual encoder, the selective state space model, and the learnable prototype codebook constitute a video anomaly detection model.

[0070] like Figure 3 As shown, the initialization of the multimodal representation (defining a learnable codebook) is as follows:

[0071] First, high-quality time-series feature sequences are obtained from the S3M module, and then the system classifies them into "normal" categories. and "abnormal" category Each entity creates and initializes a set of learnable prototypes. This set is academically known as a codebook; the core formula is as follows:

[0072] ;

[0073] in, The entire codebook representing category c (normal or abnormal); codebook The k-th prototype vector in the codeword is also called the codeword. This represents a predefined hyperparameter that represents the category. The codebook size, i.e., the number of prototypes contained in the category.

[0074] Step 4: Based on high-value temporal features and learnable prototype codebooks, the video anomaly detection model is trained end-to-end using a video-level labeled multi-instance learning framework. Prototype orthogonality constraints are applied during training to force the feature subspaces formed by the normal category prototype codebook and the anomaly category prototype codebook to be mutually orthogonal, thus obtaining the trained video anomaly detection model.

[0075] To better train the video anomaly detection model, a weighted joint optimization was performed using optimal transmission distance, prototype orthogonality loss, and standard multi-instance learning loss. The construction process of each loss is as follows:

[0076] like Figure 3 As shown, the optimal transmission distance is:

[0077] First, the problem is formalized (treating features and codebooks as probability distributions).

[0078] Within a training batch, the system performs a formal transformation on the data:

[0079] 1. Feature distribution: This involves creating a set of all high-value feature vectors belonging to category c. The whole can be regarded as an empirical probability distribution.

[0080] 2. Codebook Distribution: Distribute the codebook of category c. The whole can be regarded as a uniform discrete probability distribution.

[0081] Next, we solve and drive the process (calculating the optimal transmission loss):

[0082] The entropy-regularized optimal transmission distance (i.e., Sinkhorn distance) between the empirical probability distribution and the uniform discrete probability distribution is calculated as the matching loss. This loss function drives the prototype vector in the codebook to move to a position that can "receive" all data features at the lowest cost, thereby adaptively learning the multimodal structure of the data; the calculation formula is as follows:

[0083] ;

[0084] in, The cost matrix is ​​represented as follows: , , This represents the cost of transferring features to the prototype vector. Represents the transmission scheme matrix. This represents the entropy regularization term. This represents the optimal transmission loss for category c; Represents the prototype codebook of category c (normal or abnormal), that is, the entire codebook of that category; This represents the set of all feature vectors belonging to category c among high-value time-series features. express Specific samples in Represents the Frobenius inner product. This represents the empirical probability distribution of feature vectors belonging to category c among high-value time-series features. This represents the uniform discrete probability distribution of the prototype codebook for category c.

[0085] Ultimately, after multiple rounds of training, the prototypes in the codebook will adaptively move to the areas with the densest data distribution, thereby accurately capturing the multimodal structure of normal behavior and various abnormal behaviors.

[0086] To globally separate the representation spaces of normal and abnormal categories, this embodiment proposes a stronger codebook-level orthogonality constraint. The normal category prototype codebook... and exception category prototype codebook Represented as matrices and , These represent the prototype codebook sizes for the normal and abnormal categories, respectively. Prototype orthogonality loss. Defined as the square of the Frobenius norm of their normalized cross-correlation matrices:

[0087] ;

[0088] in, This represents the prototype orthogonality loss. Indicates to The matrix obtained by L2 normalizing the row vectors. These are represented as matrices obtained by performing row vector L2 normalization on the normal codebook matrix and the abnormal codebook matrix, respectively. Indicates matrix transpose; The symbol represents the square of the Frobenius norm; the subscript n indicates normality; the subscript a indicates anomality; and F is an abbreviation for the Frobenius norm.

[0089] Standard Multiple Instance Learning (MIL) loss:

[0090] First, this embodiment calculates the anomaly score for each video segment. For the high-value feature vector of a video segment at time step t... Its abnormal scores Defined as its prototype codebook for exception categories The function of nearest prototype vector distance:

[0091] ;

[0092] in, Hyperparameters that indicate the steepness of the control function; This represents the distance value at time step t, which indicates how close the current video segment's features are to the overall "anomaly" concept. This represents the minimization operator, where k represents the range of values ​​for the variable. k is an index that iterates from 1 to... ; Represents the L2 norm; This represents the high-value feature vector of a video segment at time step t; This represents the k-th prototype vector in the anomaly category prototype codebook; This represents the final anomaly score at time step t. The score is usually between 0 and 1, with a higher value indicating a greater likelihood of an anomaly. This represents an exponential function.

[0093] Then, based on video-level weak tags (1 indicates an anomaly, 0 indicates normal). This embodiment uses the standard Multiple Instance Learning (MIL) loss for processing:

[0094] ;

[0095] in, This represents standard multi-instance learning. Indicates batch size; Let i represent the score sequence of the i-th video; The true label indicates whether the i-th video is normal or abnormal; It is a variant of Multiple Instance Learning (MIL), which means selecting the scores of the K highest-scoring segments to calculate the loss.

[0096] Weighted joint optimization:

[0097] Finally, the ultimate optimization objective is a weighted sum of the above loss terms. The standard Multiple Instance Learning (MIL) loss, the optimal transport matching loss, and the novel prototype orthogonality loss are weighted and summed to form the final joint optimization objective function. The total loss formula is as follows:

[0098] ;

[0099] in, This represents the total loss of the joint optimization objective function. It is a single final value used to measure how well the model performs on a batch of training data. The goal of model training is to minimize this single final value. The weights represent the standard multi-instance learning loss; it is a hyperparameter used to control the weights. The significance or influence of this factor in the total loss is indicated by the subscript mil, which represents multi-instance learning. The weighting coefficient represents the optimal transmission loss and is used to control the importance of the optimal transmission matching part in the total loss.

[0100] The subscript ot represents optimal transmission. Representing the normal category (n) and the abnormal category (n) respectively. The optimal transmission (OT) loss; The weight coefficients representing the prototype orthogonality loss control the strength with which the model enforces the separation of normal and anomalous prototypes. The subscript "ortho" indicates orthogonality. If any normal prototype is oriented close to (i.e., similar to) any anomalous prototype, their dot product will be large, leading to... The value increases. This is achieved by minimizing the total loss function. The model is forced to adjust the orientation of all prototypes so that the dot product between any normal prototype and any abnormal prototype approaches 0, which is the mathematical definition of orthogonality.

[0101] Step 5: Input the video to be detected into the trained video anomaly detection model, obtain the high-value temporal features of the video to be detected, calculate the distance between the high-value temporal features of the video to be detected and the nearest prototype in the anomaly category prototype codebook, and obtain the video anomaly score based on the distance to achieve video anomaly detection.

[0102] Please refer to Figure 4This embodiment also provides a weakly supervised video anomaly detection system based on prototype orthogonality, wherein the system applies the weakly supervised video anomaly detection method based on prototype orthogonality as described above, and the system includes:

[0103] The feature extraction module is used for:

[0104] Given a video stream as the training set, a visual feature sequence is obtained by using a pre-trained visual encoder for feature extraction.

[0105] The timing processing module is used for:

[0106] Visual feature sequences are input into a selective state-space model to filter redundant information in the time series and capture key dynamics, outputting high-value time series features.

[0107] The multimodal representation module is used for:

[0108] Given a learnable prototype codebook containing prototype codebooks of normal categories and anomaly categories, a pre-trained visual encoder, a selective state-space model, and the learnable prototype codebook constitute a video anomaly detection model.

[0109] The constraint optimization module is used for:

[0110] Based on high-value temporal features and learnable prototype codebooks, a video-level labeled multi-instance learning framework is used to train the video anomaly detection model end-to-end. During the training process, prototype orthogonality constraints are applied to force the feature subspaces formed by the normal category prototype codebook and the anomaly category prototype codebook to be mutually orthogonal, thus obtaining a well-trained video anomaly detection model.

[0111] The anomaly scoring module is used for:

[0112] The required video is input into the trained video anomaly detection model to obtain the high-value temporal features of the required video, and the distance between the high-value temporal features of the required video and the nearest prototype in the anomaly category prototype codebook is calculated. Based on the distance, the video anomaly score is obtained to achieve video anomaly detection.

[0113] To verify the effectiveness and advancement of this invention, this embodiment was comprehensively tested on two industry-recognized mainstream weakly supervised video anomaly detection datasets (ShanghaiTech, UCF-Crime), and its performance was rigorously compared with existing state-of-the-art (SOTA) methods.

[0114] 1. Comparison with the State of the Omnibus (SOTA) method:

[0115] Under weak supervision, this embodiment uses frame-level AUC (Area Under the ROC Curve) as the core evaluation metric and compares it with current mainstream and latest methods. Experimental results are shown in Table 1:

[0116] Table 1. Performance comparison with existing technologies (AUC / %)

[0117]

[0118] As shown in Table 1, this invention demonstrates superior and highly competitive performance. On the ShanghaiTech dataset, the method achieved an AUC of 98.13%, performing comparably to current state-of-the-art methods with a slight advantage. On the more complex UCF-Crime dataset, the method achieved an AUC of 88.44%, surpassing all existing comparative methods, including all the latest published results, setting a new state-of-the-art (SOTA) record for this dataset.

[0119] 2. Model benefit comparison:

[0120] The experimental results are shown in Table 2:

[0121] Table 2 Comparison of Model Efficiency;

[0122]

[0123] As shown in Table 2, this invention has an overwhelming advantage in model lightweighting. Its parameter count and computational cost are far lower than mainstream methods based on Transformer or large CNNs. This order-of-magnitude reduction means that this invention can run smoothly on robot onboard chips with power consumption and computing power only a fraction of traditional solutions, greatly reducing hardware barriers and operating costs, and truly solving the deployment challenge of real-time local video analysis for robots.

[0124] 3. Computational efficiency analysis:

[0125] In addition to its superior detection accuracy, this invention also has significant advantages in computational efficiency, mainly due to its innovative framework design.

[0126] (1) Source of core advantages: The front end of this invention adopts the Selective State Space Model (S3M) as the time processor. Unlike the Transformer architecture, which is widely used in the video field but has huge computational overhead (its self-attention mechanism has a computational complexity of the square of the sequence length), S3M can process video feature sequences with linear time complexity.

[0127] (2) Beneficial Effects: This feature enables the present invention to significantly reduce the consumption of computing resources and memory when processing long videos of thousands of frames, demonstrating superior operating efficiency. This makes it more suitable for practical application scenarios with high real-time requirements, such as urban security monitoring and resource-constrained edge computing deployments.

[0128] The theoretical motivation for this invention stems from a profound understanding and resolution of two fundamental theoretical contradictions in the traditional weakly supervised video anomaly detection (WS-VAD) paradigm. Traditional methods generally employ "semantic-associative" reasoning, attempting to learn an "anomaly" as a holistic concept, mapping it to a single region in the feature space. However, this paradigm has serious theoretical limitations.

[0129] (1) First, the neglect of “anomaly multimodality” leads to a “representation bottleneck”. Therefore, the first motivation of this invention is to abandon the single-region assumption and explicitly model the multimodality of anomalies. This invention proposes to establish a “codebook” for “anomalies” consisting of multiple learnable “prototypes”, allowing each prototype to capture a specific anomaly pattern, thereby fundamentally breaking the representation bottleneck.

[0130] (2) Second, passive learning of "separability" leads to "boundary ambiguity". Therefore, the second core motivation of this invention is to shift from "passively finding boundaries" to "actively constructing structures". This invention proposes to impose a strong constraint at the global geometric level - "prototype orthogonality". By forcing the subspaces formed by normal and abnormal prototype codebooks to be mutually orthogonal, this invention actively constructs a feature space with a clear structure and natural separability globally, theoretically ensuring the robustness of decision-making.

[0131] In summary, the motivation of this invention is to fundamentally challenge and reconstruct traditional theoretical assumptions, and to establish a new theoretical framework with higher accuracy and stronger robustness through explicit multimodal modeling and active geometric structure construction.

[0132] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0133] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0134] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0135] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A weakly supervised video anomaly detection method based on prototype orthogonality, characterized in that, The method includes the following steps: Step 1: Given a video stream as the training set, use a pre-trained visual encoder to extract features and obtain a visual feature sequence. Step 2: Input the visual feature sequence into the selective state space model to filter redundant information in the time series and capture key dynamics, and output high-value time series features. Step 3: Given a learnable prototype codebook containing prototype codebooks of normal categories and anomaly categories, the pre-trained visual encoder, the selective state space model, and the learnable prototype codebook constitute a video anomaly detection model. Step 4: Based on high-value temporal features and learnable prototype codebooks, the video anomaly detection model is trained end-to-end using a video-level labeled multi-instance learning framework. Prototype orthogonality constraints are applied during training to force the feature subspaces formed by the normal category prototype codebook and the anomaly category prototype codebook to be mutually orthogonal, thus obtaining the trained video anomaly detection model. The method of using a video-level labeled multi-instance learning framework to train a video anomaly detection model end-to-end, and applying prototype orthogonality constraints during training, specifically includes the following steps: Represent the normal category prototype codebook and the abnormal category prototype codebook in matrix form to obtain the normal codebook matrix and the abnormal codebook matrix; The squared Frobenius norm of the cross-correlation matrix after normalization of the normal codebook matrix and the abnormal codebook matrix is ​​used as the prototype orthogonality loss; A standard multi-instance learning (MIM) loss is constructed based on a video-level tagging multi-instance learning framework. The prototype orthogonal loss and the standard MIM loss are weighted and jointly used as the joint optimization objective function of the video anomaly detection model. An optimal transmission loss is introduced and weighted to be combined with the orthogonal loss and the standard MIM loss for end-to-end training. The process follows the following relationship: ； wherein, denotes the total loss of the jointly optimized objective function, denotes a weight coefficient of the standard multi-instance learning loss, denotes the standard multi-instance learning loss, denotes the optimal transport loss for the normal class and the abnormal class, respectively, denotes a weight coefficient of the optimal transport loss, denotes a weight coefficient of the prototype orthogonality loss, denotes the prototype orthogonality loss; The prototype orthogonality loss has the following relationship: ； wherein, respectively represent matrices obtained by performing row vector L2normalization on a normal codebook matrix and an abnormal codebook matrix, respectively, denotes the square of the Frobenius norm, denotes matrix transposition; The standard multi-instance learning loss has the following relationship: ； wherein, denotes the batch size, denotes the score sequence of the i denotes the true label of the i denotes that the scores of the K segments with the highest scores are selected to compute the loss.​​ Step 5: Input the video to be detected into the trained video anomaly detection model, obtain the high-value temporal features of the video to be detected, calculate the distance between the high-value temporal features of the video to be detected and the nearest prototype in the anomaly category prototype codebook, and obtain the video anomaly score based on the distance to achieve video anomaly detection.

2. The weakly supervised video anomaly detection method based on prototype orthogonality according to claim 1, characterized in that, In the score sequence of the i-th video, the calculation process of the anomaly score of the video segment at time step t follows the following formula: ； wherein, denotes a hyperparameter that controls the steepness of the control function, denotes a distance value at time step t, denotes a minimization operator, denotes the size of the anomaly class prototype codebook, denotes the L2 norm, denotes a high-value feature vector of the video segment at time step t, denotes the k-th prototype vector in the anomaly class prototype codebook, denotes the exponential function, denotes the final anomaly score at time step t.

3. The weakly supervised video anomaly detection method based on prototype orthogonality according to claim 2, characterized in that, In step 1, the method of extracting visual features using a pre-trained visual encoder to obtain a visual feature sequence specifically includes the following steps: Divide the video stream evenly into N non-overlapping segments; A pre-trained visual encoder is used to encode each non-overlapping segment, resulting in several visual feature vectors; The visual feature vectors are combined according to the temporal order in the video stream to obtain the visual feature sequence.

4. The weakly supervised video anomaly detection method based on prototype orthogonality according to claim 3, characterized in that, In step 2, the method of inputting the visual feature sequence into a selective state-space model to filter redundant information in the time series and capture key dynamics, and outputting high-value time series features specifically includes the following steps: The feature vector of the current time step in the visual feature sequence is fed into three parallel micro neural networks to obtain the state transition parameters, which include the discretization step size, the input matrix, and the output matrix. Based on the discretization step size and the input matrix, a discretized version of the state update equation is constructed. The hidden state of the previous time step and the feature vector of the current time step are used to calculate the new hidden state using the discretized version of the state update equation. The output equation is constructed based on the discretization step size and the output matrix. The new hidden state is then applied to the output equation to obtain the final output of the current step. After the loop has traversed all time steps, the final outputs generated by all time steps are combined to obtain high-value time series features.

5. The weakly supervised video anomaly detection method based on prototype orthogonality according to claim 4, characterized in that, The steps for constructing the optimal transmission loss are as follows: The set of all feature vectors belonging to category c in high-value time series features is regarded as an empirical probability distribution. The prototype codebook of category c is regarded as a uniform discrete probability distribution. The optimal transmission distance, regularized by the entropy between the empirical probability distribution and the uniform discrete probability distribution, is used as the optimal transmission loss for category c.

6. The weakly supervised video anomaly detection method based on prototype orthogonality according to claim 5, characterized in that, The optimal transmission loss for category c is expressed by the following formula: ； in, The cost matrix is ​​represented as follows: ; , codebook The k-th prototype vector in This represents the cost of transferring features to the prototype vector. Represents the transmission scheme matrix. This represents the entropy regularization term. This represents the optimal transmission loss for category c. This represents the prototype codebook for category c. This represents the set of all feature vectors belonging to category c among high-value time-series features. express One of the feature vectors, Represents the Frobenius inner product. This represents the empirical probability distribution of feature vectors belonging to category c among high-value time-series features. This represents the uniform discrete probability distribution of the prototype codebook for category c.

7. A weakly supervised video anomaly detection system based on prototype orthogonality, characterized in that, The system employs the weakly supervised video anomaly detection method based on prototype orthogonality as described in any one of claims 1 to 6, and the system comprises: The feature extraction module is used for: Given a video stream as the training set, a visual feature sequence is obtained by using a pre-trained visual encoder for feature extraction. The timing processing module is used for: Visual feature sequences are input into a selective state-space model to filter redundant information in the time series and capture key dynamics, outputting high-value time series features. The multimodal representation module is used for: Given a learnable prototype codebook containing prototype codebooks of normal categories and anomaly categories, a pre-trained visual encoder, a selective state-space model, and the learnable prototype codebook constitute a video anomaly detection model. The constraint optimization module is used for: Based on high-value temporal features and learnable prototype codebooks, a video-level labeled multi-instance learning framework is used to train the video anomaly detection model end-to-end. During the training process, prototype orthogonality constraints are applied to force the feature subspaces formed by the normal category prototype codebook and the anomaly category prototype codebook to be mutually orthogonal, thus obtaining a well-trained video anomaly detection model. The anomaly scoring module is used for: The required video is input into the trained video anomaly detection model to obtain the high-value temporal features of the required video, and the distance between the high-value temporal features of the required video and the nearest prototype in the anomaly category prototype codebook is calculated. Based on the distance, the video anomaly score is obtained to achieve video anomaly detection.

Images