Video anomaly detection method and system based on time structure entropy

Abstract

This invention proposes a video anomaly detection method and system based on temporal structure entropy. The method includes: acquiring the original video stream; segmenting the original video stream into video segments; generating temporal perturbation samples from the video segments; extracting features and calculating the mean using a feature extraction module to obtain a video feature sequence; processing the video feature sequence using a temporal structure entropy calculation module to obtain a temporal structure entropy sequence; processing semantic features using an attention modulation module to obtain structure entropy-modulated attention output features; and processing the global temporal volatility prior descriptor and the structure entropy-modulated attention output features using an anomaly scoring module to obtain an anomaly score. This invention employs a lightweight design, featuring a parameter-free temporal structure entropy calculation module that calculates the temporal autocorrelation matrix through instantaneous feature centering within a sliding window, eliminating the need for trainable parameters and significantly reducing computational overhead.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a video anomaly detection method and system based on temporal structure entropy. Background Technology

[0002] Current video anomaly detection technologies are mainly divided into three categories:

[0003] 1. Reconstruction-based methods (such as Autoencoder, GAN): Anomalies are detected by reconstructing errors, assuming that abnormal events are difficult to reconstruct from normal patterns.

[0004] Typical examples include: ① Conv-AE (2016), which uses a convolutional autoencoder to learn normal patterns; and ② FutureFrame Prediction (2018), which detects anomalies by predicting the next frame. Their advantage lies in the fact that they do not require training with anomaly samples.

[0005] 2. Classification-based methods (such as 3D CNN, Two-Stream): directly model the normal / abnormal classification boundary.

[0006] Typical methods include: ① C3D (2015), which uses 3D convolution to extract spatiotemporal features; ② TSN (2016), which uses segmented sampling and a two-stream network. Their advantage lies in their high detection rate for overt anomalies (such as violent behavior).

[0007] 3. Weakly supervised methods (such as MIL, attention mechanism): Training is performed using only video-level labels.

[0008] Mainstream technologies include: ①RTFM (2021): based on feature magnitude ranking; ②MGFN (2022): multi-granularity feature fusion. Their advantage lies in reducing annotation costs.

[0009] The above-mentioned video anomaly detection technology has the following drawbacks:

[0010] 1. Reconstruction-based methods: The core drawback is that they are not sensitive to temporal anomalies and have difficulty detecting dynamic pattern mutations.

[0011] 2. Classification-based methods: The core drawback is that they rely heavily on frame-level labeled data, which makes them too costly to apply in practice.

[0012] 3. Weakly supervised methods: The core drawback is that they rely too much on feature amplitudes and are difficult to detect low-energy anomalies.

[0013] 4. Common problems: The fundamental bottleneck is that existing methods are all based on the reasoning logic of "semantics → anomaly", while actual anomalies may be visually similar to normal events (such as stealth theft) or only manifest as subtle breaks in temporal dynamics (such as sudden changes in mechanical vibration frequency).

[0014] Current technologies are limited by semantic ambiguity, strong labeling dependence, and low computational efficiency, and there is an urgent need for a new method that can directly quantify the stability of time structures. Summary of the Invention

[0015] In view of the above, the main objective of this invention is to propose a video anomaly detection method and system based on temporal structure entropy to solve the aforementioned technical problems.

[0016] This invention proposes a video anomaly detection method based on temporal structure entropy, the method comprising the following steps:

[0017] Step 1: Construct a feature extraction module based on a pre-trained visual encoder, a temporal structure entropy calculation module based on a sliding window algorithm, an attention modulation module based on a fully connected layer and an asymmetric attention mechanism, and an anomaly scoring module based on a multilayer perceptron and a linear transformation layer. The feature extraction module, temporal structure entropy calculation module, attention modulation module, and anomaly scoring module together form a video anomaly detection model.

[0018] Step 2: Obtain the original video stream, divide the original video stream into video segments, generate time perturbation samples from the video segments, and use the feature extraction module to extract features from the time perturbation samples generated for each video segment and calculate the average value to obtain the video feature sequence.

[0019] Step 3: Process the video feature sequence using the temporal structure entropy calculation module to obtain the temporal structure entropy sequence;

[0020] Step 4: Map the video feature sequence to a low-dimensional semantic space using a fully connected layer to obtain semantic features; use the attention modulation module to perform asymmetric attention calculation on the semantic features to obtain the structure entropy modulated attention output features.

[0021] Step 5: Extract the global temporal volatility prior descriptor based on the temporal structure entropy sequence, and use the anomaly scoring module to process the global temporal volatility prior descriptor and the structure entropy modulated attention output features to obtain anomaly scores.

[0022] Step 6: Construct a multivariate example learning loss function based on the anomaly score, and construct a global structure contrast loss function based on the global structure descriptor. Optimize the video anomaly detection model using the multivariate example learning loss function and the global structure contrast loss function to obtain the optimized video anomaly detection model. Input the original video stream into the optimized video anomaly detection model for processing to obtain the final anomaly score.

[0023] This invention also proposes a video anomaly detection system based on temporal structure entropy, wherein the system applies the video anomaly detection method based on temporal structure entropy as described above, and the system includes:

[0024] Build modules are used for:

[0025] A feature extraction module is built based on a pre-trained visual encoder, a temporal structure entropy calculation module is built based on a sliding window algorithm, an attention modulation module is built based on a fully connected layer and an asymmetric attention mechanism, and an anomaly scoring module is built based on a multilayer perceptron and a linear transformation layer. The video anomaly detection model is composed of the feature extraction module, the temporal structure entropy calculation module, the attention modulation module, and the anomaly scoring module.

[0026] Extraction module, used for:

[0027] The original video stream is acquired, segmented into video segments, and time perturbation samples are generated from the video segments. The feature extraction module is used to extract features from the time perturbation samples generated for each video segment and calculate the average value to obtain the video feature sequence.

[0028] The time structure entropy module is used for:

[0029] The video feature sequence is processed using the temporal structure entropy calculation module to obtain the temporal structure entropy sequence;

[0030] Modulation attention module, used for:

[0031] A fully connected layer is used to map the video feature sequence to a low-dimensional semantic space to obtain semantic features; an attention modulation module is used to perform asymmetric attention calculation on the semantic features to obtain structure entropy modulated attention output features.

[0032] The rating module is used for:

[0033] Global temporal volatility prior descriptors are extracted based on temporal structure entropy sequences. An anomaly scoring module is used to process the global temporal volatility prior descriptors and the structure entropy modulated attention output features to obtain anomaly scores.

[0034] A multivariate example learning loss function is constructed based on the anomaly score. The global temporal volatility prior descriptor is used as the global structure descriptor. A global structure contrast loss function is constructed based on the global structure descriptor. The video anomaly detection model is optimized using the multivariate example learning loss function and the global structure contrast loss function to obtain the optimized video anomaly detection model. The original video stream is input into the optimized video anomaly detection model for processing to obtain the final anomaly score.

[0035] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0036] 1. Superior detection performance: On mainstream public datasets such as UCF-Crime (complex scene anomaly detection), XD-Violence (violent behavior detection), and ShanghaiTech (multi-scene anomaly), the frame-level AUC (area under the curve) and AP (average precision) of this method reach or exceed the current state-of-the-art (SOTA) level. By quantifying local dynamic instability through temporal structure entropy and combining it with structure entropy-modulated attention, it can accurately locate abnormal segments, reduce false positives and false negatives, and is suitable for scenarios such as security monitoring and intelligent transportation. It significantly improves the accuracy of automated detection of abnormal events and reduces the cost of manual review.

[0037] 2. High computational efficiency: It adopts a lightweight design and a parameter-free time structure entropy calculation module. It calculates the time autocorrelation matrix by instantaneous feature centering within a sliding window. No trainable parameters are required, which significantly reduces the computational overhead. It is suitable for deployment in embedded devices and can be widely used in resource-constrained scenarios to achieve high frame rate and low latency real-time anomaly detection.

[0038] 3. Strong weak supervision capability: Only video-level labels are required: Through multi-instance learning loss and global structure contrast loss, it can be trained with only video-level abnormal / normal labels, without the need for expensive frame-level annotation; it automatically focuses on potential abnormal segments by utilizing temporal structure entropy, and generates video-level predictions by aggregating the maximum values, effectively solving the label sparsity problem and significantly reducing data annotation costs. It is especially suitable for long video surveillance scenarios and can quickly adapt to the anomaly detection needs of new scenarios.

[0039] 4. Good robustness: Prioritizes structural stability, quantifies temporal dynamic structural instability through temporal structural entropy rather than relying on semantic content or feature amplitude, avoiding misjudgments caused by changes in illumination, background noise, etc.; global-local collaborative calibration, combined with global volatility priors, performs macroscopic calibration on local predictions, suppresses local noise interference, and improves cross-scene generalization ability.

[0040] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0041] Figure 1 This is a flowchart of the steps of a video anomaly detection method based on temporal structure entropy proposed in this invention;

[0042] Figure 2 This is a general framework diagram of a video anomaly detection method based on temporal structure entropy proposed in this invention;

[0043] Figure 3 This is a visualization of the anomaly detection results of a video anomaly detection method based on temporal structure entropy proposed in this invention;

[0044] Figure 4 This is a system architecture diagram of a video anomaly detection system based on temporal structure entropy proposed in this invention. Detailed Implementation

[0045] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0046] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0047] Please see Figure 1 This embodiment provides a video anomaly detection method based on temporal structure entropy, the method comprising the following steps:

[0048] Step 1: Construct a feature extraction module based on a pre-trained visual encoder, a temporal structure entropy calculation module based on a sliding window algorithm, an attention modulation module based on a fully connected layer and an asymmetric attention mechanism, and an anomaly scoring module based on a multilayer perceptron and a linear transformation layer. The feature extraction module, temporal structure entropy calculation module, attention modulation module, and anomaly scoring module together form a video anomaly detection model.

[0049] Step 2: Obtain the original video stream, divide the original video stream into video segments, generate time perturbation samples from the video segments, and use the feature extraction module to extract features from the time perturbation samples generated for each video segment and calculate the average value to obtain the video feature sequence.

[0050] Please see Figure 2 In step 2, the original video stream is acquired, segmented into video segments, and time perturbation samples are generated from these video segments. The feature extraction module is then used to extract features from the time perturbation samples generated for each video segment and calculate the average value to obtain the video feature sequence. This process includes the following sub-steps:

[0051] The original video stream is acquired and divided into equal-length segments to obtain a video segment sequence.

[0052] Based on the video segment sequence, three temporal perturbation samples are generated for each video segment;

[0053] The pre-trained visual encoder is used to extract features from three temporally perturbed samples generated for each video segment and the average is calculated to obtain the feature representation of each video segment. The following relationship exists in the corresponding process:

[0054] ;

[0055] in, Indicates the first Feature representation of a video segment Indicates the index of a video segment. Indicates the first The first video clip Features of time-perturbation samples;

[0056] The feature representations of each video segment are arranged in chronological order to obtain the video feature sequence. ;

[0057] in, Represents a video feature sequence. This indicates the number of video clips.

[0058] It should be noted that each segment contains 16 frames, with a time span of 1 second; if the total number of frames in the video is not a multiple of 16, the insufficient portion at the end is filled with black frames; three time perturbation samples are generated for each segment, specifically: Sample 1 is the original 16 frames (frames 1-16), Sample 2 is a right-off offset of 1 frame (frames 2-17), and Sample 3 is a left-off offset of 1 frame (frames 0-15, with insufficient portions filled with black frames); Figure 2 In this context, TSE represents temporal structure entropy, and TVP represents global volatility prior.

[0059] Step 3: Use the temporal structure entropy calculation module to process the video feature sequence to obtain the temporal structure entropy sequence.

[0060] In step 3, the video feature sequence is processed using the temporal structure entropy calculation module to obtain the temporal structure entropy sequence, specifically including the following sub-steps:

[0061] The sliding window algorithm is used to segment the video feature sequence, and the feature representation of each video segment within the window is simultaneously centered to obtain a centered feature vector. The following relationship exists in the corresponding process:

[0062] ;

[0063] in, Indicates the first Centralized feature vectors of video segments This indicates that the mean has been calculated along the feature dimension;

[0064] Stack all the centered feature vectors within the window to obtain the centered feature matrix within the window. The following relationship exists in the corresponding process:

[0065] ;

[0066] in, This represents the centered feature matrix within the window. Indicates the size of the sliding window, and ;

[0067] The time autocorrelation matrix of the centered feature matrix within the window is calculated to obtain the time autocorrelation matrix within the window. The following relationship exists in the corresponding process:

[0068] ;

[0069] in, This represents the time autocorrelation matrix within the window. Indicates transpose;

[0070] Eigenvalue decomposition is performed on the time autocorrelation matrix within the window to obtain the eigenvalue spectrum;

[0071] Based on the eigenvalue spectrum, the eigenvalues ​​are normalized to a probability distribution, and the entropy value is calculated to obtain the structural entropy value. The following relationship exists in the corresponding process:

[0072] ;

[0073] in, This represents the normalized probability distribution of eigenvalues. Representation matrix The 1 eigenvalue, Representation matrix The 1 eigenvalue, Indicates the first Temporal structure entropy value of a video segment Representation matrix The number of non-zero eigenvalues, Indicates taking the logarithm;

[0074] The structural entropy values ​​of each video segment are concatenated to obtain a temporal structural entropy sequence. ;

[0075] in, It represents a temporal structure entropy sequence.

[0076] Step 4: Map the video feature sequence to a low-dimensional semantic space using a fully connected layer to obtain semantic features; use the attention modulation module to perform asymmetric attention calculation on the semantic features to obtain the structure entropy modulated attention output features.

[0077] In step 4, a fully connected layer is used to map the video feature sequence to a low-dimensional semantic space to obtain semantic features; an attention modulation module is used to perform asymmetric attention calculation on the semantic features to obtain the structure entropy modulated attention output features, specifically including the following sub-steps:

[0078] Fully connected layers are used to map video feature sequences to a low-dimensional semantic space to obtain semantic features. The following relationship exists in the mapping process:

[0079] ;

[0080] in, Represents semantic features, This indicates that the process has been handled by a fully connected layer.

[0081] Projecting semantic features yields a query matrix and a key matrix generated solely from the semantic feature stream, respectively. The following relationship exists in this process:

[0082] ;

[0083] in, This represents a query matrix generated solely based on the semantic feature stream. This represents the trainable projection matrix used to generate the query matrix. This represents a key matrix generated solely based on the semantic feature stream. This represents the trainable projection matrix used to generate the key matrix;

[0084] The Sigmoid function is used to process the time-structured entropy sequence to obtain dynamic weights. The following relationship exists in the corresponding process:

[0085] ;

[0086] in, Indicates dynamic weights, This indicates that the signal has been processed by the Sigmoid function. Represents the arithmetic mean of the temporal structure entropy sequence;

[0087] The dynamic weights are multiplied element-wise with the semantic features and then projected to obtain the value matrix generated by the structural value stream. The following relationship exists in the corresponding process:

[0088] ;

[0089] in, This represents the value matrix generated by the structural value stream. This indicates element-wise multiplication. Represents the trainable value projection matrix;

[0090] Attention is calculated based on the query matrix, key matrix, and value matrix to obtain the structure entropy modulated attention output features. The following relationship exists in the corresponding process:

[0091] ;

[0092] in, This represents the structural entropy-modulated attention output feature. This represents the attention mechanism. This indicates that the data has been processed using the softmax function. This represents the dimension of the key vector.

[0093] Step 5: Extract the global temporal volatility prior descriptor based on the temporal structure entropy sequence, and use the anomaly scoring module to process the global temporal volatility prior descriptor and the structure entropy modulated attention output features to obtain anomaly scores.

[0094] In step 5, a global temporal volatility prior descriptor is extracted based on the temporal structure entropy sequence. The anomaly scoring module is then used to process the global temporal volatility prior descriptor and the structure entropy modulated attention output features to obtain anomaly scores. This process includes the following sub-steps:

[0095] Based on the extraction of global temporal volatility prior descriptors from temporal structure entropy sequences, the following relationship exists in the corresponding process:

[0096] ;

[0097] in, This represents a priori descriptor for global time volatility. The standard deviation of the time-structured entropy sequence. This indicates taking the minimum value. This indicates taking the maximum value;

[0098] A multilayer perceptron is used to map the global temporal volatility prior descriptor to the feature space, and then multiply it element-wise with the structural entropy-modulated attention output features to obtain the attention features modulated by the global temporal volatility prior. The following relationship exists in the corresponding process:

[0099] ;

[0100] in, This represents the attention features modulated by a global time-varying prior. Indicates the scaling factor. This indicates that the data has been processed by a multilayer perceptron.

[0101] After subtracting the semantic features from the structural entropy-modulated attention output features and performing layer normalization, the residual features are obtained. The following relationship exists in the corresponding process:

[0102] ;

[0103] in, Representing residual characteristics, This indicates that the process has undergone layer normalization.

[0104] The attention features modulated by the global time fluctuation prior are added to the residual features to obtain the final fused features. The following relationship exists in the corresponding process:

[0105] ;

[0106] in, Indicates the final characteristics after fusion;

[0107] The fused final features are sequentially processed through a linear transformation layer and a Sigmoid function to obtain anomaly scores. The following relationship exists in the corresponding process:

[0108] ;

[0109] in, Indicates abnormal scores. This indicates that the process has been processed through a linear transformation layer.

[0110] Step 6: Construct a multivariate example learning loss function based on the anomaly score, use the global temporal volatility prior descriptor as the global structure descriptor, construct a global structure contrast loss function based on the global structure descriptor, optimize the video anomaly detection model using the multivariate example learning loss function and the global structure contrast loss function, and obtain the optimized video anomaly detection model. Input the original video stream into the optimized video anomaly detection model for processing to obtain the final anomaly score.

[0111] In step 6, a multivariate example learning loss function is constructed based on the anomaly score. The global temporal volatility prior descriptor is used as the global structure descriptor, and a global structure contrast loss function is constructed based on the global structure descriptor. The video anomaly detection model is optimized using the multivariate example learning loss function and the global structure contrast loss function to obtain the optimized video anomaly detection model. The original video stream is input into the optimized video anomaly detection model for processing to obtain the final anomaly score. The multivariate example learning loss function is constructed based on the anomaly score, and the following relationship exists in the corresponding process:

[0112] ;

[0113] in, Indicates the learning loss of multiple examples. Indicates the number of training samples. Indicates the first The tags for each video. Indicates the first The highest score of a segment in the video;

[0114] Among them, the global structure contrastive loss function is constructed based on the global structure descriptor, and the following relationship exists in the corresponding process:

[0115] ;

[0116] in, Indicates the centroid of a normal sample. Represents the normal sample set. This represents the global structure comparison loss. This indicates that the code has been processed using L2 regularization. Represents a set of abnormal samples. Indicates the boundary hyperparameters;

[0117] It should be noted that the global structure descriptor in the global structure contrast loss function directly reuses the global temporal volatility prior descriptor output in step 4, and its statistics characterize the overall dynamic stability of the video.

[0118] The video anomaly detection model is optimized by using a multivariate example learning loss function and a global structure comparison loss function. The original video stream is then input into the optimized video anomaly detection model for processing to obtain the final anomaly score.

[0119] Furthermore, the multivariate example learning loss and the global structure contrast loss are weighted and fused to obtain the total loss. The corresponding relationship in this process is as follows:

[0120] ;

[0121] in, Indicates the total loss. This represents all learnable parameters in the model.

[0122] To verify the effectiveness of this invention, comprehensive testing was conducted on three major datasets: ShanghaiTech, UCF-Crime, and XD-Violence. ShanghaiTech contains 437 videos across 13 scenarios, covering general-level anomalous events; UCF-Crime contains 1900 long videos, covering complex anomaly types; and XD-Violence focuses on violent behavior detection, ensuring the reliability of the model's generalization ability assessment.

[0123] 1. Comparison with existing technologies (SOTA)

[0124] Under weak supervision, frame-level AUC (Area Under the ROC Curve) was used as the main evaluation metric, and the results were compared with current mainstream methods. The experimental results are shown in Table 1.

[0125] Table 1. Performance comparison with the state-of-the-art (SOTA) method (AUC / %)

[0126]

[0127] Key conclusions:

[0128] Under the condition of using only RGB mode, the TIENet proposed in this invention achieves AUC of 98.03% and 88.34% on ShanghaiTech and UCF-Crime, respectively, which is significantly better than VadCLIP (88.02%) which depends on multimodality.

[0129] On the XD-Violence dataset, the TIENet proposed in this invention outperforms existing methods in both AP (87.23%) and APA (87.95%), validating its ability to detect complex violent scenes.

[0130] Lightweight Comparison:

[0131] As shown in Table 2, TIENet has significantly fewer parameters (0.46M) and a smaller model size (1.76MB) than other methods, while maintaining the highest performance, demonstrating the design advantages of "high efficiency and lightweight".

[0132] Table 2 Comparison of Model Efficiency

[0133]

[0134] 2. Ablation test

[0135] To verify the contributions of temporal structure entropy (TSE), structure entropy modulated attention (SEMA), and global volatility prior (TVP), modules were progressively stacked for testing (UCF-Crime dataset):

[0136] Table 3 Ablation test results (AUC / %)

[0137]

[0138] Key conclusions:

[0139] TSE module: When introduced independently, AUC increased by 0.73%, proving the ability of time structure entropy to quantify dynamic instability.

[0140] SEMA mechanism: By modulating attention through structural entropy, performance was further improved by 0.93%, verifying the effectiveness of "structure-guided semantics".

[0141] TVP and contrast loss: Global calibration brought the AUC to 88.34%, demonstrating the necessity of multi-level information integration.

[0142] 3. Hyperparameter sensitivity analysis

[0143] Investigating the impact of sliding window size W on the TSE module (UCF-Crime dataset):

[0144] Table 4. Effect of window size W (AUC / %)

[0145]

[0146] Key conclusions:

[0147] The optimal window size is W=5, which balances local dynamic capture and computational efficiency.

[0148] The performance fluctuation is less than 0.2% when W∈[3,7], demonstrating the robustness of the method to hyperparameters.

[0149] 4. Visualization of abnormal scenarios

[0150] Please see Figure 3 Through keyframe visualization and TSE curve visualization, TIENet performs exceptionally well in the following scenarios:

[0151] Traffic accidents: The TSE value suddenly increases at the moment of collision (the peak value is 300% higher than the baseline), while traditional methods miss the detection due to background noise.

[0152] Stealth behavior: The amplitude of semantic features changes very little, but TSE accurately marks abnormal periods through "dynamic instability".

[0153] Crowd gathering: The TSE remains high, reflecting the formation of a new structure, which is clearly distinguishable from short-term energy fluctuations.

[0154] Please see Figure 4 This embodiment also provides a video anomaly detection system based on temporal structure entropy, wherein the system applies the video anomaly detection method based on temporal structure entropy as described above, and the system includes:

[0155] Build modules are used for:

[0156] A feature extraction module is built based on a pre-trained visual encoder, a temporal structure entropy calculation module is built based on a sliding window algorithm, an attention modulation module is built based on a fully connected layer and an asymmetric attention mechanism, and an anomaly scoring module is built based on a multilayer perceptron and a linear transformation layer. The video anomaly detection model is composed of the feature extraction module, the temporal structure entropy calculation module, the attention modulation module, and the anomaly scoring module.

[0157] Extraction module, used for:

[0158] The original video stream is acquired, segmented into video segments, and time perturbation samples are generated from the video segments. The feature extraction module is used to extract features from the time perturbation samples generated for each video segment and calculate the average value to obtain the video feature sequence.

[0159] The time structure entropy module is used for:

[0160] The video feature sequence is processed using the temporal structure entropy calculation module to obtain the temporal structure entropy sequence;

[0161] Modulation attention module, used for:

[0162] A fully connected layer is used to map the video feature sequence to a low-dimensional semantic space to obtain semantic features; an attention modulation module is used to perform asymmetric attention calculation on the semantic features to obtain structure entropy modulated attention output features.

[0163] The rating module is used for:

[0164] Global temporal volatility prior descriptors are extracted based on temporal structure entropy sequences. An anomaly scoring module is used to process the global temporal volatility prior descriptors and the structure entropy modulated attention output features to obtain anomaly scores.

[0165] A multivariate example learning loss function is constructed based on the anomaly score. The global temporal volatility prior descriptor is used as the global structure descriptor. A global structure contrast loss function is constructed based on the global structure descriptor. The video anomaly detection model is optimized using the multivariate example learning loss function and the global structure contrast loss function to obtain the optimized video anomaly detection model. The original video stream is input into the optimized video anomaly detection model for processing to obtain the final anomaly score.

[0166] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0167] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0168] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0169] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A video anomaly detection method based on temporal structure entropy, characterized in that, The method includes the following steps: Step 1: Construct a feature extraction module based on a pre-trained visual encoder, a temporal structure entropy calculation module based on a sliding window algorithm, an attention modulation module based on a fully connected layer and an asymmetric attention mechanism, and an anomaly scoring module based on a multilayer perceptron and a linear transformation layer. The feature extraction module, temporal structure entropy calculation module, attention modulation module, and anomaly scoring module together form a video anomaly detection model. Step 2: Obtain the original video stream, divide the original video stream into video segments, generate time perturbation samples from the video segments, and use the feature extraction module to extract features from the time perturbation samples generated for each video segment and calculate the average value to obtain the video feature sequence. Step 3: Process the video feature sequence using the temporal structure entropy calculation module to obtain the temporal structure entropy sequence; Step 4: Use a fully connected layer to map the video feature sequence to a low-dimensional semantic space to obtain semantic features; The attention modulation module is used to perform asymmetric attention computation on semantic features to obtain structural entropy modulated attention output features; Step 5: Extract the global temporal volatility prior descriptor based on the temporal structure entropy sequence, and use the anomaly scoring module to process the global temporal volatility prior descriptor and the structure entropy modulated attention output features to obtain anomaly scores. Step 6: Construct a multivariate example learning loss function based on the anomaly score, use the global temporal volatility prior descriptor as the global structure descriptor, construct a global structure contrast loss function based on the global structure descriptor, optimize the video anomaly detection model using the multivariate example learning loss function and the global structure contrast loss function, and obtain the optimized video anomaly detection model. Input the original video stream into the optimized video anomaly detection model for processing to obtain the final anomaly score. In step 3, the video feature sequence is processed using the temporal structure entropy calculation module to obtain the temporal structure entropy sequence, specifically including the following sub-steps: The sliding window algorithm is used to segment the video feature sequence, and the feature representation of each video segment within the window is subjected to real-time feature centering to obtain a centered feature vector. Stack all the centered feature vectors within the window to obtain the centered feature matrix within the window; The time autocorrelation matrix of the centered feature matrix within the window is calculated to obtain the time autocorrelation matrix within the window. Eigenvalue decomposition is performed on the time autocorrelation matrix within the window to obtain the eigenvalue spectrum; Based on the eigenvalue spectrum, the eigenvalues ​​are normalized to a probability distribution, and the entropy value is calculated to obtain the structural entropy value; The structural entropy values ​​of each video segment are concatenated to obtain a temporal structural entropy sequence. ; in, Represents a time-structured entropy sequence. Indicates the first Temporal structure entropy value of a video segment Indicates the number of video clips. Indicates the index of a video segment; In step 4, a fully connected layer is used to map the video feature sequence to a low-dimensional semantic space to obtain semantic features; an attention modulation module is used to perform asymmetric attention calculation on the semantic features to obtain structure entropy modulated attention output features. Specifically, this includes the following sub-steps: Fully connected layers are used to map video feature sequences to a low-dimensional semantic space to obtain semantic features; Project the semantic features to obtain the query matrix and the key matrix generated solely based on the semantic feature flow, respectively. The Sigmoid function is used to process the temporal entropy sequence to obtain dynamic weights; The dynamic weights are multiplied element-wise with the semantic features and then projected to obtain the value matrix generated by the structural value stream. Attention is calculated based on the query matrix, key matrix, and value matrix to obtain the structural entropy modulated attention output features.

2. The video anomaly detection method based on temporal structure entropy according to claim 1, characterized in that, In step 2, the original video stream is acquired, segmented into video segments, and time-perturbation samples are generated from the video segments. The feature extraction module is then used to extract features from the time-perturbation samples generated for each video segment and calculate the average value to obtain the video feature sequence. Specifically, this includes the following sub-steps: The original video stream is acquired and divided into equal-length segments to obtain a video segment sequence. Based on the video segment sequence, three temporal perturbation samples are generated for each video segment; The pre-trained visual encoder is used to extract features from three temporally perturbed samples generated for each video segment and the average is calculated to obtain the feature representation of each video segment. The following relationship exists in the corresponding process: ； in, Indicates the first Feature representation of a video segment Indicates the first The first video clip Features of time-perturbation samples; The feature representations of each video segment are arranged in chronological order to obtain the video feature sequence. ; in, This represents a video feature sequence.

3. The video anomaly detection method based on temporal structure entropy according to claim 2, characterized in that, In the steps of segmenting the video feature sequence using the sliding window algorithm and performing real-time feature centering on the feature representation of each video segment within the window to obtain a centered feature vector, the following relationship exists: ； in, Indicates the first Centralized feature vectors of video segments This indicates that the mean has been calculated along the feature dimension; In the step of stacking all the centered feature vectors within the window to obtain the centered feature matrix within the window, the following relationship exists: ； in, This represents the centered feature matrix within the window. Indicates the size of the sliding window; In the step of calculating the time autocorrelation matrix of the centered feature matrix within the window to obtain the time autocorrelation matrix within the window, the following relationship exists: ； in, This represents the time autocorrelation matrix within the window. Indicates transpose; In the steps of normalizing eigenvalues ​​to a probability distribution based on the eigenvalue spectrum and calculating the entropy value to obtain the structural entropy value, the following relationship exists: ； in, This represents the normalized probability distribution of eigenvalues. Representation matrix The 1 eigenvalue, Representation matrix The 1 eigenvalue, Representation matrix The number of non-zero eigenvalues, This indicates taking the logarithm.

4. The video anomaly detection method based on temporal structure entropy according to claim 3, characterized in that, In the step of mapping video feature sequences to a low-dimensional semantic space using fully connected layers to obtain semantic features, the following relationship exists: ； in, Represents semantic features, This indicates that the process has been handled by a fully connected layer. In the step of projecting semantic features to obtain the query matrix and the key matrix generated solely based on the semantic feature stream, the following relationship exists: ； in, This represents a query matrix generated solely based on the semantic feature stream. This represents the trainable projection matrix used to generate the query matrix. This represents a key matrix generated solely based on the semantic feature stream. This represents the trainable projection matrix used to generate the key matrix; In the process of using the Sigmoid function to process the temporal structure entropy sequence to obtain dynamic weights, the following relationship exists: ； in, Indicates dynamic weights, This indicates that the signal has been processed by the Sigmoid function. Represents the arithmetic mean of the temporal structure entropy sequence; In the step of element-wise multiplication of dynamic weights and semantic features, followed by projection, to obtain the value matrix generated from the structural value stream, the following relationship exists: ； in, This represents the value matrix generated by the structural value stream. This indicates element-wise multiplication. Represents the trainable value projection matrix; In the step of performing attention calculation based on the query matrix, key matrix, and value matrix to obtain the structural entropy modulated attention output features, the following relationship exists: ； in, This represents the structural entropy-modulated attention output feature. This represents the attention mechanism. This indicates that the data has been processed using the softmax function. This represents the dimension of the key vector.

5. The video anomaly detection method based on temporal structure entropy according to claim 4, characterized in that, In step 5, a global temporal volatility prior descriptor is extracted based on the temporal structure entropy sequence. The anomaly scoring module then processes the global temporal volatility prior descriptor and the structure entropy-modulated attention output features to obtain anomaly scores. Specifically, this includes the following sub-steps: Extracting global temporal volatility prior descriptors based on temporal structure entropy sequences; The global temporal volatility prior descriptor is mapped to the feature space using a multilayer perceptron, and then multiplied element-wise with the structural entropy modulated attention output feature to obtain the attention feature modulated by the global temporal volatility prior. After subtracting the semantic features from the structural entropy-modulated attention output features, layer normalization is performed to obtain the residual features. The attention features modulated by global temporal fluctuations prior are added to the residual features to obtain the final fused features; The fused final features are then processed sequentially through a linear transformation layer and a Sigmoid function to obtain anomaly scores.

6. The video anomaly detection method based on temporal structure entropy according to claim 5, characterized in that, In the step of extracting the global temporal volatility prior descriptor based on the temporal structure entropy sequence, the following relationship exists: ； in, This represents a priori descriptor for global time volatility. The standard deviation of the time-structured entropy sequence. This indicates taking the minimum value. This indicates taking the maximum value; In the step of mapping the global temporal volatility prior descriptor to the feature space using a multilayer perceptron and multiplying it element-wise with the structural entropy-modulated attention output features to obtain the attention features modulated by the global temporal volatility prior, the following relationship exists: ； in, This represents the attention features modulated by a global time-varying prior. Indicates the scaling factor. This indicates that the data has been processed by a multilayer perceptron. In the step of subtracting the semantic features from the structural entropy-modulated attention output features and then performing layer normalization to obtain the residual features, the following relationship exists: ； in, Representing residual characteristics, This indicates that the process has undergone layer normalization. In the step of adding the attention features modulated by the global time fluctuation prior to the residual features to obtain the fused final features, the following relationship exists: ； in, Indicates the final characteristics after fusion; In the step of processing the fused final features sequentially through a linear transformation layer and a Sigmoid function to obtain the anomaly score, the following relationship exists: ； in, Indicates abnormal scores. This indicates that the process has been processed through a linear transformation layer.

7. The video anomaly detection method based on temporal structure entropy according to claim 6, characterized in that, In step 6, a multivariate example learning loss function is constructed based on the anomaly score. The global temporal volatility prior descriptor is used as the global structure descriptor, and a global structure contrast loss function is constructed based on the global structure descriptor. The video anomaly detection model is optimized using the multivariate example learning loss function and the global structure contrast loss function to obtain the optimized video anomaly detection model. The original video stream is then input into the optimized video anomaly detection model for processing to obtain the final anomaly score. The multivariate example learning loss function is constructed based on the anomaly score, and the following relationship exists in the process: ； in, Indicates the learning loss of multiple examples. Indicates the number of training samples. Indicates the first The tags for each video. Indicates the first The highest score of the video segment; Among them, the global structure contrastive loss function is constructed based on the global structure descriptor, and the following relationship exists in the corresponding process: ； in, Indicates the centroid of a normal sample. Represents the normal sample set. This represents the global structure comparison loss. This indicates that the code has been processed using L2 regularization. Represents a set of abnormal samples. This represents the boundary hyperparameters.

8. A video anomaly detection system based on temporal structure entropy, characterized in that, The system employs the video anomaly detection method based on temporal structure entropy as described in any one of claims 1 to 7, and the system comprises: Build modules are used for: A feature extraction module is built based on a pre-trained visual encoder, a temporal structure entropy calculation module is built based on a sliding window algorithm, an attention modulation module is built based on a fully connected layer and an asymmetric attention mechanism, and an anomaly scoring module is built based on a multilayer perceptron and a linear transformation layer. The video anomaly detection model is composed of the feature extraction module, the temporal structure entropy calculation module, the attention modulation module, and the anomaly scoring module. Extraction module, used for: The original video stream is acquired, segmented into video segments, and time perturbation samples are generated from the video segments. The feature extraction module is used to extract features from the time perturbation samples generated for each video segment and calculate the average value to obtain the video feature sequence. The time structure entropy module is used for: The video feature sequence is processed using the temporal structure entropy calculation module to obtain the temporal structure entropy sequence; Modulation attention module, used for: A fully connected layer is used to map the video feature sequence to a low-dimensional semantic space to obtain semantic features; an attention modulation module is used to perform asymmetric attention calculation on the semantic features to obtain structure entropy modulated attention output features. The rating module is used for: Global temporal volatility prior descriptors are extracted based on temporal structure entropy sequences. An anomaly scoring module is used to process the global temporal volatility prior descriptors and the structure entropy modulated attention output features to obtain anomaly scores. A multivariate example learning loss function is constructed based on the anomaly score. The global temporal volatility prior descriptor is used as the global structure descriptor. A global structure contrast loss function is constructed based on the global structure descriptor. The video anomaly detection model is optimized using the multivariate example learning loss function and the global structure contrast loss function to obtain the optimized video anomaly detection model. The original video stream is input into the optimized video anomaly detection model for processing to obtain the final anomaly score.

Images