Weakly supervised video anomaly detection method based on bag constraint expectation maximization learning

Abstract

The application belongs to the technical field of computer vision, and discloses a weakly supervised video anomaly detection method based on bag constraint expectation maximization learning; a training video is divided into segments and features are extracted, and an initial anomaly probability is obtained through a segment classifier; a video-level bag probability is aggregated, a marginal bag likelihood is constructed in combination with a video-level label, and an initial posterior probability of each segment is estimated through Bayesian posterior inference; the features are projected to an affinity feature space, a feature affinity matrix between segments is calculated, and a refined posterior probability is obtained through weighted propagation and fusion of the initial posterior probability; the refined posterior probability is subjected to gradient truncation, an expectation maximization distillation loss is constructed, and the expectation maximization distillation loss is combined with a bag likelihood loss to form a joint loss function; discriminative expectation maximization optimization training is carried out based on the joint loss function, and an anomaly detection result is output; the method makes the training target directly correspond to a weakly supervised label original structure, does not require manual hyperparameters, and is efficient, stable and good in interpretability.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to video anomaly detection technology, particularly to a weakly supervised video anomaly detection method based on bag-constrained expectation-maximization learning. Background Technology

[0002] Video anomaly detection is an important research direction in computer vision, with broad application prospects in areas such as intelligent surveillance, public safety, urban management, and industrial inspection. Weakly Supervised Video Anomaly Detection (WSVAD) refers to a video anomaly detection task that trains a model using only video-level labels (i.e., knowing only whether a video segment contains an anomalous event, without needing to label the specific time interval of the anomaly). Because the cost of obtaining weakly supervised labels is far lower than that of frame-by-frame or segment-by-segment precise labeling, this technology has received widespread attention in recent years.

[0003] Currently, Multiple Instance Learning (MIL) is the most mainstream training paradigm in weakly supervised video anomaly detection. In this framework, a video is regarded as a "bag", each segment in the video is regarded as an "instance", and the video-level label is the bag-level label. Around this framework, existing methods have mainly developed three technical routes: (1) the MIL method based on Top-k selection, which selects the k highest-scoring segments from the abnormal video as positive instances and selects segments from the normal video as negative instances, and uses ranking loss or binary classification loss for training, simplifying anomaly localization into the problem of selecting a few high-scoring segments; (2) the method based on pseudo-label self-training, which first uses the initial model to generate segment-level pseudo-labels, and then performs multi-stage iterative self-training to gradually refine the pseudo-labels; (3) the method based on anomaly proportion prior, which assumes that the time proportion of abnormal events in the video follows a certain prior distribution (such as the Beta distribution), and constructs a constrained optimization objective by introducing proportion prior, and adopts a multi-stage optimization strategy to gradually adjust the anomaly proportion estimate and update the model parameters.

[0004] However, the existing methods still have the following drawbacks: (1) The training objective does not match the weak supervision annotation structure, resulting in distortion of the supervision signal. Existing methods all indirectly convert video-level labels into segment-level supervision signals through top-k selection, pseudo-label generation, or proportional prior, rather than directly modeling the original structure of the weak supervision annotation (i.e., only knowing whether the entire video contains anomalies, but not the specific location). Therefore, there is a deviation between the training objective and the original weak supervision constraints. This deviation leads to distortion of the supervision signal. The model does not learn the real weak supervision learning objective, but an indirectly transformed and distorted objective. This is the fundamental reason for the limited performance of existing methods.

[0005] (2) Sensitive to hyperparameters and poor generalization ability. Hyperparameters such as the k-value and outlier ratio of top-k directly determine which segments are selected as positive or negative examples, thus directly affecting the training direction of the model. Therefore, these hyperparameters have a huge impact on the final performance. However, the optimal k-value and outlier ratio vary significantly in different datasets or different scenarios, requiring tedious parameter tuning for each scenario. This results in poor generalization ability of the method and makes it difficult to adapt to diverse real-world application scenarios.

[0006] (3) The training process is complex, inefficient, and unstable. Methods such as multi-stage self-training and multi-stage ratio optimization require multiple rounds of iterative training. Each round requires regenerating pseudo-labels or re-estimating the anomaly ratio and retraining the model, resulting in a complex training process, high computational cost, and long training cycle. More seriously, since the pseudo-labels generated in the initial stage have large errors, self-training based on inaccurate pseudo-labels will further accumulate errors, leading to model drift and unstable training process.

[0007] (4) Insufficient interpretability. The anomaly scoring process of existing methods is often dominated by the black-box prediction of deep neural networks, lacking clear probabilistic interpretation and semantic basis. Why are these k segments selected in the top-k selection? What criteria are used to generate pseudo-labels? What is the rationale for the anomaly ratio prior? These questions lack interpretable theoretical support and are difficult to meet the needs of high-reliability application scenarios.

[0008] Therefore, how to design a weakly supervised video anomaly detection method that allows the training objective to directly correspond to the original structure of the weakly supervised annotation, does not rely on manual hyperparameters, is highly efficient and stable in training, and has good interpretability is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0009] This invention aims to address the problems of existing methods that rely on indirect methods such as top-k selection, pseudo-label generation, or anomalous proportion priors to transform video-level labels, resulting in a mismatch between the training objective and the original structure of weakly supervised annotations, distortion of the supervision signal, and consequently hyperparameter sensitivity, inefficient and unstable training, and insufficient interpretability. The invention provides a weakly supervised video anomaly detection method based on bag-constrained expectation maximization learning.

[0010] In a first aspect, the present invention provides a weakly supervised video anomaly detection method based on packet-constrained expectation-maximization learning, comprising the following steps: Acquire training video data with video-level labels, divide the training video data into segments and extract feature representations of each segment, and perform binary classification prediction on the feature representations through a segment classifier to obtain the initial anomaly probability of each segment. The initial anomaly probabilities are aggregated to obtain the video-level packet probabilities. The packet probabilities and the video-level labels are combined to construct the marginal packet likelihood. Based on the marginal packet likelihood, the initial posterior probability of each segment under the video-level label conditions is estimated by Bayesian posterior inference. The feature representation is projected onto the affinity feature space to obtain the affinity feature vector. Based on the affinity feature vector, the feature affinity matrix between segments is calculated. The initial posterior probability is weighted and propagated using the feature affinity matrix to obtain the propagated posterior probability. The propagated posterior probability is then weighted and fused with the initial posterior probability to obtain the refined posterior probability. Gradient truncation is performed on the refined posterior probability and combined with the initial anomaly probability to construct the expected maximum distillation loss. The expected maximum distillation loss is then weighted and combined with the bag likelihood loss corresponding to the marginal bag likelihood to obtain the joint loss function. Discriminative expectation-maximization optimization training is performed based on the joint loss function, and anomaly detection results for each segment are output based on the video to be detected.

[0011] As an optional implementation of the first aspect of this application, the extraction of feature representations of each fragment includes: The training video data is evenly divided into multiple video segments according to a preset time interval; The video feature extraction network is used to extract features from each video segment to obtain the initial feature vector of each segment; The initial feature vector is sequentially subjected to linear projection, normalization, nonlinear activation, and random deactivation by a feature encoder to obtain the feature representation; the feature encoder includes a linear projection layer, a normalization layer, an activation function layer, and a random deactivation layer connected in sequence.

[0012] As an optional implementation of the first aspect of this application, obtaining the initial anomaly probability of each segment includes: The fragment classifier includes a linear classification layer and a sigmoid activation function; the linear classification layer maps the feature representation to a one-dimensional classification logic value, and the sigmoid activation function maps the one-dimensional classification logic value to a probability value between zero and one, which serves as the initial anomaly probability; wherein, the linear classification layer inputs the feature representation into the sigmoid activation function after linearly combining the weight vector and the bias term.

[0013] As an optional implementation of the first aspect of this application, obtaining the video-level packet probability, constructing the marginal packet likelihood, and obtaining the initial posterior probability of each segment includes: The Noisy-OR aggregation method is used to multiply the initial anomaly probabilities of each segment by taking their complements, and then subtracting the product result from one to obtain the packet probability. The marginal packet likelihood includes maximizing the packet probability for abnormal videos and minimizing the packet probability for normal videos. Under the condition that the video-level label indicates that the video has anomalies, the product of the initial anomaly probability of each segment and the marginal bag likelihood of the corresponding video-level label is calculated, and the ratio of the product to the bag probability is the initial posterior probability; under the condition that the video-level label indicates that the video does not have anomalies, the initial posterior probability is zero; the initial posterior probability satisfies the bag constraint condition that is consistent with the weighted sum of the initial posterior probabilities of all segments and the video-level label.

[0014] As an optional implementation of the first aspect of this application, the calculation of the feature affinity matrix between segments includes: The feature representation is projected from the original feature space to a low-dimensional affinity feature space through an affinity projection layer to obtain the projected features of each segment; the affinity projection layer consists of a linear transformation layer and a layer normalization layer. The features of each segment after projection are normalized by L2 to obtain the affinity feature vector; Calculate the inner product of the affinity feature vectors of any two segments, divide the inner product by the affinity temperature coefficient to obtain the scaled inner product, take the exponent with the natural constant as the base of the scaled inner product as the numerator; with one video segment fixed in the two segments, make the other video segment traverse all video segments, sum the numerators as the denominator. The ratio of the numerator to the denominator is used as an element in the characteristic affinity matrix.

[0015] As an optional implementation of the first aspect of this application, obtaining the refined posterior probability includes: The propagation posterior probability is obtained by performing matrix multiplication between the feature affinity matrix and the initial posterior probability. The propagation posterior probability and the initial posterior probability are weighted and fused based on the posterior refinement intensity coefficient to obtain the refined posterior probability; wherein, the weight corresponding to the initial posterior probability is one minus the posterior refinement intensity coefficient, the weight corresponding to the propagation posterior probability is the posterior refinement intensity coefficient, and the value range of the posterior refinement intensity coefficient is from zero to one.

[0016] As an optional implementation of the first aspect of this application, the construction of the expected maximum distillation loss, and the weighted combination of the bag likelihood loss corresponding to the marginal bag likelihood to obtain the joint loss function, includes: The formula for calculating the expected maximum distillation loss is as follows: ,in This indicates an expectation of maximizing distillation loss. This indicates gradient truncation, which means stopping the gradient propagation operation. Indicates the first The refined posterior probability of each segment, Indicates the first The initial anomaly probability of each video segment. This represents a very small constant used to prevent numerical anomalies in logarithmic operations. Indicates the total number of segments contained in the video; The formula for calculating the bag likelihood loss is as follows: ,in Indicates the package likelihood loss; Indicates video The probability of a packet; This indicates a video tag; it is used when a video is considered abnormal. When the video is a normal video ; Indicates to From 1 to Summation operation; The formula for calculating the joint loss function is as follows: ,in Indicates the total joint loss, This represents the weighting coefficients that are expected to maximize distillation losses.

[0017] As an optional implementation of the first aspect of this application, the output of the anomaly detection results for each segment includes: Based on the video to be detected, anomaly probability scores are output for each segment to form an anomaly score curve; When a binary abnormal event is output, each segment in the abnormal score curve is judged based on a judgment threshold. Segments with an abnormal probability score exceeding the judgment threshold are judged as abnormal segments, and consecutive abnormal segments are merged into an abnormal event. The judgment threshold is the threshold that performs best on a preset evaluation index.

[0018] Secondly, embodiments of this application provide a weakly supervised video anomaly detection system based on packet-constrained expectation-maximization learning, including: The feature extraction and initial anomaly prediction module is used to acquire training video data with video-level labels, divide the training video data into segments and extract the feature representation of each segment, and perform binary classification prediction on the feature representation through a segment classifier to obtain the initial anomaly probability of each segment. The package likelihood construction and Bayesian posterior inference module is used to aggregate and calculate the initial anomaly probability to obtain the video-level package probability. The module combines the package probability and the video-level label to construct the marginal package likelihood. Based on the marginal package likelihood, the module estimates the initial posterior probability of each segment under the video-level label condition through Bayesian posterior inference. The affinity propagation and posterior refinement module is used to project the feature representation onto the affinity feature space to obtain the affinity feature vector, calculate the feature affinity matrix between segments based on the affinity feature vector, use the feature affinity matrix to perform weighted propagation on the initial posterior probability to obtain the propagated posterior probability, and perform weighted fusion of the propagated posterior probability and the initial posterior probability to obtain the refined posterior probability. The joint loss construction module is used to perform gradient truncation on the refined posterior probability and combine it with the initial anomaly probability to construct the expected maximum distillation loss. The expected maximum distillation loss is then weighted and combined with the bag likelihood loss corresponding to the marginal bag likelihood to obtain the joint loss function. The discriminant expectation-maximization training and detection module is used to perform discriminant expectation-maximization optimization training based on the joint loss function, and output the anomaly detection results of each segment based on the video to be detected.

[0019] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the method described in the first aspect.

[0020] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0021] Compared with existing technologies, this invention proposes a weakly supervised video anomaly detection method based on packet-constrained expectation-maximization learning, which has the following beneficial effects: This invention directly models weakly supervised video anomaly detection as a bag-constrained latent variable learning problem. By constructing video-level marginal bag likelihood and obtaining segment-level posterior probabilities based on Bayesian posterior inference, the training target directly corresponds to the original structure of the weakly supervised annotation. This avoids the distortion of the supervision signal introduced by indirect transformation methods such as top-k selection, pseudo-label generation, or anomaly proportion prior, fundamentally improving the effectiveness of weakly supervised learning and enhancing detection performance.

[0022] This invention does not rely on top-k hyperparameters and anomaly ratio priors. It automatically estimates fragment-level anomaly probabilities solely through Bayesian posterior inference and feature affinity propagation, significantly reducing the cost of hyperparameter tuning. It also has strong generalization ability and can be directly transferred to different datasets and application scenarios.

[0023] This invention employs a single-stage discriminative expectation-maximization training strategy. By constructing a joint loss function that includes bag likelihood loss and expectation-maximization distillation loss, it unifies E-step posterior inference and M-step parameter update within an end-to-end joint optimization framework. This eliminates the need for multi-stage iterative self-training, resulting in a simple, efficient, stable, and easy-to-implement and deploy training process.

[0024] In this invention, the posterior probability itself has a clear probabilistic meaning, and the feature affinity propagation has a clear semantic interpretation. The whole method has good interpretability, is easy to analyze and debug, and can meet the needs of high reliability application scenarios.

[0025] The overall network structure of this invention is lightweight, mainly composed of a linear transformation layer and basic statistical operations. It has low computational complexity and a small number of parameters, making it suitable for deployment in edge devices and real-time monitoring systems. Attached Figure Description

[0026] Figure 1 The flowchart shows a weakly supervised video anomaly detection method based on bag-constrained expectation-maximization learning. Figure 2 This is a schematic diagram illustrating the bag likelihood construction, Bayesian posterior inference, affinity propagation, and posterior refinement of the present invention. Figure 3 This is a schematic diagram of the overall network structure of the present invention; Figure 4 This is a schematic diagram of a weakly supervised video anomaly detection system based on bag-constrained expectation-maximization learning. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0029] Example 1 Please see Figure 1 This is a flowchart of a weakly supervised video anomaly detection method based on packet-constrained expectation-maximization learning provided in an embodiment of the present invention. The method may include the following steps: S1: Obtain training video data with video-level labels, divide the training video data into segments and extract the feature representation of each segment, and perform binary classification prediction on the feature representation through a segment classifier to obtain the initial anomaly probability of each segment.

[0030] Obtain training video data with video-level labels. Specifically, video-level labels include labels for abnormal videos and labels for normal videos. Video-level labels are weakly supervised labels, only indicating whether a video segment contains an abnormal event, without needing to indicate the specific time the abnormality occurred.

[0031] The training video data is segmented. Specifically, the training video is evenly divided into multiple video segments at preset time intervals. The preset time interval can be set to 1 second, meaning that each second of video data constitutes one video segment; for a video with a frame rate of 30fps, each segment contains 30 frames.

[0032] Extract initial feature vectors for each segment. Specifically, use a pre-trained 3D convolutional network or other video feature extraction network to extract features from each video segment, obtaining the initial feature vector for each segment. Pre-trained video feature extraction networks include, but are not limited to, I3D networks, SlowFast networks, and VideoMAE networks. The initial feature vector dimension can be 512, 1024, or 2048, etc.

[0033] The extracted initial feature vectors are encoded and transformed to obtain the feature representations of each segment. Specifically, a feature encoder is used to encode and transform the initial feature vectors. The feature encoder consists of a linear transformation layer, a layer normalization layer, a GELU activation function layer, and a Dropout layer connected sequentially. The linear transformation layer maps the initial features from the input dimension to the hidden dimension, the layer normalization layer accelerates training convergence, the GELU activation function introduces non-linearity, and the Dropout layer prevents overfitting. The input dimension can be 512, the hidden dimension can be 128, and the Dropout ratio can be 0.5.

[0034] The fragment classifier performs binary classification prediction on the feature representations of each fragment to obtain the initial anomaly probability of each fragment. Specifically, the fragment classifier consists of a linear classification layer and a sigmoid activation function. The linear classification layer maps the hidden dimension features of the feature representation to a 1-dimensional classification logit, and the sigmoid activation function maps the logit to a probability value between 0 and 1.

[0035] The calculation process for the initial anomaly probability is as follows: , , .in Indicates the first The initial feature vector of each video segment, Indicates the first The original video data of each video segment, This indicates the feature extraction process of the pre-trained video feature extraction network. Indicates the first Feature representation of a video segment Indicates the learnable parameters Encoding transformation processing of parameterized feature encoders, Indicates the first The initial anomaly probability of each video segment. This indicates the processing of the Sigmoid activation function. This represents the transpose of the weight vector of the linear classification layer. This represents the bias term for the linear classification layer.

[0036] S2: Aggregate the initial anomaly probabilities to obtain the video-level packet probabilities. Combine the packet probabilities and the video-level labels to construct the marginal packet likelihood. Based on the marginal packet likelihood, estimate the initial posterior probability of each segment under the video-level label conditions through Bayesian posterior inference.

[0037] See Figure 2 The diagram illustrates packet likelihood construction and Bayesian posterior inference. Specifically, the initial anomalous probabilities of all segments in a video are aggregated to calculate the probability that the video contains at least one anomalous segment, i.e., the video-level packet probability. Assuming that the anomalous events of each segment are independent, the probability that all segments are normal is the product of the normal probabilities of each segment, and the packet probability is 1 minus the probability that all segments are normal.

[0038] The aggregation calculation uses the Noisy-OR aggregation method, which calculates 1 minus the probability that all segments are normal. The specific formula is as follows: ,in Indicates video The packet probability, that is, the probability that the video contains at least one anomalous segment; This indicates the total number of segments contained in the video. Indicates to From 1 to The operation of multiple multiplication.

[0039] Marginal bag likelihood is constructed by combining bag probability and video-level labels. Specifically, for anomalous videos (labels...) ), maximize packet probability; for normal videos (tags) Minimize the packet probability, that is, maximize the probability that all segments are normal.

[0040] Based on the marginal bag likelihood, Bayesian posterior inference is used to estimate the initial posterior probability of each segment under the video-level label conditions. Specifically, for anomalous videos, bag-constrained posterior inference is performed according to Bayes' theorem. Given that the video is known to be anomalous, the... The initial posterior probability of a segment being anomalous is equal to the initial anomalous probability of that segment multiplied by the marginal bag likelihood of the corresponding video-level label, then divided by the bag probability. The initial posterior probability satisfies the bag constraint, meaning the weighted sum of the initial posterior probabilities of all segments is consistent with the video-level label. For normal videos, the bag constraint dictates that the initial posterior probability of all segments is set to zero. For segments without real content, their posterior probability is set to zero. The bag constraint here does not require the sum of the posterior marginal probabilities of all segments to equal the video-level label; it constrains that when... When, the posterior probability is defined only on the segment label configuration where "at least one anomalous segment exists"; when At that time, all segments were normal.

[0041] The core idea of ​​bag-constrained posterior inference is to directly model the original structure of weakly supervised annotation (i.e., video-level labels) without introducing additional assumptions such as top-k selection or anomaly ratio priors. This avoids the problem of mismatch between the training objective and the weakly supervised annotation structure, and also eliminates the impact of hyperparameters (such as k value, anomaly ratio, etc.) on performance.

[0042] S3: Project the feature representation onto the affinity feature space to obtain the affinity feature vector, calculate the feature affinity matrix between segments based on the affinity feature vector, use the feature affinity matrix to perform weighted propagation on the initial posterior probability to obtain the propagated posterior probability, and perform weighted fusion of the propagated posterior probability and the initial posterior probability to obtain the refined posterior probability.

[0043] Since the initial posterior probability obtained solely through bag-constrained posterior inference is based on independent predictions of each segment and lacks consideration of semantic relationships between segments, it may suffer from inaccurate posterior estimation and temporal discontinuities. This step introduces a feature affinity propagation mechanism, enabling semantically similar segments to share anomaly interpretations, thereby obtaining a more accurate and consistent posterior probability estimate. (See also...) Figure 2 The diagram shown illustrates affinity propagation and posterior refinement.

[0044] The feature representation is projected onto the affinity feature space to obtain affinity feature vectors. Specifically, the feature representation is projected from the original feature space to a low-dimensional affinity feature space through an affinity projection layer to obtain projected features. The projected features are then subjected to L2 normalization to obtain affinity feature vectors. The affinity projection layer consists of a linear transformation layer and a layer normalization layer. L2 normalization ensures that all feature vectors lie on a unit hypersphere, facilitating subsequent similarity calculations. The affinity feature space can be set to 64 dimensions.

[0045] The projection and normalization process is as follows ,in Indicates the first Affinity feature vectors of video segments, This represents the weight matrix of the affinity projection layer. This indicates the bias term for the affinity projection layer. This represents L2 norm operations.

[0046] The feature affinity matrix between segments is calculated based on affinity feature vectors. Specifically, the feature affinity between each pair of segments is calculated using an exponential similarity measure, and each row is normalized using Softmax to obtain the feature affinity matrix. A larger inner product indicates that the two segments are more semantically similar and have a higher affinity. The affinity temperature coefficient is used to control the sharpness of the affinity distribution, and its value can be set to 0.2.

[0047] The formula for calculating the characteristic affinity matrix is ​​as follows: ,in The eigenvalue matrix represents the eigenvalue matrix. Line 1 The element of the column, i.e., the first The segment is relative to the first The affinity weight of each segment, Indicates the first Transpose of the affinity feature vectors of each segment Indicates the first Affinity feature vectors of each segment, This represents the affinity temperature coefficient, used to control the sharpness of the affinity distribution. This indicates the treatment of exponential functions with the natural constant as the base. Indicates to From 1 to The summation operation.

[0048] The propagated posterior probability is obtained by weighting the initial posterior probability using the feature affinity matrix. Specifically, the essence of propagation is that the propagated posterior probability of each segment is equal to the weighted average of the initial posterior probabilities of all segments, with the weights determined by feature affinity. Semantically similar segments will "transmit" anomalous evidence to each other, allowing segments related to anomalous events to obtain higher posterior probabilities, while isolated noise responses are suppressed.

[0049] The formula for weighted propagation is as follows: ,in Indicates the first The posterior probability of propagation for each segment. Indicates the first The initial posterior probability of each segment.

[0050] The propagated posterior probability is weighted and fused with the initial posterior probability to obtain the refined posterior probability. Specifically, the initial posterior probability and the propagated posterior probability are weighted and fused using adjustable fusion weights. The fusion weights are determined by the posterior refinement strength coefficient. The value is controlled, ranging from 0 to 1, with a preferred value of 0.4. When When it degenerates to the initial posterior, when The timing is entirely determined by the posterior propagation.

[0051] The formula for weighted fusion is: ,in Indicates the first The refined posterior probability of each segment, This represents the posterior refinement intensity coefficient, used to control the weight ratio of the propagation posterior in the fusion process, and its value ranges from 0 to 1.

[0052] The core idea of ​​feature affinity posterior refinement is to share anomaly interpretations among semantically consistent segments by propagating posterior probabilities between semantically similar segments, thereby improving the accuracy and temporal consistency of posterior probabilities and alleviating the problem of posterior noise caused by inaccurate initial predictions.

[0053] S4: Perform gradient truncation on the refined posterior probability and construct the expected maximum distillation loss by combining it with the initial anomaly probability. Then, perform a weighted combination of the expected maximum distillation loss and the bag likelihood loss corresponding to the marginal bag likelihood to obtain the joint loss function.

[0054] Gradient truncation is applied to the refined posterior probabilities. Specifically, the refined posterior probabilities are stopped during backpropagation, meaning they are not included in gradient calculation and are treated as fixed soft labels. This is crucial for achieving discriminative expectation maximization (EM), decoupling the expectation step (E-step) and maximization step (M-step) in the computational graph.

[0055] An Expected Maximization Distillation Loss is constructed by combining the initial anomaly probabilities. Specifically, based on the initial anomaly probabilities of the fragments and the refined posterior probabilities after gradient truncation, an Expected Maximization Distillation Loss in the form of binary cross-entropy is constructed. This loss measures the difference between the initial anomaly probabilities and the refined posterior probabilities of the fragments, and only updates the parameters of the fragment classifier and feature encoder.

[0056] The formula for maximizing distillation loss is as follows: ,in This indicates an expectation of maximizing distillation loss. This indicates gradient truncation, which means stopping the gradient propagation operation. Indicates the first The refined posterior probability of each segment, Indicates the first The initial anomaly probability of each video segment. It represents a very small constant used to prevent numerical anomalies in logarithmic operations.

[0057] Construct the bag likelihood loss corresponding to the marginal bag likelihood. Specifically, the loss for normal video is expressed as the average log-likelihood, applying a uniform constraint to all segments of the normal video. The bag likelihood loss corresponding to the marginal bag likelihood is: ,in Indicates the package likelihood loss; This indicates a video tag; it is used when a video is considered abnormal. When the video is a normal video ; Indicates to From 1 to The summation operation.

[0058] The joint loss function is obtained by weighting the expectation-maximization distillation loss and the bag likelihood loss corresponding to the marginal bag likelihood. Specifically, the bag likelihood loss ensures that the prediction satisfies the video-level weak supervision constraint, while the expectation-maximization distillation loss uses the more refined segment-level posterior inferred from the E-step to guide the classifier's learning.

[0059] The formula for calculating the joint loss function is as follows: ,in Indicates the total joint loss, This represents the weighting coefficient for maximizing distillation loss, used to balance the relative importance of the two losses. Its value ranges from 0.1 to 2.0, with a preferred value of 0.5.

[0060] S5: Perform discriminative expectation-maximization optimization training based on the joint loss function, and output the anomaly detection results of each segment based on the video to be detected.

[0061] Discriminative expectation-maximization optimization training is performed based on the joint loss function. Specifically, the learnable parameters of the feature encoder, fragment classifier, and affinity projection layer are alternately updated using a stochastic gradient descent optimizer and a backpropagation algorithm. The optimization training employs the Adam optimizer, with an initial learning rate set to 1×10⁻⁶. -4 The weight decay coefficient is set to 1×10. -5 A cosine annealing learning rate scheduling strategy is adopted, with a total of 50 training epochs. After a preset number of training epochs, the AUC metric is calculated on the test dataset, and the model with the highest AUC is saved as the best model.

[0062] Discriminative Expectation-Maximization (EM) optimization performs post-hoc estimation and parameter updates within each mini-batch training iteration: First, initial posterior probabilities are calculated based on the initial anomaly probabilities of the current segment and the video-level labels. These initial posterior probabilities are then refined using the feature affinity matrix between segments, and the refined results are used as soft-label supervision signals for gradient truncation. Subsequently, the feature encoder and segment classifier parameters are updated by minimizing the bag-likelihood loss and the Expectation-Maximization distillation loss. This process does not rely on additional multi-stage training or explicit iterative pseudo-label selection.

[0063] The anomaly detection results for each segment of the video to be detected are output. Specifically, for each segment of the video to be detected, an anomaly probability score is output, forming an anomaly score curve. When binary anomaly events need to be output, a validation set is used to determine the judgment threshold. Calculate a pre-defined evaluation metric (such as F1 score, Youden index, or recall rate that meets a specified false positive rate) on the candidate threshold set, and select the threshold with the best performance as the candidate threshold. When the segment score is greater than or equal to When a segment is deemed abnormal, it is considered an anomaly; consecutive abnormal segments are merged into an anomalous event. The threshold is determined solely using training / validation data, and the test set is used only for final performance evaluation.

[0064] In summary, this invention proposes a weakly supervised video anomaly detection method based on packet-constrained expectation-maximization learning in this embodiment. See also... Figure 3 The diagram shown is a schematic of the overall network structure of the present invention. The overall network structure of the present invention includes: initial feature vector input of video segment, feature encoder (including linear transformation layer, layer normalization layer, activation function layer, random deactivation layer), classification head (including linear classification layer, sigmoid activation), affinity projection head (including linear transformation layer, normalization layer), affinity propagation and posterior refinement module (including affinity matrix calculation, neighborhood feature propagation, weighted fusion output). This method employs bag-constrained Bayesian posterior inference to directly model the original structure of video-level weakly supervised labels. It refines the posterior probability propagation between semantically similar segments through feature affinity matrices and unifies E-step posterior inference and M-step distillation learning within an end-to-end single-stage joint optimization framework. This approach aims to address the problems in existing weakly supervised video anomaly detection methods, such as mismatch between training objectives and weakly supervised label structures leading to distorted supervision signals, reliance on top-k selection and anomaly proportion priors resulting in highly sensitive hyperparameters and poor generalization ability, complex and inefficient multi-stage iterative pseudo-label training processes prone to model drift, and insufficient interpretability due to the lack of semantic association modeling in posterior inference.

[0065] Example 2 In Embodiment 2 of this application, the specific implementation methods in each step of Embodiment 1 above can be replaced: Alternatives to feature extraction networks: In addition to pre-trained networks such as I3D, SlowFast, and VideoMAE, other 3D convolutional networks or video Transformers such as C3D, R(2+1)D, X3D, and VideoSwin can be used as feature extraction networks; multimodal features such as optical flow features and audio features can also be used as supplementary inputs.

[0066] Alternatives to the feature encoder structure: In addition to the "linear + layer normalization + GELU + Dropout" structure, the feature encoder can also use other encoding structures such as multilayer perceptron (MLP), one-dimensional convolutional network, self-attention layer, etc.; the activation function can be replaced with ReLU, LeakyReLU, SiLU, etc.; the normalization method can be replaced with batch normalization, instance normalization, etc.

[0067] Alternatives to packet probability aggregation calculation: In addition to using the Noisy-OR aggregation method of "1 - product of normal probabilities of each segment" to calculate packet probabilities, other aggregation methods such as generalized mean aggregation, attention-weighted aggregation, and max pooling aggregation can be used to calculate video-level packet probabilities.

[0068] Alternatives to calculating the feature affinity matrix: In addition to using exponential inner product similarity to calculate the feature affinity matrix, other similarity measures such as Gaussian kernel similarity, cosine similarity, and exponential transformation of Euclidean distance can also be used; in addition to using Softmax row normalization, the affinity matrix can also be normalized by row summation, symmetric normalization, etc.

[0069] Alternatives to the propagation of the initial posterior probability: In addition to propagation using the first-order affinity matrix, other propagation methods such as multi-hop propagation (i.e., the k-th power of the affinity matrix), graph convolutional network (GCN) propagation, and attention-weighted propagation can be used to refine the neighborhood propagation of the posterior probability.

[0070] Alternatives to the joint loss function: In addition to using the binary cross-entropy form, the expected-maximization distillation loss can also use other metrics such as mean squared error (MSE) and KL divergence; In addition to linear weighting, the weighting of the total joint loss can also use adaptive weighting, dynamic weight adjustment, etc. to balance the two losses.

[0071] Alternatives to optimizers and training strategies: In addition to the Adam optimizer, other optimizers such as SGD, AdamW, and RMSprop can be used; learning rate scheduling strategies, in addition to cosine annealing, can include step decay, exponential decay, warmup + cosine annealing, etc.; data augmentation, model ensemble, and other techniques can be introduced during training to further improve performance.

[0072] Example 3 Please see Figure 4 The figure shows a schematic diagram of the weakly supervised video anomaly detection system based on packet-constrained expectation-maximization learning proposed in the third embodiment of this application. The system includes the following key modules: The feature extraction and initial anomaly prediction module 100 is used to acquire training video data with video-level labels, divide the training video data into segments and extract the feature representation of each segment, and perform binary classification prediction on the feature representation through a segment classifier to obtain the initial anomaly probability of each segment. The package likelihood construction and Bayesian posterior inference module 200 is used to aggregate and calculate the initial anomaly probability to obtain the video-level package probability, combine the package probability and the video-level label to construct the marginal package likelihood, and estimate the initial posterior probability of each segment under the video-level label condition through Bayesian posterior inference based on the marginal package likelihood. The affinity propagation and posterior refinement module 300 is used to project the feature representation onto the affinity feature space to obtain the affinity feature vector, calculate the feature affinity matrix between segments based on the affinity feature vector, use the feature affinity matrix to perform weighted propagation on the initial posterior probability to obtain the propagated posterior probability, and perform weighted fusion of the propagated posterior probability and the initial posterior probability to obtain the refined posterior probability. The joint loss construction module 400 is used to perform gradient truncation on the refined posterior probability and combine it with the initial anomaly probability to construct the expected maximum distillation loss. The expected maximum distillation loss is then weighted and combined with the bag likelihood loss corresponding to the marginal bag likelihood to obtain the joint loss function. The discriminant expectation-maximization training and detection module 500 is used to perform discriminant expectation-maximization optimization training based on the joint loss function and output the anomaly detection results of each segment based on the video to be detected.

[0073] The weakly supervised video anomaly detection system based on packet-constrained expectation-maximization learning in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), etc. This application embodiment does not impose specific limitations.

[0074] The weakly supervised video anomaly detection system based on bag-constrained expectation-maximization learning in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0075] The weakly supervised video anomaly detection system based on packet-constrained expectation-maximization learning provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method embodiment based on bag-constrained expectation-maximization learning for weakly supervised video anomaly detection will not be described again here to avoid repetition.

[0076] Optionally, embodiments of this application also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the embodiments of the weakly supervised video anomaly detection method based on package-constrained expectation maximization learning described above, and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0077] This application also provides a readable storage medium storing a program or instructions that, when executed by a processor, implement various processes of an embodiment of a weakly supervised video anomaly detection method based on packet-constrained expectation maximization learning, and achieve the same technical effect. To avoid repetition, these will not be described again here.

[0078] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0079] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0080] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0081] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A weakly supervised video anomaly detection method based on bag-constrained expectation-maximization learning, characterized in that, Includes the following steps: Acquire training video data with video-level labels, divide the training video data into segments and extract feature representations of each segment, and perform binary classification prediction on the feature representations through a segment classifier to obtain the initial anomaly probability of each segment. The initial anomaly probabilities are aggregated to obtain the video-level packet probabilities. The packet probabilities and the video-level labels are combined to construct the marginal packet likelihood. Based on the marginal packet likelihood, the initial posterior probability of each segment under the video-level label conditions is estimated by Bayesian posterior inference. The feature representation is projected onto the affinity feature space to obtain the affinity feature vector. Based on the affinity feature vector, the feature affinity matrix between segments is calculated. The initial posterior probability is weighted and propagated using the feature affinity matrix to obtain the propagated posterior probability. The propagated posterior probability is then weighted and fused with the initial posterior probability to obtain the refined posterior probability. Gradient truncation is performed on the refined posterior probability and combined with the initial anomaly probability to construct the expected maximum distillation loss. The expected maximum distillation loss is then weighted and combined with the bag likelihood loss corresponding to the marginal bag likelihood to obtain the joint loss function. Discriminative expectation-maximization optimization training is performed based on the joint loss function, and anomaly detection results for each segment are output based on the video to be detected.

2. The weakly supervised video anomaly detection method based on bag-constrained expectation-maximization learning as described in claim 1, characterized in that, The extraction of feature representations for each segment includes: The training video data is evenly divided into multiple video segments according to a preset time interval; The video feature extraction network is used to extract features from each video segment to obtain the initial feature vector of each segment; The initial feature vector is sequentially subjected to linear projection, normalization, nonlinear activation, and random deactivation by a feature encoder to obtain the feature representation; the feature encoder includes a linear projection layer, a normalization layer, an activation function layer, and a random deactivation layer connected in sequence.

3. The weakly supervised video anomaly detection method based on bag-constrained expectation-maximization learning as described in claim 1, characterized in that, The initial anomaly probability of each segment is obtained, including: The fragment classifier includes a linear classification layer and a sigmoid activation function; the linear classification layer maps the feature representation to a one-dimensional classification logic value, and the sigmoid activation function maps the one-dimensional classification logic value to a probability value between zero and one, which serves as the initial anomaly probability; wherein, the linear classification layer inputs the feature representation into the sigmoid activation function after linearly combining the weight vector and the bias term.

4. The weakly supervised video anomaly detection method based on bag-constrained expectation-maximization learning as described in claim 1, characterized in that, The process of obtaining video-level packet probabilities, constructing marginal packet likelihoods, and obtaining initial posterior probabilities for each segment includes: The Noisy-OR aggregation method is used to multiply the initial anomaly probabilities of each segment by taking their complements, and then subtracting the product result from one to obtain the packet probability. The marginal packet likelihood includes maximizing the packet probability for abnormal videos and minimizing the packet probability for normal videos. Under the condition that the video-level label indicates that the video has anomalies, the product of the initial anomaly probability of each segment and the marginal bag likelihood of the corresponding video-level label is calculated, and the ratio of the product to the bag probability is the initial posterior probability; under the condition that the video-level label indicates that the video does not have anomalies, the initial posterior probability is zero; the initial posterior probability satisfies the bag constraint condition that is consistent with the weighted sum of the initial posterior probabilities of all segments and the video-level label.

5. The weakly supervised video anomaly detection method based on bag-constrained expectation-maximization learning as described in claim 1, characterized in that, The computational feature affinity matrix between fragments includes: The feature representation is projected from the original feature space to a low-dimensional affinity feature space through an affinity projection layer to obtain the projected features of each segment; the affinity projection layer consists of a linear transformation layer and a layer normalization layer. The features of each segment after projection are normalized by L2 to obtain the affinity feature vector; Calculate the inner product of the affinity feature vectors of any two segments, divide the inner product by the affinity temperature coefficient to obtain the scaled inner product, take the exponent with the natural constant as the base of the scaled inner product as the numerator; with one video segment fixed in the two segments, make the other video segment traverse all video segments, sum the numerators as the denominator. The ratio of the numerator to the denominator is used as an element in the characteristic affinity matrix.

6. The weakly supervised video anomaly detection method based on bag-constrained expectation-maximization learning as described in claim 1, characterized in that, The process of obtaining the refined posterior probability includes: The propagation posterior probability is obtained by performing matrix multiplication between the feature affinity matrix and the initial posterior probability. The propagation posterior probability and the initial posterior probability are weighted and fused based on the posterior refinement intensity coefficient to obtain the refined posterior probability; wherein, the weight corresponding to the initial posterior probability is one minus the posterior refinement intensity coefficient, the weight corresponding to the propagation posterior probability is the posterior refinement intensity coefficient, and the value range of the posterior refinement intensity coefficient is from zero to one.

7. The weakly supervised video anomaly detection method based on bag-constrained expectation-maximization learning as described in claim 1, characterized in that, The construction of the expected maximum distillation loss, combined with the bag likelihood loss corresponding to the marginal bag likelihood, yields a joint loss function, including: The formula for calculating the expected maximum distillation loss is as follows: ,in This indicates an expectation of maximizing distillation loss. This indicates gradient truncation, which means stopping the gradient propagation operation. Indicates the first The refined posterior probability of each segment, Indicates the first The initial anomaly probability of each video segment. This represents a very small constant used to prevent numerical anomalies in logarithmic operations. Indicates the total number of segments contained in the video; The formula for calculating the bag likelihood loss is as follows: ,in Indicates the package likelihood loss; Indicates video The probability of a packet; This indicates a video tag; it is used when a video is considered abnormal. When the video is a normal video ; Indicates to From 1 to Summation operation; The formula for calculating the joint loss function is as follows: ,in Indicates the total joint loss, This represents the weighting coefficients that are expected to maximize distillation losses.

8. The weakly supervised video anomaly detection method based on bag-constrained expectation-maximization learning as described in claim 1, characterized in that, The anomaly detection results for each output segment include: Based on the video to be detected, anomaly probability scores are output for each segment to form an anomaly score curve; When a binary abnormal event is output, each segment in the abnormal score curve is judged based on a judgment threshold. Segments with an abnormal probability score exceeding the judgment threshold are judged as abnormal segments, and consecutive abnormal segments are merged into an abnormal event. The judgment threshold is the threshold that performs best on a preset evaluation index.

9. A weakly supervised video anomaly detection system based on bag-constrained expectation-maximization learning, characterized in that, include: The feature extraction and initial anomaly prediction module is used to acquire training video data with video-level labels, divide the training video data into segments and extract the feature representation of each segment, and perform binary classification prediction on the feature representation through a segment classifier to obtain the initial anomaly probability of each segment. The package likelihood construction and Bayesian posterior inference module is used to aggregate and calculate the initial anomaly probability to obtain the video-level package probability. The module combines the package probability and the video-level label to construct the marginal package likelihood. Based on the marginal package likelihood, the module estimates the initial posterior probability of each segment under the video-level label condition through Bayesian posterior inference. The affinity propagation and posterior refinement module is used to project the feature representation onto the affinity feature space to obtain the affinity feature vector, calculate the feature affinity matrix between segments based on the affinity feature vector, use the feature affinity matrix to perform weighted propagation on the initial posterior probability to obtain the propagated posterior probability, and perform weighted fusion of the propagated posterior probability and the initial posterior probability to obtain the refined posterior probability. The joint loss construction module is used to perform gradient truncation on the refined posterior probability and combine it with the initial anomaly probability to construct the expected maximum distillation loss. The expected maximum distillation loss is then weighted and combined with the bag likelihood loss corresponding to the marginal bag likelihood to obtain the joint loss function. The discriminant expectation-maximization training and detection module is used to perform discriminant expectation-maximization optimization training based on the joint loss function, and output the anomaly detection results of each segment based on the video to be detected.

10. An electronic device, characterized in that, The method includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the weakly supervised video anomaly detection method based on packet-constrained expectation-maximization learning as described in any one of claims 1-8.

Images