Abnormal event detection method, apparatus, device, storage medium, and product

Abstract

This application relates to the field of computer vision technology and discloses an anomaly event detection method, apparatus, device, storage medium, and product. The method includes: based on a deep neural network encoder, mapping video event samples to a feature space to obtain video features and initial clusters; based on the expectation-maximization algorithm, obtaining optimized clusters, an updated deep neural network encoder, and a prior distribution based on the video features and initial clusters; and detecting anomaly events in the video based on the optimized clusters, the updated deep neural network encoder, and the prior distribution. This application, through the expectation-maximization algorithm, can iteratively optimize the initial clusters and deep neural network encoder based on video features and initial clusters, and learn the prior distribution, improving cluster accuracy and better adapting to data characteristics. This results in more accurate extraction of video features, better differentiation between normal and abnormal events during anomaly detection, and improved detection accuracy.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an abnormal event detection method, apparatus, device, storage medium and product. Background Technology

[0002] The determination of abnormal events in real-world scenarios depends on a human expectation pattern of the event's occurrence. This expectation is often highly subjective and abstract. However, existing research shows that video data can be considered as sampled from a mixed distribution of multiple event types, and the human-defined expectation pattern can be characterized by the common prior distribution of event types.

[0003] When solving for the prior distribution in existing anomaly detection technologies, the number of clusters in the clustering algorithm needs to be manually specified in advance. This specified value usually does not match the actual number of events, leading to distribution estimation bias. Furthermore, the detection model relies entirely on the distribution likelihood function estimated from normal samples to identify anomalies, which can easily misclassify anomalies that are similar to normal samples as normal, thus causing detection bias. Summary of the Invention

[0004] The main objective of this application is to provide an abnormal event detection method, apparatus, equipment, storage medium, and product, aiming to solve the technical problem that existing abnormal event detection methods are prone to detection deviations.

[0005] To achieve the above objectives, this application proposes an abnormal event detection method, the method comprising:

[0006] Based on a deep neural network encoder, video features and initial clusters are obtained by mapping video event samples to a feature space;

[0007] Based on the expectation-maximization algorithm, the optimized cluster, the updated deep neural network encoder, and the prior distribution are obtained according to the video features and the initial cluster.

[0008] Anomaly events in the video are detected based on the optimized clustering, the updated deep neural network encoder, and the prior distribution.

[0009] In one embodiment, the video features include normal event features and abnormal event features. The step of obtaining the optimized cluster, the updated deep neural network encoder, and the prior distribution based on the expectation-maximization algorithm, according to the video features and the initial cluster, includes:

[0010] Based on the expectation-maximization algorithm, the optimal cluster is obtained according to the relative relationship between the video features and the initial prior distribution, and the relative relationship between the video features and the initial cluster. The optimal cluster includes the optimal normal cluster and the optimal abnormal cluster.

[0011] Based on normal event features, abnormal event features, the optimal cluster, and the preset prior distribution, the deep neural network encoder and the optimal cluster are optimized according to the comparison clustering loss and the prior consistency constraint loss to obtain the optimization result.

[0012] When the optimization results satisfy the convergence condition, the optimized cluster, the updated deep neural network encoder, and the prior distribution are obtained.

[0013] In one embodiment, the step of obtaining an optimal cluster based on the expectation-maximization algorithm, according to the relative relationship between the video features and the initial prior distribution, and the relative relationship between the video features and the initial cluster, wherein the optimal cluster includes an optimal normal cluster and an optimal abnormal cluster, includes:

[0014] Based on the expectation-maximization algorithm, combined with the prior-guided cluster search mechanism, the relative relationship between the video features and the initial prior distribution, and the relative relationship between the video features and the initial cluster are determined to obtain the judgment result;

[0015] Based on the judgment result, if the metric of the normal event feature and the nearest initial cluster center is less than the metric of the normal event feature and the initial prior distribution center, then the normal event feature is assigned to the initial cluster, and the initial cluster is taken as the best normal cluster;

[0016] If the metric of the normal event feature and the initial prior distribution center is less than the metric of the initial cluster center at any distance, then a new cluster is established for the normal event feature, and the new cluster is taken as the optimal normal cluster.

[0017] The abnormal event feature with the highest abnormality score is selected as the abnormality candidate feature, and the optimal abnormal cluster is obtained based on the abnormality candidate feature.

[0018] Based on the optimal normal cluster and the optimal abnormal cluster, the optimal cluster corresponding to the video feature is obtained.

[0019] In one embodiment, the step of optimizing the deep neural network encoder and the optimal cluster based on normal event features, abnormal event features, the optimal cluster, and a preset prior distribution, and by comparing clustering loss and prior consistency constraint loss to obtain the optimization result, includes:

[0020] Select normal event features with the lowest abnormality scores as normal candidate features;

[0021] Based on the contrastive clustering loss, the similarity between the normal candidate features and the best normal cluster is increased, and the similarity between the normal candidate features and the best abnormal cluster is decreased, to obtain the first cluster optimization result;

[0022] Based on the comparative clustering loss, the similarity between the abnormal candidate features and the best normal cluster is reduced, and the similarity between the abnormal candidate features and the best abnormal cluster is increased to obtain the second clustering optimization result;

[0023] The deep neural network encoder and the preset prior distribution are optimized based on the prior consistency constraint loss to obtain the encoder optimization result and the prior distribution optimization result.

[0024] Based on the first cluster optimization result, the second cluster optimization result, the encoder optimization result, and the prior distribution optimization result, the optimization result is obtained.

[0025] In one embodiment, the step of detecting anomalous events in the video based on the optimized clustering, the updated deep neural network encoder, and the prior distribution includes:

[0026] The updated deep neural network encoder is used to extract feature vectors from the video.

[0027] Based on the optimized clustering, the distance between the feature vector and the prior distribution is calculated to obtain the anomaly score;

[0028] The abnormal events in the video are obtained based on the abnormality score.

[0029] In one embodiment, the step of obtaining video features and initial clusters by mapping video event samples to a feature space based on a deep neural network encoder includes:

[0030] Based on a deep neural network encoder, initial video features are obtained by mapping event samples to a feature space;

[0031] The initial video features are normalized using the Euclidean norm to obtain video features; initial clusters are obtained by preliminary clustering of the video features.

[0032] Furthermore, to achieve the above objectives, this application also proposes an abnormal event detection device, which includes:

[0033] The feature extraction module is used to obtain video features and initial clusters by mapping video event samples to the feature space based on a deep neural network encoder.

[0034] The parameter iteration module is used to obtain the optimized cluster, the updated deep neural network encoder, and the prior distribution based on the video features and the initial cluster.

[0035] An anomaly detection module is used to detect anomalous events in the video based on the optimized clustering, the updated deep neural network encoder, and the prior distribution.

[0036] In addition, to achieve the above objectives, this application also proposes an abnormal event detection device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the abnormal event detection method as described above.

[0037] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the abnormal event detection method described above.

[0038] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the abnormal event detection method described above.

[0039] The technical solution proposed in this application, based on a deep neural network encoder, obtains video features and initial clusters by mapping video event samples to a feature space. Using the expectation-maximization algorithm, it obtains optimized clusters, an updated deep neural network encoder, and a prior distribution based on the video features and initial clusters. Abnormal events in the video are then detected based on these optimized clusters, the updated deep neural network encoder, and the prior distribution. This application, through the expectation-maximization algorithm, iteratively optimizes the initial clusters and the deep neural network encoder based on the video features and initial clusters, and learns the prior distribution. This improves cluster accuracy and better adapts to data characteristics, thereby extracting video features more accurately and better distinguishing between normal and abnormal events during anomaly detection, thus improving detection accuracy. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating an embodiment of the abnormal event detection method of this application.

[0043] Figure 2 This is a flowchart illustrating Embodiment 2 of the abnormal event detection method of this application;

[0044] Figure 3 This is a schematic diagram of the expected value maximization algorithm in this application;

[0045] Figure 4 This is a flowchart illustrating Embodiment 3 of the abnormal event detection method of this application;

[0046] Figure 5 This is a schematic diagram of the module structure of the abnormal event detection device according to an embodiment of this application;

[0047] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the abnormal event detection method in this application embodiment.

[0048] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0050] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0051] When solving for the prior distribution in existing anomaly detection technologies, the number of clusters in the clustering algorithm needs to be manually specified in advance. This specified value usually does not match the actual number of events, leading to distribution estimation bias. Furthermore, the detection model relies entirely on the distribution likelihood function estimated from normal samples to identify anomalies, which can easily misclassify anomalies that are similar to normal samples as normal, thus causing detection bias.

[0052] Therefore, in order to overcome the above-mentioned defects, this application provides a solution that uses the expectation-maximization algorithm to iteratively optimize the initial cluster and deep neural network encoder based on video features and the initial cluster, and learns the prior distribution. This improves the cluster accuracy and can better adapt to the data characteristics, thereby extracting video features more accurately, better distinguishing between normal and abnormal events in the anomaly detection process, and improving the detection accuracy.

[0053] It should be noted that the executing entity of each embodiment of this application can be a computing service system with data processing, network communication, and program execution functions, such as an electronic system, an anomaly event detection system, or a model capable of implementing the above functions. The following description uses an anomaly event detection model (hereinafter referred to as "the model") as an example to illustrate the following embodiments.

[0054] Based on this, embodiments of this application provide an abnormal event detection method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the abnormal event detection method of this application.

[0055] In this embodiment, the abnormal event detection method includes steps S10 to S30:

[0056] Step S10: Based on a deep neural network encoder, video features and initial clusters are obtained by mapping video event samples to the feature space.

[0057] It should be noted that a deep neural network encoder is a deep learning model that compresses input data into a low-dimensional feature representation through multiple layers of neural networks. This feature representation can be used for subsequent classification, regression, or clustering tasks. The feature space is a high-dimensional space where each dimension represents a feature. In this space, similar data points (such as video frames with similar attributes) will cluster together. Video features are representative information extracted from video event samples, such as edges, textures, colors, and shapes, which can be used to describe the attributes and patterns of video content. The initial cluster refers to the cluster centers and their corresponding set of data points formed before the clustering algorithm begins, based on certain initial conditions (such as random selection or based on certain rules).

[0058] Understandably, a deep neural network encoder is used to map video event samples (which may contain video frames, video clips, or other forms of data) to a feature space, thereby extracting representative video features. After the video event samples are mapped to the feature space, they can be clustered or classified according to the video features to obtain initial clusters, which will serve as the starting point for subsequent optimization processes.

[0059] Step S20: Based on the expectation-maximization algorithm, obtain the optimized cluster, the updated deep neural network encoder, and the prior distribution according to the video features and the initial cluster.

[0060] It should be noted that the Expectation-Maximization (EM) algorithm proposed in this application is an iterative optimization algorithm, commonly used in problems with latent variables, especially when estimating model parameters. In this application, both the prior distribution solution and the anomaly detection are based on the Expectation-Maximization algorithm, which mainly includes two steps:

[0061] E-step (Expectation Step): Calculate the expected value of the initial clustering parameters under the current parameter estimation (i.e., estimate the distribution of the latent variables). Specifically, this step calculates the conditional probability distribution of the initial clustering parameters using the current parameter estimation; that is, given the observed data and the current parameters, estimate the expected value of the initial clustering parameters.

[0062] M-step (Maximization Step): Based on the expected values ​​of the initial clustering parameters obtained in the E-step, the likelihood function is maximized to update the model parameters. That is, the parameters are re-estimated using the estimated values ​​of the initial clustering parameters obtained in the E-step, so that the deep neural network encoder can better fit the observed data.

[0063] It should be understood that the E-step and M-step of the expectation-maximization algorithm are continuously iterated and optimized to the expected convergence condition, which can yield the optimized cluster, the updated deep neural network encoder, and the prior distribution.

[0064] Understandably, because this application involves latent variables (i.e., parameters of the initial cluster) set based on subjective prior assumptions, traditional maximum likelihood estimation methods are difficult to use. Directly solving for maximum likelihood estimation can be very complex, or even impossible. The expectation-maximization algorithm simplifies this process through step-by-step (E-step and M-step) optimization, allowing deep neural network encoders to perform indirect optimization under the condition of the initial cluster. This makes the computation at each step more efficient, especially in large-scale datasets and complex models, where the expectation-maximization algorithm can often effectively find a local optimum.

[0065] Step S30: Detect abnormal events in the video based on the optimized cluster, the updated deep neural network encoder, and the prior distribution.

[0066] It should be understood that the updated deep neural network encoder first converts each frame of the video into a feature vector, and these feature vectors are then input into an optimized cluster to determine whether the frame contains normal or abnormal events. Then, based on the prior distribution, the algorithm becomes more sensitive to certain types of abnormal events while ignoring those unlikely to be abnormal. The optimized cluster accurately reflects the similarity of different events, enabling more effective identification of abnormal patterns. The updated deep neural network encoder, through its feature extraction capabilities, maps video frames into a feature space, providing high-quality input for anomaly detection. The prior distribution, as a preliminary hypothesis or expectation of video event samples, can more accurately distinguish between normal and abnormal events.

[0067] It should be noted that this application implements anomaly detection training under sparse annotation, that is, it proposes a method for the discrimination and screening of anomalous events in anomalous videos under weak supervision (using only video-level anomaly annotations). It can complete the learning of normal and anomalous semantic frameworks with only video-level anomaly event annotations, which greatly reduces the dependence on the annotation scale and improves the universality of the model.

[0068] This embodiment, based on a deep neural network encoder, obtains video features and initial clusters by mapping video event samples to a feature space. Using the expectation-maximization algorithm, it obtains optimized clusters, an updated deep neural network encoder, and a prior distribution based on the video features and initial clusters. Abnormal events in the video are then detected based on these optimized clusters, the updated deep neural network encoder, and the prior distribution. The expectation-maximization algorithm iteratively optimizes the initial clusters and the deep neural network encoder based on the video features and initial clusters, and learns the prior distribution, improving cluster accuracy and better adapting to data characteristics. This results in more precise extraction of video features, better differentiation between normal and abnormal events during anomaly detection, and improved detection accuracy.

[0069] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Step S20 may include steps S201 to S203:

[0070] Step S201: Based on the expectation-maximization algorithm, according to the relative relationship between the video features and the initial prior distribution, and the relative relationship between the video features and the initial cluster, the optimal cluster is obtained. The optimal cluster includes the optimal normal cluster and the optimal abnormal cluster.

[0071] It should be noted that the initial prior distribution is the distribution of beliefs or assumptions about the possible values ​​of a parameter or variable before any data is observed; the optimal cluster is the cluster that best reflects the inherent structure of the dataset obtained through optimization algorithms during the clustering process; the optimal normal cluster is the cluster that contains the characteristics of normal events in the optimal cluster; the optimal abnormal cluster is the cluster that contains the characteristics of abnormal events in the optimal cluster.

[0072] It is understandable that in the actual operation of step S201, in order to accurately distinguish and classify normal event features and abnormal event features in the video, it is necessary not only to rely on the direct relationship between the video features themselves and the initial clusters, but also to integrate the additional information provided by the prior distribution to improve the accuracy and robustness of clustering, especially when facing complex and ever-changing video data. Therefore, a priori-guided cluster search mechanism is introduced to connect the relative relationships between video features, the initial prior distribution, and the initial cluster. Step S201 may include: based on the expectation-maximization algorithm, combined with the priori-guided cluster search mechanism, determining the relative relationship between the video features and the initial prior distribution, and the relative relationship between the video features and the initial cluster, to obtain a determination result; based on the determination result, if the metric between the normal event feature and the nearest initial cluster center is less than the metric between the normal event feature and the initial prior distribution center, then the normal event feature is assigned to the initial cluster, and the initial cluster is taken as the best normal cluster; if the metric between the normal event feature and the initial prior distribution center is less than the metric of any distance from the initial cluster center, then a new cluster is established for the normal event feature, and the new cluster is taken as the best normal cluster; selecting the abnormal event feature with the highest abnormal score as the abnormal candidate feature, and obtaining the best abnormal cluster based on the abnormal candidate feature; based on the best normal cluster and the best abnormal cluster, obtaining the best cluster corresponding to the video feature.

[0073] It should be understood that the prior-guided cluster search mechanism first utilizes the core idea of ​​the expectation-maximization algorithm to iteratively update the clustering parameters and model parameters to maximize the likelihood function of the data. In this process, it not only considers the distance (i.e., metric) from video features to each initial cluster center, but also evaluates the degree of deviation of these features from the initial prior distribution center. This dual consideration ensures that the clustering process can both respect the natural distribution of the data and be guided by prior knowledge to a certain extent, thereby avoiding getting trapped in local optima.

[0074] Specifically, for each normal event feature, the metric from its nearest initial cluster center and its metric from its initial prior distribution center are calculated. If the metric from the normal event feature to its nearest initial cluster center is less than the metric from its initial prior distribution center, it indicates that the feature better matches the distribution pattern of the existing clusters. Therefore, it is assigned to the corresponding initial cluster and considered part of the optimal normal cluster. Conversely, if the metric from the feature to its initial prior distribution center is smaller, it means that it may represent a new, not yet fully expressed normal pattern. In this case, a new cluster should be created for it and marked as the optimal normal cluster.

[0075] Meanwhile, an anomaly score-based strategy was adopted for the identification and clustering of anomalous event features. By using pre-defined scoring criteria or models, features with the highest anomaly scores were selected as candidate anomaly features. These features often deviate from the expected normal patterns, indicating potential anomalous events. Subsequently, based on these candidate anomaly features, the expectation-maximization algorithm was used for further iterative optimization to form optimal anomaly clusters that closely surround the anomalous features.

[0076] Step S202: Based on normal event features, abnormal event features, the optimal cluster, and the preset prior distribution, optimize the deep neural network encoder and the optimal cluster according to the comparison cluster loss and the prior consistency constraint loss to obtain the optimization result.

[0077] It should be noted that cluster loss is typically used to evaluate the performance of clustering algorithms, that is, to measure the distance or difference between a data point and its cluster center. It can be used in unsupervised or semi-supervised learning scenarios, where the labels of data points are unknown or only partially known. The goal of cluster loss is to make data points within the same cluster as close as possible, while data points in different clusters are as far apart as possible.

[0078] Furthermore, prior consistency constraint loss is often used in semi-supervised learning or transfer learning scenarios, where there is prior knowledge or assumptions about the data distribution. This loss function aims to ensure that the model's output is consistent with this prior knowledge. For example, in image classification tasks, if the similarity relationship between certain categories is known (e.g., cats and dogs are more similar than cats and cars), this prior knowledge can be incorporated into the loss function to constrain the model's output.

[0079] Understandably, in order to further improve the accuracy and generalization ability of the model, in step S202, contrastive clustering loss and prior consistency constraint loss are introduced to fine-tune the deep neural network encoder and the best cluster, so as to strengthen the correlation between normal event features and the best normal cluster, while also ensuring that abnormal event features can be accurately classified into the best abnormal cluster.

[0080] Further, step S202 may also include: selecting the normal event feature with the lowest anomaly score as the normal candidate feature; increasing the similarity between the normal candidate feature and the best normal cluster based on the contrastive clustering loss, and decreasing the similarity between the normal candidate feature and the best anomaly cluster to obtain a first clustering optimization result; decreasing the similarity between the anomaly candidate feature and the best normal cluster based on the contrastive clustering loss, and increasing the similarity between the anomaly candidate feature and the best anomaly cluster to obtain a second clustering optimization result; optimizing the deep neural network encoder and the preset prior distribution according to the prior consistency constraint loss to obtain the encoder optimization result and the prior distribution optimization result; and obtaining the optimization result based on the first clustering optimization result, the second clustering optimization result, the encoder optimization result, and the prior distribution optimization result.

[0081] Understandably, features with the lowest anomalous scores are selected from all normal event features. These features best match the expectation of normal events and are therefore chosen as normal candidate features. Next, contrastive clustering loss is used to enhance the similarity between normal candidate features and the best normal cluster, while weakening the association with the best anomalous cluster. This ensures that normal event features are more tightly clustered around the best normal cluster in the feature space, thereby enhancing the model's ability to identify normal events. Simultaneously, a similar strategy is adopted for anomalous candidate features, but in the opposite direction: the similarity between anomalous candidate features and the best normal cluster is reduced, while the association with the best anomalous cluster is increased, ensuring that anomalous event features can be accurately identified and classified into the best anomalous cluster.

[0082] Furthermore, a prior consistency constraint loss is introduced to further optimize the deep neural network encoder and the preset prior distribution. This ensures that the model can not only fully utilize the current data information during the optimization process but also maintain consistency with prior knowledge, thereby avoiding overfitting and getting trapped in local optima. Finally, the results of all the above optimization steps, including the first cluster optimization result, the second cluster optimization result, the encoder optimization result, and the prior distribution optimization result, are combined to obtain the final optimization result.

[0083] Step S203: When the optimization result satisfies the convergence condition, obtain the optimized cluster, the updated deep neural network encoder, and the prior distribution.

[0084] Understandably, when the optimization process reaches the preset convergence condition, the change in the clustering result is less than a preset threshold or the maximum number of iterations is reached, thus obtaining the optimized cluster. During the iterative optimization process, the deep neural network encoder can adjust its weights and biases according to the input data and the corresponding clustering labels in order to better capture the inherent structure and patterns of the data. When the optimization process converges, a fully trained encoder is obtained, i.e., the updated deep neural network encoder. The prior distribution, as a constraint or regularization term, affects the clustering result and the training of the encoder. When the optimization process converges, the obtained prior distribution is the version that best matches the current data and clustering result.

[0085] For ease of understanding, please refer to Figure 3 This explanation is provided, but does not limit the scope of this application. Figure 3 This is a schematic diagram of the Expectation-Maximization (EM) algorithm of this application. In the E-step of the Expectation-Maximization algorithm, the deep neural network encoder (hereinafter referred to as the "encoder") remains frozen and extracts features from the video. Then, a prior-guided clustering search mechanism is applied to determine clusters of features of normal / abnormal events, and the prior distribution is subsequently updated. In the M-step of the Expectation-Maximization algorithm, the encoder is updated using an anomaly-aware clustering method based on prior consistency. This method maximizes the log-likelihood of the identified clusters and maps abnormal events to low-probability regions of the prior distribution, while ensuring prior consistency.

[0086] exist Figure 3 In the first step, the encoder E(·) is applied to the normal event / abnormal event V. n and V a Mapping to the feature space yields video features X. n =E(V) n ) and X a =E(V) a ), where X n and X a All values ​​were normalized using the Euclidean norm (i.e., L2 norm). Under the Gaussian assumption, the mean parameters of each cluster were determined in the feature space using the expectation-maximization algorithm. (N is the number of clusters, M is the number of anomaly categories) and the prior distribution mean parameter μ0, and iteratively update the encoder E(·).

[0087] In the E-step, this application develops a cluster search method inspired by the Dirichlet process, namely a prior-guided cluster search mechanism. This mechanism dynamically determines the number of possible clusters in the dataset during training and uses these cluster parameters to determine the prior distribution, overcoming the limitation of existing technologies that require hyperparameters to specify the number of events. The specific algorithm for the prior-guided cluster search mechanism can be:

[0088]

[0089] in, and It has the following forms:

[0090]

[0091]

[0092]

[0093] In the formula, n j n is the number of elements in the j-th cluster; -i,j It is the j-th cluster that is removed The number of elements after; It refers to the feature vector obtained after the encoder extracts the features of the i-th slice in a normal event video; It refers to the cluster center of the j-th normal cluster; This is a metric proposed in this application, used to describe the distance of a video slice feature vector relative to a principal component cluster; This is also a metric proposed in this application, used to indicate the distance of the video slice feature vector relative to the prior distribution center; α is a pre-set hyperparameter; x 1 This refers to any eigenvector within the cluster. Furthermore, for the j-th anomalous cluster... Utilizing the features of each anomalous video with anomaly label j The K video features with the highest D0 (representing the low probability density of the feature in the prior) are selected as anomaly candidates and added to queue Q. j In the middle. After traversal, abnormal clusters are found. Through Q j The mean of the data is updated to estimate anomalous clusters, which enhances the subsequent learning of the prior distribution and helps to improve the distinction between normal and anomalous events.

[0094] In the M-step, the main task is to maximize the likelihood of the clustering parameters while ensuring prior consistency. The encoder E(·) is updated through a novel prior consistent anomaly-aware clustering method, which includes two losses: anomaly-aware contrastive clustering and prior consistency constraint.

[0095] In the anomaly perception contrastive clustering loss step: since the anomaly fragments do not have frame-level labels provided, the first step is to select... The segment features with the highest potential (i.e., the features most likely to correspond to the anomalous segments, which are mainly measured by the anomalous score in this application) are selected as anomalous candidates. Similar to the E-step, the K video features with the highest D0 are selected as anomalous candidates. Simultaneously, K video features with the lowest D0 are selected as normal candidates. For normal video features x n and normal candidates By comparing the clustering loss ProtoNCE, the similarity between the candidate cluster and normal clusters is increased, while the similarity between the candidate cluster and anomalous clusters is decreased. For anomalous candidates... By comparing cluster loss to reduce the similarity between the sample and normal clusters, and increasing the similarity between the sample and anomalous clusters, the anomaly center initialization and anomaly perception comparative clustering in the E-step successfully filtered out anomalous segments and increased their difference from normal samples in videos with only video-level annotations, by leveraging the relationship between samples and prior distributions.

[0096] In the prior consistency constraint loss stage, the similarity between normal samples relative to the unified prior μ0 is enhanced by comparing the cluster loss, which helps the encoder capture global semantics and reduces the model's tendency to overfit to specific clusters.

[0097] Understandably, the prior distribution learning bias problem can be addressed through a prior-consistent anomaly-aware clustering methods: by maximizing the likelihood of identified clusters and minimizing the likelihood of anomalies in the prior distribution, the separation between normal and anomalous instances is enhanced, while global semantics are captured through prior consistency constraints. This process allows us to learn an unbiased prior distribution that satisfies the condition that normal event types reside in their high-density regions while anomalous event types reside in their low-density regions, thus overcoming boundary bias.

[0098] This embodiment obtains optimal clusters based on the expectation-maximization algorithm, according to the relative relationship between video features and the initial prior distribution, and the relative relationship between video features and the initial clusters. The optimal clusters include the optimal normal clusters and the optimal abnormal clusters. Based on normal event features, abnormal event features, the optimal clusters, and the preset prior distribution, the deep neural network encoder and the optimal clusters are optimized according to the comparison cluster loss and the prior consistency constraint loss, obtaining the optimization results. When the optimization results meet the convergence condition, the optimized clusters, the updated deep neural network encoder, and the prior distribution are obtained, thereby accurately distinguishing between normal clusters and abnormal clusters, ensuring high-quality clusters and encoders after convergence, and improving the accuracy of anomaly detection.

[0099] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 Step S30 may include steps S301 to S303:

[0100] Step S301: Use the updated deep neural network encoder to extract the feature vector of the video.

[0101] Understandably, a trained and updated deep neural network encoder is used to process video. After the update, the deep neural network encoder can receive video as input and extract key features from the video through its internal multi-layer neural network structure. These features are encoded into vectors in high-dimensional space, called feature vectors, which can capture visual information such as motion, color, and texture in the video.

[0102] Step S302: Based on the optimized cluster, calculate the distance between the feature vector and the prior distribution to obtain the anomaly score.

[0103] The feature vectors of the video are evaluated using the previously optimized clustering results (i.e., optimized clusters) and the prior distribution. First, the feature vectors are compared with the cluster centers, and the distance between each cluster center is calculated. Then, the difference between the feature vectors and the normal distribution is further evaluated by combining the prior distribution. This difference is quantified as an anomaly score, which is used to measure the degree of anomaly in the video.

[0104] Step S303: Obtain the abnormal events in the video based on the abnormal score.

[0105] The abnormality scores obtained from the above steps are used to identify abnormal events in the videos. First, a threshold can be set to distinguish between normal and abnormal videos. The choice of threshold may be based on the statistical characteristics of the training data, domain knowledge, or experimental validation. Then, the abnormality score of each video is compared with the threshold. If the abnormality score is higher than the threshold, the video is considered to contain an abnormal event; otherwise, it is considered normal.

[0106] For example, in an open-set scenario, the anomalies in the input may not conform to the distribution of any anomaly cluster. In this case, we detect anomalies by directly measuring the degree of violation between the input video and the prior distribution, i.e., the distance between the input and the prior, and thus obtain the following anomaly score:

[0107] score open (x t ) = 1 - x t ·μ0 (4)

[0108] In the formula, x t This refers to the feature vector corresponding to the test video; μ0 is the prior distribution center; score open It refers to x t The distance relative to μ0 represents the anomaly score of the video when performing open set anomaly detection.

[0109] It should be noted that this application achieved state-of-the-art results in anomaly detection experiments under an open-world setting. Performance testing was conducted on the UCF-Crime anomaly detection dataset. The test and training sets of this dataset contain the same 13 classes of anomalies. Some anomalies were removed from the training dataset so that some anomalies in the test set were invisible to the model, thus constructing an anomaly detection scenario in an open world. The anomaly detection results of this application's method under the open-set scenario setting on the UCF-Crime dataset are shown in Table 1 below. The first row represents the number of visible anomaly classes in the training set. As can be seen from the performance comparison in the table, the method of this application achieves high performance under any visible anomaly setting, far superior to other methods in this field. Furthermore, compared to methods using K-means for prior distribution estimation, the method of this application also achieves a significant advantage, effectively demonstrating the effectiveness of the anomaly detection method of this application.

[0110] Table 1. Test results of abnormal event detection methods

[0111]

[0112]

[0113] This embodiment extracts feature vectors from the video using an updated deep neural network encoder, calculates the distance between the feature vectors and the prior distribution based on the optimized clustering, obtains anomaly scores, and identifies anomalous events in the video based on the anomaly scores. Thus, it can accurately extract video segment features using the optimized deep neural network encoder and calculate anomaly scores by combining the distance between the optimized clustering and the prior distribution, achieving efficient and accurate detection of anomalous events in video segments.

[0114] In the third embodiment, step S10 may include steps S101 to S103:

[0115] Step S101: Based on a deep neural network encoder, initial video features are obtained by mapping event samples to the feature space.

[0116] Step S102: Normalize the initial video features using the Euclidean norm to obtain video features.

[0117] It should be noted that the Euclidean norm is the standard method for measuring the length of a vector, and its calculation formula is the square root of the sum of the squares of the vector's elements; normalization is the process of dividing each element of the initial video features by its Euclidean norm, thereby obtaining a feature vector with a unit norm.

[0118] In this step, the initial video features obtained in the previous step are normalized. Normalization is a commonly used data preprocessing technique that scales the norm (or length) of a feature vector to a specific range, typically between 0 and 1 or with a unit norm. The Euclidean norm is used to measure the length of the feature vector, and normalization is performed accordingly. The normalized feature vector is the video feature.

[0119] Step S103: Initial clusters are obtained by performing preliminary clustering on the video features.

[0120] It should be noted that clustering is an unsupervised learning method that can group similar samples into one class, thereby revealing the inherent structure and distribution of the data.

[0121] Understandably, preliminary clustering of normalized video features can be performed using clustering algorithms (such as K-means, DBSCAN, etc.) to obtain preliminary clustering results, which are the initial clusters. These initial clusters represent the natural grouping of video features in the feature space.

[0122] This embodiment obtains initial video features by mapping event samples to a feature space based on a deep neural network encoder. The initial video features are then normalized using the Euclidean norm to obtain video features. Initial clusters are obtained by performing preliminary clustering on the video features. This enables the efficient extraction of features from video event samples using a deep neural network encoder. The normalization process enhances feature stability, and the preliminary clustering step lays the foundation for subsequent optimization, thereby improving the overall accuracy and efficiency of abnormal event detection.

[0123] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the abnormal event detection method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0124] This application also provides an abnormal event detection device, please refer to... Figure 5 The abnormal event detection device includes:

[0125] Feature extraction module 10 is used to obtain video features and initial clusters by mapping video event samples to feature space based on a deep neural network encoder;

[0126] The parameter iteration module 20 is used to obtain the optimized cluster, the updated deep neural network encoder, and the prior distribution based on the video features and the initial cluster.

[0127] Anomaly detection module 30 is used to detect anomalous events in the video based on the optimized clustering, the updated deep neural network encoder, and the prior distribution.

[0128] The abnormal event detection device provided in this application, employing the abnormal event detection method in the above embodiments, can solve the technical problem that existing abnormal event detection methods are prone to detection deviations. Compared with the prior art, the beneficial effects of the abnormal event detection device provided in this application are the same as those of the abnormal event detection method provided in the above embodiments, and other technical features in the abnormal event detection device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0129] This application provides an abnormal event detection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the abnormal event detection method in Embodiment 1 above.

[0130] The following is for reference. Figure 6 The diagram illustrates a structural schematic suitable for implementing an anomaly detection device according to embodiments of this application. The anomaly detection device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Detection), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The abnormal event detection device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0131] like Figure 6As shown, the abnormal event detection device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the abnormal event detection device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the anomaly detection device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show anomaly detection devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0132] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0133] The abnormal event detection device provided in this application, employing the abnormal event detection method in the above embodiments, can solve the technical problem that existing abnormal event detection methods are prone to detection deviations. Compared with the prior art, the beneficial effects of the abnormal event detection device provided in this application are the same as those of the abnormal event detection method provided in the above embodiments, and other technical features of this abnormal event detection device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0134] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0135] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0136] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the abnormal event detection method in the above embodiments.

[0137] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0138] The aforementioned computer-readable storage medium may be included in the anomaly detection device; or it may exist independently and not be assembled into the anomaly detection device.

[0139] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the abnormal event detection device, enable the abnormal event detection device to: obtain video features and initial clusters by mapping video event samples to a feature space based on a deep neural network encoder; obtain optimized clusters, updated deep neural network encoders, and prior distributions based on the expectation-maximization algorithm according to the video features and initial clusters; and detect abnormal events in the video based on the optimized clusters, updated deep neural network encoders, and prior distributions.

[0140] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0142] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0143] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described abnormal event detection method, thereby solving the technical problem that existing abnormal event detection methods are prone to detection bias. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the abnormal event detection method provided in the above embodiments, and will not be repeated here.

[0144] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described abnormal event detection method.

[0145] The computer program product provided in this application can solve the technical problem that existing abnormal event detection methods are prone to detection bias. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the abnormal event detection methods provided in the above embodiments, and will not be repeated here.

[0146] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An abnormal event detection method, characterized in that, The method includes the following steps: Based on a deep neural network encoder, video features and initial clusters are obtained by mapping video event samples to a feature space. The video features include normal event features and abnormal event features. Based on the expectation-maximization algorithm, the optimized cluster, the updated deep neural network encoder, and the prior distribution are obtained according to the video features and the initial cluster. Anomalies in the video are detected based on the optimized clustering, the updated deep neural network encoder, and the prior distribution. The steps of obtaining the optimized cluster, the updated deep neural network encoder, and the prior distribution based on the expectation-maximization algorithm, according to the video features and the initial cluster, include: Based on the expectation-maximization algorithm, combined with the prior-guided cluster search mechanism, the relative relationship between the video features and the initial prior distribution, and the relative relationship between the video features and the initial cluster are determined to obtain the optimal cluster, which includes the optimal normal cluster and the optimal abnormal cluster. Based on normal event features, abnormal event features, the optimal cluster, and the preset prior distribution, the deep neural network encoder and the optimal cluster are optimized according to the comparison clustering loss and the prior consistency constraint loss to obtain the optimization result. When the optimization results satisfy the convergence condition, the optimized cluster, the updated deep neural network encoder, and the prior distribution are obtained.

2. The abnormal event detection method as described in claim 1, characterized in that, The step of determining the relative relationship between the video features and the initial prior distribution, and the relative relationship between the video features and the initial cluster, based on the expectation-maximization algorithm and a prior-guided cluster search mechanism, to obtain the optimal cluster, includes: Based on the expectation-maximization algorithm, combined with the prior-guided cluster search mechanism, the relative relationship between the video features and the initial prior distribution, and the relative relationship between the video features and the initial cluster are determined to obtain the judgment result; Based on the judgment result, if the metric of the normal event feature and the nearest initial cluster center is less than the metric of the normal event feature and the initial prior distribution center, then the normal event feature is assigned to the initial cluster, and the initial cluster is taken as the best normal cluster; If the metric of the normal event feature and the initial prior distribution center is less than the metric of the initial cluster center at any distance, then a new cluster is established for the normal event feature, and the new cluster is taken as the optimal normal cluster. The abnormal event feature with the highest abnormality score is selected as the abnormality candidate feature, and the optimal abnormal cluster is obtained based on the abnormality candidate feature. Based on the optimal normal cluster and the optimal abnormal cluster, the optimal cluster corresponding to the video feature is obtained.

3. The abnormal event detection method as described in claim 2, characterized in that, The step of optimizing the deep neural network encoder and the optimal cluster based on normal event features, abnormal event features, the optimal cluster, and a preset prior distribution, and by comparing cluster loss and prior consistency constraint loss to obtain the optimization result, includes: Select normal event features with the lowest abnormality scores as normal candidate features; Based on the contrastive clustering loss, the similarity between the normal candidate features and the best normal cluster is increased, and the similarity between the normal candidate features and the best abnormal cluster is decreased, to obtain the first cluster optimization result; Based on the comparative clustering loss, the similarity between the abnormal candidate features and the best normal cluster is reduced, and the similarity between the abnormal candidate features and the best abnormal cluster is increased to obtain the second clustering optimization result; The deep neural network encoder and the preset prior distribution are optimized based on the prior consistency constraint loss to obtain the encoder optimization result and the prior distribution optimization result. Based on the first cluster optimization result, the second cluster optimization result, the encoder optimization result, and the prior distribution optimization result, the optimization result is obtained.

4. The abnormal event detection method according to any one of claims 1 to 3, characterized in that, The step of detecting anomalous events in the video based on the optimized clustering, the updated deep neural network encoder, and the prior distribution includes: The updated deep neural network encoder is used to extract feature vectors from the video. Based on the optimized clustering, the distance between the feature vector and the prior distribution is calculated to obtain the anomaly score; The abnormal events in the video are obtained based on the abnormality score.

5. The abnormal event detection method according to any one of claims 1 to 3, characterized in that, The step of obtaining video features and initial clusters by mapping video event samples to a feature space based on a deep neural network encoder includes: Based on a deep neural network encoder, initial video features are obtained by mapping event samples to a feature space; The initial video features are normalized using the Euclidean norm to obtain the video features. Initial clusters are obtained by performing preliminary clustering on the video features.

6. An abnormal event detection device, characterized in that, The abnormal event detection device includes: The feature extraction module is used to obtain video features and initial clusters by mapping video event samples to a feature space based on a deep neural network encoder. The video features include normal event features and abnormal event features. The parameter iteration module is used to obtain the optimized cluster, the updated deep neural network encoder, and the prior distribution based on the expectation-maximization algorithm, according to the video features and the initial cluster. An anomaly detection module is used to detect anomalous events in the video based on the optimized clustering, the updated deep neural network encoder, and the prior distribution. The parameter iteration module is also used to determine the relative relationship between the video features and the initial prior distribution, and the relative relationship between the video features and the initial cluster, based on the expectation-maximization algorithm and the prior-guided cluster search mechanism, to obtain the optimal cluster, which includes the optimal normal cluster and the optimal abnormal cluster. The parameter iteration module is further configured to optimize the deep neural network encoder and the optimal cluster based on normal event features, abnormal event features, the optimal cluster and the preset prior distribution, according to the comparison cluster loss and the prior consistency constraint loss, to obtain the optimization result; when the optimization result satisfies the convergence condition, the optimized cluster, the updated deep neural network encoder and the prior distribution are obtained.

7. An abnormal event detection device, characterized in that, The abnormal event detection device includes: a memory, a processor, and an abnormal event detection program stored in the memory and executable on the processor. When the abnormal event detection program is executed by the processor, it implements the abnormal event detection method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores an abnormal event detection program, which, when executed by a processor, implements the abnormal event detection method as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, The computer program product includes an anomaly detection program, which, when executed by a processor, implements the anomaly detection method as described in any one of claims 1 to 5.

Images