A video anomaly detection method and system

Abstract

The present application belongs to the technical field of computer vision, video anomaly detection and video anomaly detection model design, and particularly relates to a video anomaly detection method, comprising: acquiring segment feature data of a video to be detected; inputting the segment feature data into a video anomaly detection model to acquire a detection result, wherein the video anomaly detection model comprises: an anomaly feature enhancement model, an anomaly score calculation model, an anomaly positioning model based on classification guidance and an anomaly score optimization model. The present application enhances the feature data of the video to be detected by the anomaly feature enhancement model in terms of time sequence information and motion information, so that it is more suitable for the anomaly detection task, and according to the anomaly positioning model based on classification guidance, parameters in the video anomaly process classification are mined and the anomaly score obtained by the anomaly score calculation model is optimized, so that a detection result with higher accuracy is obtained. Meanwhile, the anomaly positioning model based on classification guidance is convenient to use and can be installed in a memory for use as plug as needed.

Description

Technical Field

[0001] This invention belongs to the technical fields of computer vision, video anomaly detection, and video anomaly detection model design, and specifically relates to a video anomaly detection method and system. Background Technology

[0002] Surveillance cameras are ubiquitous in cities. As an important part of the urban security system, video surveillance exists to record everything that happens in the city. Anomaly detection based on surveillance video can automatically detect and judge abnormal events from these videos, which helps to assist public security and traffic police officers in law enforcement and is an important means to improve the level of public safety.

[0003] Deep learning-based video anomaly detection algorithms leverage the powerful learning capabilities of deep learning to learn behavioral patterns and scene information from both normal and abnormal videos, and then use this information to identify anomalies in the videos. Strongly supervised video anomaly detection methods require video segment-level labels in the training dataset, which incurs significant annotation costs. Weakly supervised methods, on the other hand, only require video-level labels in the dataset. While their detection efficiency is slightly lower compared to strongly supervised methods, they significantly reduce the cost of video annotation. Therefore, weakly supervised anomaly detection methods are more applicable in real-world scenarios.

[0004] Most existing weakly supervised video anomaly detection methods rely on multi-instance learning, which involves dividing a video into several non-overlapping segments, treating each segment as an instance, and forming a packet with all instances within the video. The packets are then categorized into positive and negative packets based on whether they contain anomalous instances (positive packets contain anomalous instances, while negative packets contain only normal instances). Each instance in a packet is then scored and ranked for anomaly detection, and the highest-scoring instances from both the positive and negative packets are selected. A loss function is designed to maximize the difference between these scores.

[0005] Currently, video anomaly detection methods based on weak supervision still have some shortcomings:

[0006] On the one hand, existing models all use 3D feature extraction models pre-trained on action recognition datasets, such as I3D and C3D, to perform anomaly detection tasks based on the extracted features. Although these models can extract spatiotemporal features of videos very well, the extracted features are general features of videos and do not consider the specific features required for "anomaly detection". Therefore, there is still room for optimization for the extracted features.

[0007] On the other hand, since there are only video-level labels, there is no effective guidance at the video segment level. Furthermore, only a small portion of the video segments participate in the model's optimization process during each training iteration, with the remaining video segment features merely serving for score ranking and filtering, resulting in extremely low feature utilization efficiency. These problems lead to the model's inability to learn all abnormal behavior patterns on large datasets, resulting in inaccurate anomaly scores. When using this model for video anomaly detection, the distribution of anomaly scores deviates significantly from reality, easily leading to missed detections, false positives, and inaccurate anomaly localization. Summary of the Invention

[0008] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a video anomaly detection method that can enhance the general features of video extraction and optimize the anomaly detection results of video.

[0009] To achieve the above and other related objectives, the present invention provides a video anomaly detection method, comprising: acquiring segment feature data of a video to be detected; inputting the segment feature data into a video anomaly detection model to obtain a detection result, wherein the video anomaly detection model comprises: an anomaly feature enhancement model, which enhances the segment feature data with temporal and action information to obtain segment enhanced feature data; an anomaly score calculation model, which calculates an initial segment anomaly score based on the segment enhanced feature data; a classification-guided anomaly localization model, which calculates a segment anomaly probability score and a segment localization guidance score based on the segment enhanced feature data; an anomaly score optimization model, which calculates a final anomaly score of the video to be detected based on the initial segment anomaly score and the segment anomaly probability score; and optimizing the initial segment anomaly score using the segment localization guidance score to obtain a final segment anomaly score; wherein the final anomaly score of the video to be detected and the final segment anomaly score are the detection result.

[0010] According to a specific embodiment of the present invention, the anomaly feature enhancement model includes a long-range temporal feature enhancement module and a short-range action feature enhancement module. The step of enhancing the segment feature data with temporal and action information to obtain segment enhanced feature data includes: reducing the dimensionality of the segment feature data using a first fully connected neural network and a second fully connected neural network to obtain first-dimensionalized segment feature data and second-dimensionalized segment feature data with the same dimension; wherein the first fully connected neural network and the second fully connected neural network have the same model structure but different parameter settings; inputting the first-dimensionalized segment feature data and the second-dimensionalized segment feature data into a long-range temporal feature model to obtain segment full-time feature data; inputting the segment full-time feature data into a short-range action feature model to obtain segment action attention feature data; and integrating the segment feature data and the segment action attention feature data to obtain segment enhanced feature data.

[0011] According to a specific embodiment of the present invention, the step of inputting the first dimensionality-reduced fragment feature data and the second dimensionality-reduced fragment feature data into a long-range temporal feature model to obtain fragment full-time feature data includes: inputting the first dimensionality-reduced feature data into an LSTM neural network of the long-range temporal feature model to obtain fused feature data; inputting the second dimensionality-reduced feature data into a nonlocal neural network of the long-range temporal feature model and performing residual processing on the output result to obtain global feature data; wherein the nonlocal neural network is composed of four one-dimensional convolutions, and the convolutional characteristics of the four one-dimensional convolutions are completely consistent; concatenating the fused feature data and the global feature data to obtain fragment full-time feature data; wherein the size of the fragment full-time feature data is consistent with the size of the fragment feature data.

[0012] According to a specific embodiment of the present invention, the step of inputting the full-time feature data of the segment into a short-range motion feature model to obtain segment motion attention feature data includes: averaging the full-time feature data of the segment along the time dimension; inputting the averaged full-time feature data of the segment into a first fully connected neural network of the short-range motion feature model for dimensionality reduction, and then inputting the dimensionality-reduced full-time feature data of the segment into a second fully connected neural network of the short-range motion feature model for dimensionality increase to obtain motion attention weights; multiplying the motion attention weights by the full-time feature data of the segment in a broadcast manner, and adding the multiplication result to the segment feature data to obtain the segment motion attention feature data; wherein the segment motion attention feature data has the same size as the segment feature data.

[0013] According to a specific embodiment of the present invention, the step of calculating the initial fragment anomaly score based on the fragment enhancement feature data includes: inputting the fragment enhancement feature data into the first fully connected neural network of the anomaly score calculation model for a first dimensionality reduction, and calculating the dimensionality-reduced fragment enhancement feature data using the ReLU function and the Dropout function to obtain first intermediate data; inputting the first intermediate data into the second fully connected neural network of the anomaly score calculation model for a second dimensionality reduction, and calculating the dimensionality-reduced first intermediate data using the ReLU function and the Dropout function to obtain second intermediate data; inputting the second intermediate data into the third fully connected neural network of the anomaly score calculation model for a third dimensionality reduction, and calculating the dimensionality-reduced second intermediate data using the Sigmoid function to obtain the initial fragment anomaly score with dimension 1.

[0014] According to a specific embodiment of the present invention, the steps of calculating the segment anomaly probability score and the segment localization guidance score based on the segment enhancement feature data include: transposing the segment enhancement feature data to obtain transposed segment feature data; inputting the transposed segment feature data into the first one-dimensional convolution of the classification-guided anomaly localization model for a first dimensionality reduction, and calculating the dimensionality-reduced transposed segment feature data using the ReLU function to obtain third intermediate data; inputting the third intermediate data into the second one-dimensional convolution of the classification-guided anomaly localization model for a second dimensionality reduction, and calculating the dimensionality-reduced third intermediate data using the ReLU function to obtain fourth intermediate data; inputting the fourth intermediate data into the third one-dimensional convolution of the classification-guided anomaly localization model for a third dimensionality reduction to obtain shared parameters of dimension 1; calculating the shared parameters using the Softmax function to obtain the segment anomaly probability score; and calculating the shared parameters using the Sigmoid function to obtain the segment localization guidance score.

[0015] According to a specific embodiment of the present invention, the step of calculating the final abnormal score of the video to be detected based on the initial segment abnormal score and the segment abnormal probability score includes: calculating the product of the segment abnormal probability score and the initial segment abnormal score, and performing linear accumulation to obtain the final abnormal score of the video to be detected.

[0016] According to a specific embodiment of the present invention, the step of optimizing the initial segment anomaly score using the segment localization guidance score to obtain the final segment anomaly score includes: normalizing the score based on the mean of the segment localization guidance score to obtain an optimized score; calculating the Hadamard product of the initial segment anomaly score and the optimized score, and processing the calculation result using the Sigmoid function to obtain the final segment anomaly score.

[0017] According to a specific embodiment of the present invention, the training steps of the video anomaly detection model include: obtaining video-level labels for a training video dataset; extracting features from each training video to obtain segment features; feeding the segments into the video anomaly detection model to obtain video anomaly scores and video segment anomaly scores for the training videos; calculating weakly supervised multi-instance loss and temporal smoothness loss based on the video segment anomaly scores; calculating binary cross-entropy loss based on the video anomaly scores; using the sum of the weakly supervised multi-instance loss, temporal smoothness loss, and binary cross-entropy loss as the final loss; and performing backpropagation on the video anomaly detection model based on the final loss.

[0018] A video anomaly detection system includes: a video feature extraction module for acquiring segment feature data of a video to be detected; and a video anomaly detection module for acquiring detection results of the segment feature data. The video anomaly detection module further includes: an anomaly feature enhancement module for enhancing the segment feature data with temporal and action information to obtain enhanced segment feature data; an anomaly score calculation module for calculating an initial segment anomaly score based on the enhanced segment feature data; a classification-guided anomaly localization module for calculating a segment anomaly probability score and a segment localization guidance score based on the enhanced segment feature data; and an anomaly score optimization module for calculating a final anomaly score of the video to be detected based on the initial segment anomaly score and the segment anomaly probability score; and optimizing the initial segment anomaly score using the segment localization guidance score to obtain a final segment anomaly score. The final anomaly score of the video to be detected and the final segment anomaly score are the detection results.

[0019] The technical advantage of this invention lies in enhancing the temporal and action information of the feature data of the video to be detected through an anomaly feature enhancement model, making it more suitable for anomaly detection tasks and improving the utilization rate of feature data in the video. Simultaneously, it provides a classification-guided anomaly localization model that leverages the anomaly localization capabilities of parameters generated during the classification process to optimize the detection results. Furthermore, this classification-guided anomaly localization model can be used in conjunction with any anomaly detection model, improving detection performance, and is easy to use; it can be stored in memory for plug-and-play operation. Further, the video anomaly detection model of this embodiment is trained based on three loss functions, further improving the model's accuracy. Attached Figure Description

[0020] Figure 1 This is a structural block diagram of a specific embodiment of a video anomaly detection method provided by the present invention;

[0021] Figure 2 This is a flowchart illustrating a specific embodiment of a video anomaly detection method provided by the present invention.

[0022] Figure 3 A flowchart illustrating a specific embodiment of the anomaly feature enhancement model provided by the present invention;

[0023] Figure 4 A flowchart illustrating a specific embodiment of the anomaly score calculation model and the anomaly localization model based on classification guidance provided by this invention;

[0024] Figure 5 A flowchart illustrating a specific embodiment of the anomaly score optimization model provided by the present invention;

[0025] Figure 6 This is a flowchart illustrating a specific embodiment of a video anomaly detection system provided by the present invention.

[0026] Figure 7 This is a schematic diagram of a specific embodiment of a computer device provided by the present invention;

[0027] Figure 8 This is a schematic diagram of another specific embodiment of a computer device provided by the present invention. Detailed Implementation

[0028] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0029] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the illustrations only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0030] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0031] First, it should be noted that this application provides a video anomaly detection model based on weak supervision, which labels the video through a weakly supervised anomaly score calculation model. It also adds two new models: an anomaly feature enhancement model and an anomaly localization model based on classification guidance.

[0032] The anomaly feature model processes the feature data of the video to be detected by combining long-distance temporal feature enhancement and short-distance action feature enhancement. This enhances the temporal features by integrating features from all previous and global time dimensions, and enhances the action features by integrating features from nearby time dimensions. This results in the original feature data being represented in a way that better suits the anomaly detection task, improving the utilization rate of feature data in the video.

[0033] The classification-guided anomaly localization model is a plug-and-play branch module applicable to any weakly supervised anomaly detection model. The anomaly score is optimized by acquiring the anomaly probability score and localization guidance score from the feature data of the video to be detected. Simultaneously, a classification loss function is used to perform partially supervised training on the video anomaly detection model, providing guidance and optimizing the model.

[0034] Example 1

[0035] Please see Figure 1-2As shown, a video anomaly detection method includes:

[0036] Step S10: Obtain the segment feature data of the video to be detected.

[0037] In this approach, a video is divided into n segments, each called a fragment. Since the location of anomalies needs to be identified, the anomaly score for each fragment is required; fragments with high anomaly scores are identified as anomalies. However, in weakly supervised video anomaly detection, we only have the anomaly score label for the entire video, not the anomaly score label for each individual video fragment.

[0038] A pre-trained 3D convolutional neural network is used as a feature extractor to first obtain feature data for each segment of the video to be detected. Typically, the extracted feature data for each segment is input into an anomaly score calculation model to obtain the anomaly score for each segment of the video. However, due to the differences between action recognition and anomaly detection tasks, the feature data is suboptimal when processed for anomaly detection. Therefore, it is necessary to enhance the original feature dataset to better match the characteristics of the anomaly detection task.

[0039] The video to be detected is subjected to feature extraction by a pre-trained 3D convolutional neural network to obtain a matrix f1 of segment feature data, and the size of f1 is [t,d], where t represents the number of segments in the video to be detected and d is the dimension of each segment.

[0040] Step S20: Input the segment feature data into the video anomaly detection model to obtain the detection result.

[0041] The video anomaly detection model includes: an anomaly feature enhancement model, an anomaly score calculation model, an anomaly localization model based on classification guidance, and an anomaly score optimization model.

[0042] Specifically, in application, the anomaly feature enhancement model is used to enhance the temporal and action information of the fragment feature data to obtain enhanced fragment feature data. For example... Figure 3 As shown, f1 is first reduced in dimensionality using a first fully connected neural network Fc1 and a second fully connected neural network Fc2. To avoid the feature data learning to have too many features, the model parameters of Fc1 and Fc2 are set to be different, so as to obtain a matrix f2 of the first dimensionality-reduced feature data and a second dimensionality-reduced feature data f3 of the same dimension, both of which are [t, d / 2].

[0043] The anomaly feature enhancement model is composed of a long-range temporal feature enhancement module and a short-range action feature enhancement module connected in series. The long-range temporal feature enhancement module is further constructed in parallel using a long short-term memory network and a non-local neural network to enhance the original feature data over a long time period. The short-range action feature enhancement module generates action attention weights based on two fully connected neural networks, and uses these weights to enhance the original features over a short time period.

[0044] In one specific embodiment, since video information is represented as continuous video segments that remain continuous in time, these features can be represented as sequence features in the time dimension. The presence of a preceding element in the sequence affects the progression of subsequent elements, while subsequent elements are continuations of preceding elements.

[0045] like Figure 3 As shown, the data in f2 is expanded along a fragment dimension of size t to obtain {x1, x2…x}. t The data is then sequentially input into a Long Short-Term Memory (LSTM) network to ensure that the data corresponding to each segment in the matrix is ​​interconnected. The input and output dimensions of the LSM network are both set to d / 2. After processing by the LSM network, a set of interconnected data segments {y1, y2…y} is obtained. t}, and integrate them to obtain fused feature data f 2，1 .

[0046] The data in f3 is input into a one-dimensional convolution of a nonlocal neural network. Leveraging the advantages of nonlocal operations, the interaction between any two time points is calculated, regardless of their temporal distance, thereby enabling the features to capture global temporal information. The nonlocal neural network consists of four one-dimensional convolutions, all with a kernel size of 1, a stride of 1, zero padding, and d / 2 output channels.

[0047] First, f3 is processed by the first three one-dimensional convolutions to obtain three different data f of size [t, d / 2]. 3，1 f 3，2 and f 3，3 f 3，1 [t, d / 2] and the transpose of f 3，2 Matrix multiplication is performed on [d / 2, t] to generate global temporal attention weights f of size [t, t]. t Then the global temporal attention weight f t Applying to f via matrix multiplication 3，3 Optimization was performed to obtain data f with global time attention applied. 3，4[t, d / 2]. Through the fourth one-dimensional convolution pair f 3，4 Post-processing is performed to obtain data f. 3，5 [t, d / 2]. Simultaneously, residual operations are used to combine f3 and f... 3，5 Integrate the data to obtain global feature data f. 3，6 Finally, f 2，1 and f 3，6 Based on the feature dimension, the full-time feature data f4[t, d] of the segment is obtained by splicing.

[0048] The process involves stitching the data to restore its original size while simultaneously giving it greater attention across the temporal dimension. The acquired full-time feature data f4 aggregates rich information over a long time dimension. However, for video data, short-time features, namely motion information, are also crucial. These motion features are specifically manifested in dimension d of the video segment, meaning that dimension d contains important motion information. Therefore, motion attention weights are used to capture key motion information in the video.

[0049] In one specific embodiment, f4 is averaged over the time dimension to adjust the data size to [1, d]. This average is then input into the first fully connected neural network Fc-1 and the second fully connected neural network Fc-2 in the short-distance action feature enhancement model. Fc-1 is used for dimensionality reduction, reducing the data size to [1, d / 4], which is then fed into Fc-2 for dimensionality enhancement, ultimately generating the action attention weight f. 4，d [1, d].

[0050] The weight f is focused on through the aforementioned action. 4，d Optimize f4 for the full-time feature data of the segment, but due to the difference in dimensions, the two values ​​will be broadcast multiplied, i.e., f 4，d The value is copied t times and stacked along its first dimension to form a variant f of the same size as f4. 4，d [t, d], then multiply the two values ​​element-wise. This step is performed automatically in PyTorch by utilizing the properties of tensors.

[0051] In this way, action attention weights can be applied to the feature data of each segment, resulting in action-attention data f5[t, d].

[0052] Finally, to prevent damage to some useful original feature information, f1 and f5 are integrated to obtain the fragment action attention feature data f6[t, d].

[0053] It's important to note that in traditional weakly supervised anomaly detection, the feature data extracted and processed from the video is fed into an anomaly score calculation model to calculate anomaly scores at the video segment level. Then, the model is optimized by applying a multi-instance loss, which involves selecting the highest-scoring segment or a few segments from both normal and abnormal videos and widening the score gap between them to optimize the model. In this setup, only a very small portion of the video segments are used to optimize the model. This results in low feature utilization and, due to insufficient guidance information, significant room for improvement in the generated anomaly scores.

[0054] Therefore, this application provides a classification-guided anomaly localization model to optimize the initial segment anomaly score calculated by the anomaly score calculation model. On the one hand, it improves the utilization rate of features by generating video-level anomaly probability scores and using classification loss for constraint optimization. On the other hand, it optimizes the segment scores of the video by generating localization guidance scores.

[0055] Specifically, in application, similar to traditional weakly supervised anomaly score calculation models, the anomaly score calculation model described in this application embodiment can utilize the segment feature data of the video to be detected to calculate the anomaly score at the video segment level. For example... Figure 4 As shown, the anomaly score calculation model consists of three fully connected neural networks cascaded together to reduce the dimensionality of f6, successively reducing the feature dimension of the data to d / 4, d / 16, and 1, thus obtaining the initial fragment anomaly score. The fragment enhancement feature data is input into the first fully connected neural network of the anomaly score calculation model for the first dimensionality reduction. The dimensionality-reduced fragment enhancement feature data is then calculated using the ReLU function and a Dropout function with a dropout rate of 0.7 to obtain first intermediate data with a dimension of d / 4. This first intermediate data is then input into the second fully connected neural network of the anomaly score calculation model for the second dimensionality reduction. The dimensionality-reduced first intermediate data is then calculated using the ReLU function and a Dropout function with a dropout rate of 0.7 to obtain second intermediate data with a dimension of d / 16. Finally, the second intermediate data is input into the third fully connected neural network of the anomaly score calculation model for the third dimensionality reduction to obtain parameter p1. This p1 is then calculated using the Sigmoid function to obtain the initial fragment anomaly score S1[t, 1] with a dimension of 1.

[0056] In existing technologies, this score is used as the final anomaly score for a video segment. This score is a value between 0 and 1, and its magnitude reflects the degree of anomaly in the video. Combined with a set threshold, the video segment can be classified. This score is used to calculate the multi-instance loss, and optimization is continuously used to improve the model's ability to generate anomaly scores. This application optimizes this structure by using a localization-guided model based on classification guidance.

[0057] Specifically, in applications, such as Figure 4 As shown, the anomaly localization model based on classification guidance consists of three one-dimensional convolutional neural networks, Conv1, Conv2, and Conv3, connected in series. The transposed f7[d, t] of f6 is sequentially input into the one-dimensional convolutions. Each one-dimensional convolution performs a convolution operation on the time dimension, and then the data is dimensionality reduced, similarly reducing the feature dimensions to d / 4, d / 16, and 1 sequentially. Specifically, the kernel size of the first one-dimensional convolution is set to 3, with a stride of 1 and padding of 1; the kernel size of the second and third convolutions is set to 1, with a stride of 1 and padding of 0. The transposed fragment feature data is input into the first one-dimensional convolution of the classification-guided anomaly localization model for the first dimensionality reduction. The dimensionality-reduced transposed fragment feature data is then calculated using the ReLU function to obtain the third intermediate data. This third intermediate data is then input into the second one-dimensional convolution of the classification-guided anomaly localization model for the second dimensionality reduction. The dimensionality-reduced third intermediate data is then calculated using the ReLU function to obtain the fourth intermediate data. Finally, the fourth intermediate data is input into the third one-dimensional convolution of the classification-guided anomaly localization model for the third dimensionality reduction, resulting in a shared parameter p2 with dimension 1. The fragment anomaly probability score S2[t, 1] and the fragment localization guidance score S3[t, 1] are then obtained by calculating the Softmax and Sigmoid functions of the shared parameter p2. Based on the characteristics of the Softmax function, the sum of the fragment anomaly probability scores S2 for all videos to be detected is 1. Since the process of generating anomaly probability scores can provide guidance for anomaly localization, fragments with high anomaly probabilities are more likely to be anomalous fragments, while fragments with low anomaly probabilities are more likely to be normal fragments. Therefore, the Sigmoid function is used to process the shared parameter p2 to generate the fragment localization guidance score S3.

[0058] It should be noted that the anomaly localization model based on classification guidance is a combination of one-dimensional convolutions. Compared with using fully connected layers, one-dimensional convolutions not only reduce the amount of computation, but also, in the time dimension, can comprehensively consider the features of the convolution kernel size and time step to generate a score that is more in line with the characteristics of the task.

[0059] Specifically, in applications, such as Figure 5As shown, the detection results of the video to be detected are obtained through the following steps:

[0060] Since the anomaly probability score is the proportion of a segment in the anomaly score of the video to be detected, the final anomaly score S4[t, 1] of the video to be detected is obtained by calculating the product of the segment anomaly probability score and the initial segment anomaly score and then linearly summing them.

[0061]

[0062] in, The anomaly score for the i-th segment in the video to be detected. Let i be the anomaly probability score of the i-th segment in the video to be detected, where i∈[1,t].

[0063] Furthermore, S3 is composed of Composition, that is in, Let S1 be the localization guidance score for the i-th segment in the video to be detected. To enhance the optimization effect of S3, the following steps are taken in S3: After normalization, we get

[0064]

[0065] in, After combination, we obtain That is, the optimization score

[0066] The principle behind this formula is to update the parameters by using the average positioning score as a threshold. By dividing the scores, scores above the average are assigned a value greater than 1, and scores below the average are assigned a value less than 1, which is then used as the final optimized score.

[0067] Finally, the optimized score and the initial fragment anomaly score are combined using the Hadamard product to widen the score difference between the located abnormal fragment and other normal fragments, thereby optimizing the initial fragment anomaly score. The Sigmoid activation function is then used to limit the calculation result to between 0 and 1, yielding the final fragment anomaly score S5[t, 1].

[0068] Furthermore, weakly supervised datasets only provide video-level labels, while our goal is anomaly detection within video segments. Past weakly supervised models' multi-instance loss works by widening the score gap between the video segments with the highest anomaly scores in normal and abnormal videos, thus enabling the model to generate relatively accurate anomaly scores for video segments. This indirect and imprecise supervision method significantly limits model optimization. Therefore, we calculate the final anomaly scores of the training videos and the final segment anomaly scores, and combine these with the corresponding video labels to design an additional classification loss. This greatly leverages the role of video-level labels, providing the model with a direct and precise constraint.

[0069] This application also provides a training method for a video anomaly detection model, including: obtaining anomaly scores S of video segments from a training video. x Given video tag y, calculate its multi-instance loss L. mil :

[0070]

[0071] in, middle Let S be the final anomaly score of the j-th segment in the training video. The topk(Sx) operation means taking the final anomaly scores of the top k segments with the largest final anomaly scores in a training video. k is calculated by the number of segments t in the training video and the hyperparameter α, where α is 3.

[0072] To add additional supervisory information, a general time smoothness loss L was used. smooth To increase constraints,

[0073]

[0074] Simultaneously obtain the video anomaly score S of all training videos. y The number of training videos is N, and the video label y corresponds to each training video. n To calculate the binary classification cross-entropy loss L cls

[0075]

[0076] in, This is the video anomaly score for the nth training video.

[0077] Finally, the final loss L is obtained by integrating the obtained multi-instance loss, temporal smoothness loss, and binary classification cross-entropy loss. sum :

[0078] L sum =L cls +L mil+λL smooth

[0079] The hyperparameter λ is set to 0.001.

[0080] According to the final loss L sum The video anomaly detection model is backpropagated to optimize it.

[0081] It should be noted that the steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they contain the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0082] Example 2

[0083] Please see Figure 6 As shown, embodiments of this application also provide a video anomaly detection system, including:

[0084] The video feature extraction module 10 is used to obtain segment feature data of the video to be detected.

[0085] The video anomaly detection module 20 is used to obtain the detection results of the segment feature data.

[0086] The video anomaly detection module 20 includes:

[0087] The anomaly feature enhancement module 21 enhances the segment feature data with temporal and action information to obtain segment enhanced feature data.

[0088] The anomaly score calculation module 22 calculates the initial anomaly score of the segment based on the segment enhancement feature data.

[0089] The anomaly localization module 23, based on classification guidance, calculates the anomaly probability score and the localization guidance score of the segment based on the segment enhancement feature data.

[0090] The anomaly score optimization module 24 calculates the final anomaly score of the video to be detected based on the initial segment anomaly score and the segment anomaly probability score; it optimizes the initial segment anomaly score using the segment localization guidance score to obtain the final segment anomaly score; wherein the final anomaly score and the final segment anomaly score of the video to be detected are the detection results.

[0091] It should be noted that the video anomaly detection system provided in the above embodiments and the video anomaly detection method provided in Embodiment 1 belong to the same concept. The specific operation methods of each module and unit have been described in detail in the method embodiments and will not be repeated here. In practical applications, the video anomaly detection method provided in Embodiment 1 can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0092] Example 3

[0093] Please see Figure 7 As shown, embodiments of this application also provide a computer device, which can be a server. The computer device includes a processor 110, a memory 120, an internal memory 130, and a network interface 140 connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media. The non-volatile storage media stores an operating system 121, computer programs 122, and a database 123. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a login method for cross-application testing.

[0094] Please see Figure 8 As shown, embodiments of this application also provide another computer device, which can be a client. This computer device includes a processor 210, a memory 220, an internal memory 230, a network interface 240, a display screen 260, and an input device 250 connected via a system bus. The processor of this computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium. The non-volatile storage medium stores an operating system 221 and a computer program 222. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a login method for cross-application testing.

[0095] In summary, the technical advantages of this invention lie in enhancing the temporal and action information of the feature data of the video to be detected through an anomaly feature enhancement model, making it more suitable for anomaly detection tasks and improving the utilization rate of feature data in the video. Simultaneously, it provides a classification-guided anomaly localization model that leverages the anomaly localization capabilities of parameters generated during the classification process to optimize the detection results. Furthermore, this classification-guided anomaly localization model can be used in conjunction with any anomaly detection model, improving detection performance, and is easy to use; it can be stored in memory for plug-and-play operation. Further, training the video anomaly detection model of this embodiment based on three loss functions further improves the model's accuracy.

[0096] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A video anomaly detection method, characterized in that, include: Obtain the feature data of the video segments to be detected; The segment feature data is input into a video anomaly detection model to obtain detection results, wherein the video anomaly detection model includes: An anomaly feature enhancement model enhances the segment feature data with temporal and action information to obtain segment enhanced feature data. An anomaly score calculation model calculates an initial segment anomaly score based on the segment enhancement feature data. The anomaly localization model based on classification guidance calculates an anomaly probability score and a segment localization guidance score based on the segment enhancement feature data. The steps include: transposing the segment enhancement feature data to obtain transposed segment feature data; inputting the transposed segment feature data into the first one-dimensional convolution of the classification-guided anomaly localization model for first-stage dimensionality reduction, and then using the ReLU function to calculate the third intermediate data; inputting the third intermediate data into the second one-dimensional convolution of the classification-guided anomaly localization model for second-stage dimensionality reduction, and then using the ReLU function to calculate the fourth intermediate data; inputting the fourth intermediate data into the third one-dimensional convolution of the classification-guided anomaly localization model for third-stage dimensionality reduction, obtaining shared parameters of dimension 1; calculating the segment anomaly probability score using the Softmax function; and calculating the segment localization guidance score using the Sigmoid function. An anomaly score optimization model calculates the final anomaly score of the video to be detected based on the initial segment anomaly score and the segment anomaly probability score; the initial segment anomaly score is optimized using the segment localization guidance score to obtain the final segment anomaly score; the final anomaly score and the final segment anomaly score of the video to be detected are the detection results.

2. The video anomaly detection method according to claim 1, characterized in that, The anomaly feature enhancement model includes a long-range temporal feature enhancement module and a short-range motion feature enhancement module; The step of enhancing the segment feature data with temporal and action information to obtain segment enhanced feature data includes: The fragment feature data is dimensionality reduced by using a first fully connected neural network and a second fully connected neural network to obtain first and second dimensionality-reduced fragment feature data with the same dimension; wherein the first and second fully connected neural networks have the same model structure but different parameter settings. The first and second dimensionality-reduced segment feature data are input into the long-distance temporal feature model to obtain the segment full-time feature data; The full-time feature data of the segment is input into the short-distance action feature model to obtain the segment action attention feature data; By integrating the fragment feature data and fragment action attention feature data, fragment enhancement feature data is obtained.

3. The video anomaly detection method according to claim 2, characterized in that, The step of inputting the first dimensionality-reduced fragment feature data and the second dimensionality-reduced fragment feature data into a long-range temporal series feature model to obtain the fragment full-time feature data includes: The first dimensionality reduction feature data is input into the LSTM neural network of the long-range temporal feature model to obtain fused feature data; The second dimensionality-reduced feature data is input into the nonlocal neural network of the long-distance temporal feature model, and the output result is processed by residual processing to obtain global feature data; wherein, the nonlocal neural network is composed of four one-dimensional convolutions, and the convolution characteristics of the four one-dimensional convolutions are completely consistent; The fused feature data and the global feature data are concatenated to obtain segment full-time feature data; wherein the size of the segment full-time feature data is the same as the size of the segment feature data.

4. The video anomaly detection method according to claim 2, characterized in that, The step of inputting the full-time feature data of the segment into the short-range action feature model to obtain the segment action attention feature data includes: The full-time feature data of the aforementioned segment is averaged over the time dimension. The mean-processed full-time feature data of the segment is input into the first fully connected neural network of the short-distance action feature model for dimensionality reduction, and then the dimensionality-reduced full-time feature data of the segment is input into the second fully connected neural network of the short-distance action feature model for dimensionality increase to obtain action attention weights. The action attention weight is broadcast multiplied by the segment's full-time feature data, and the multiplication result is added to the segment's feature data to obtain the segment's action attention feature data; wherein the segment's action attention feature data has the same size as the segment's feature data.

5. The video anomaly detection method according to claim 1, characterized in that, The step of calculating the initial fragment anomaly score based on the fragment enhancement feature data includes: The fragment enhancement feature data is input into the first fully connected neural network of the anomaly score calculation model for the first dimensionality reduction, and the dimensionality-reduced fragment enhancement feature data is calculated using the ReLU function and the Dropout function to obtain the first intermediate data; The first intermediate data is input into the second fully connected neural network of the anomaly score calculation model for a second dimensionality reduction, and the dimensionality-reduced first intermediate data is calculated using the ReLU function and the Dropout function to obtain the second intermediate data; The second intermediate data is input into the third fully connected neural network of the anomaly score calculation model for a third dimensionality reduction, and the dimensionality-reduced second intermediate data is calculated using the Sigmoid function to obtain the initial segment anomaly score with a dimension of 1.

6. The video anomaly detection method according to claim 1, characterized in that, The step of calculating the final anomaly score of the video to be detected based on the initial segment anomaly score and the segment anomaly probability score includes: The product of the segment anomaly probability score and the initial segment anomaly score is calculated and linearly accumulated to obtain the final anomaly score of the video to be detected.

7. The video anomaly detection method according to claim 1, characterized in that, The step of optimizing the initial segment anomaly score using the segment localization guidance score to obtain the final segment anomaly score includes: The average score of the segment location guidance score is normalized to obtain the optimized score; The Hadamard product is calculated on the initial fragment anomaly score and the optimized score, and the calculation result is processed by the Sigmoid function to obtain the final fragment anomaly score.

8. The video anomaly detection method according to claim 1, characterized in that, The training steps of the video anomaly detection model include: Obtain video-level labels from the training video dataset; Feature extraction is performed on each training video to obtain the segment features of the training video; The segment is fed into the video anomaly detection model to obtain the video anomaly score of the training video and the video segment anomaly score. Calculate the weakly supervised multi-instance loss and temporal smoothness loss based on the anomaly score of the video segment; Calculate the binary cross-entropy loss based on the video anomaly score; The sum of the weakly supervised multi-instance loss, temporal smoothness loss, and binary classification cross-entropy loss is used as the final loss; The video anomaly detection model is backpropagated based on the final loss.

9. A video anomaly detection system, characterized in that, include: The video feature extraction module is used to obtain segment feature data of the video to be detected; The video anomaly detection module is used to obtain the detection results of the segment feature data; The video anomaly detection module includes: An anomaly feature enhancement module enhances the segment feature data with temporal and action information to obtain segment enhanced feature data. The anomaly score calculation module calculates the initial anomaly score of the segment based on the segment enhancement feature data. The anomaly localization module based on classification guidance calculates an anomaly probability score and a segment localization guidance score based on the segment enhancement feature data. The steps include: transposing the segment enhancement feature data to obtain transposed segment feature data; inputting the transposed segment feature data into the first one-dimensional convolution of the anomaly localization model based on classification guidance for a first dimensionality reduction, and then using the ReLU function to calculate the third intermediate data; inputting the third intermediate data into the second one-dimensional convolution of the anomaly localization model based on classification guidance for a second dimensionality reduction, and then using the ReLU function to calculate the fourth intermediate data; inputting the fourth intermediate data into the third one-dimensional convolution of the anomaly localization model based on classification guidance for a third dimensionality reduction, obtaining shared parameters of dimension 1; calculating the shared parameters using the Softmax function to obtain the segment anomaly probability score; and calculating the shared parameters using the Sigmoid function to obtain the segment localization guidance score. The anomaly score optimization module calculates the final anomaly score of the video to be detected based on the initial segment anomaly score and the segment anomaly probability score; it optimizes the initial segment anomaly score using the segment localization guidance score to obtain the final segment anomaly score; wherein the final anomaly score and the final segment anomaly score of the video to be detected are the detection results.

Images