Video anomaly detection method and system based on appearance semantics and motion amplitude
By combining appearance semantics and motion amplitude in video anomaly detection methods, the differences between normal and anomalous events are explicitly modeled, solving the problems of sample imbalance and ambiguous data definition in video anomaly detection, and achieving higher detection accuracy and stability.
Patent Information
- Application Number
- CN202410916946.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-10
AI Technical Summary
In existing video anomaly detection methods, the imbalance between normal and abnormal samples and the lack of clear definition of abnormal data lead to poor performance of supervised methods. Furthermore, the training objectives of the proxy task are inconsistent with the anomaly detection objectives, which limits the improvement of model performance.
A video anomaly detection method based on appearance semantics and motion amplitude is adopted. Features are extracted through image encoder and optical flow estimation model, appearance anomaly score and motion anomaly score are calculated, and the video frame is judged as abnormal by weighted summation, thus explicitly modeling the difference between normal and abnormal events.
It improves the accuracy and stability of anomaly detection, can more comprehensively consider abnormal situations, has anti-interference and interpretability, avoids overfitting, and enhances the robustness of the model.
Smart Images

Figure CN118470611B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video anomaly detection, and in particular to a video anomaly detection method and system based on appearance semantics and motion amplitude. Background Art
[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.
[0003] Video anomaly detection plays a vital role in maintaining public safety. Video anomaly detection refers to identifying behaviors or activities that are contrary to normal patterns from video data. By applying intelligent algorithms for anomaly detection, not only can human and material resources be greatly saved, but also the accuracy of detection can be improved, providing more reliable protection for public safety. The core goal of this task is to identify events in the video that deviate from the normal data pattern. However, video anomaly detection faces two major challenges. First, there is a significant imbalance in the number of normal and abnormal samples, because abnormal events have a low probability of occurrence, while normal events dominate in most surveillance videos. Second, the lack of clear definition of abnormal data makes it impossible to collect all types of abnormal events. These challenges make it impractical to use supervised methods to solve this problem. Therefore, a semi-supervised approach is usually used to deal with this problem, that is, only normal data is used for training to learn normal patterns. This approach can more effectively address the challenges of video anomaly detection and improve the accuracy and efficiency of detection.
[0004] Early methods usually rely on handcrafted features or suboptimal non-end-to-end features to model normal data patterns. With the development of deep learning technology, researchers usually use various proxy tasks to fit normal datasets for video anomaly detection. The mainstream proxy task methods include reconstruction-based methods and prediction-based methods. Reconstruction-based methods are trained on the training set, and the training goal is to reconstruct the input data so that the model is more inclined to reconstruct normal data. During the test process, the model usually produces a larger reconstruction error for untrained data distributions. However, as the generalization ability of the model increases, the model can also effectively reconstruct abnormal samples. In order to enhance the fitting bias, additional fitting constraints are often introduced to weaken the generalization ability of the model. Prediction-based methods are more suitable for learning time series video data, which is usually more difficult than reconstruction tasks, which helps to enhance the fitting bias of training data. However, for datasets with relatively simple motion patterns, prediction tasks also require more constraints to further increase the fitting bias. At the same time, prediction methods may face difficulties in learning complex normal motion patterns, which makes it difficult to ensure that the prediction error of normal data is smaller. In addition, some researchers perform anomaly detection by adopting other proxy tasks or combining multiple proxy tasks. A single proxy task tends to be biased towards detecting a certain type of anomaly, while multiple proxy tasks can fit normal data from different perspectives, resulting in complementary effects. This approach helps to detect various types of abnormal events more comprehensively and improve the accuracy and reliability of anomaly detection.
[0005] However, the inconsistency between the training objective of the proxy task and the anomaly detection objective at test time limits the improvement of model performance. The training objective of the proxy task is to enable the model to learn the ability to achieve specific task objectives, rather than to force the model to capture the data distribution of the normal mode. The training objective of the proxy task cannot guarantee that the model fits the normal mode well or that the model is biased towards the normal mode. Therefore, distinguishing normal events from abnormal events through this implicit fitting difference is limited by the mismatch between the proxy task objective and the anomaly detection task objective. Summary of the invention
[0006] In order to solve the above problems, the present invention proposes a video anomaly detection method and system based on appearance semantics and motion amplitude, which uses a neural network model to extract the target foreground and calculate the appearance anomaly score of the target foreground; obtains the optical flow map corresponding to the target foreground through a supervised optical flow estimation model, and then calculates the motion anomaly score; and aligns the training target and the anomaly detection target by explicitly modeling the differences in appearance and motion between normal events and abnormal events, which can improve the accuracy and stability of detection.
[0007] In some embodiments, the following technical solutions are adopted:
[0008] A video anomaly detection method based on appearance semantics and motion amplitude, comprising:
[0009] Obtain the target foreground of the video frame to be tested, and input the target foreground into the trained image encoder to obtain the image coding feature;
[0010] Clustering the image coding features to obtain the category label of each target foreground, and setting text prompts corresponding to the number of categories based on the number of categories; the text prompts include a learnable word embedding vector, and the learnable word embedding vector is optimized to obtain an optimized text prompt for each cluster category;
[0011] Calculate the similarity between the image encoding features of the target foreground and the text prompts of each cluster category, and calculate the appearance anomaly score of the target foreground by the maximum similarity;
[0012] Obtain the optical flow map corresponding to the target foreground of the video frame to be tested, input the optical flow map into the trained adaptive encoder to obtain the encoding features, and then obtain the motion anomaly score of the target foreground through a binary classifier;
[0013] The appearance anomaly score and the motion anomaly score of the target foreground are weightedly summed to obtain the anomaly score of the entire video frame, and whether the video frame is abnormal is determined based on the anomaly score.
[0014] In other embodiments, the following technical solutions are adopted:
[0015] A video anomaly detection system based on appearance semantics and motion amplitude includes an appearance semantics detection module, a motion amplitude detection module and an anomaly judgment module; wherein,
[0016] The appearance semantic detection module includes:
[0017] An image coding feature extraction unit is used to obtain a target foreground of a video frame to be tested, and input the target foreground into a trained image encoder to obtain an image coding feature;
[0018] A pseudo-label acquisition unit, configured to cluster the image encoding features to obtain a category label for each target foreground, and to set a text prompt corresponding to the number of categories based on the number of categories; the text prompt includes a learnable word embedding vector;
[0019] A text prompt acquisition unit, used to optimize the learnable word embedding vector to obtain an optimized text prompt for each cluster category;
[0020] An appearance anomaly score calculation unit, used to calculate the similarity between the image encoding features of the target foreground and the text prompts of each cluster category, and calculate the appearance anomaly score of the target foreground through the maximum similarity;
[0021] The motion amplitude detection module includes:
[0022] An optical flow map acquisition unit, used to acquire an optical flow map corresponding to a target foreground of a video frame to be tested;
[0023] A motion anomaly score acquisition unit is used to input the optical flow map into the trained adaptive encoder to obtain the encoding features, and then obtain the motion anomaly score of the target foreground through a binary classifier;
[0024] The anomaly judgment module is used to perform weighted summation of the appearance anomaly score and the motion anomaly score of the target foreground to obtain the anomaly score of the entire video frame, and judge whether the video frame is abnormal according to the anomaly score.
[0025] In other embodiments, the following technical solutions are adopted:
[0026] A terminal device comprises a processor and a memory, wherein the processor is used to implement instructions; the memory is used to store a plurality of instructions, wherein the instructions are suitable for being loaded by the processor and executing the above-mentioned video anomaly detection method based on appearance semantics and motion amplitude.
[0027] In other embodiments, the following technical solutions are adopted:
[0028] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor of a terminal device and executing the above-mentioned video anomaly detection method based on appearance semantics and motion amplitude.
[0029] Compared with the prior art, the present invention has the following beneficial effects:
[0030] (1) The proposed method aligns the training objective and the anomaly detection objective by explicitly modeling the differences between normal and abnormal events in appearance and motion. In terms of appearance, anomalies are detected based on the semantic difference between normal and abnormal appearances; in terms of motion, anomalies are detected based on the difference in motion amplitude between normal and abnormal motions. By combining appearance and motion information, possible abnormal situations can be more comprehensively considered.
[0031] (2) When performing appearance anomaly detection, the present invention uses a learnable word embedding vector to learn the semantic prototype of a specific category and model the textual semantic representation of the normal data pattern. During the test process, anomalies are detected by evaluating the semantic difference between the input foreground and the learned normal pattern semantic prototype. The method of the present invention starts from the perspective of the concept of anomaly and performs anomaly detection based on semantic differences, which has better anti-interference and interpretability. The pseudo-label of the foreground is obtained by clustering and a learnable text prompt is set for each category. The text prompt is learned using semantic alignment loss, and the text prompt represents the semantic prototype of the normal pattern.
[0032] (3) In order to better capture the changes in motion amplitude during motion amplitude anomaly detection, the present invention uses optical flow to represent motion information, and introduces pseudo-abnormal motion flow to assist in training classification tasks. An adaptive encoder is proposed to assist in training the classifier to model the motion amplitude differences of positive and abnormal data, and two different decoding tasks are used to guide the training of the adaptive encoder. The trained adaptive encoder can dynamically adjust its encoding strategy according to different input data to obtain features that are discriminative and robust. Subsequently, these features are used to train a classifier to model the differences in motion amplitude. This can ensure the accuracy of classification, avoid the introduction of noise, and prevent overfitting.
[0033] (4) Our method does not rely on implicit fitting biases for anomaly detection; instead, it explicitly models the differences in appearance and motion between normal and abnormal events to align the goal of anomaly detection, which is more robust and more interpretable.
[0034] Other features and advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 Flow chart of a method for detecting anomalies in a video based on appearance semantics and motion amplitude in an embodiment of the present invention;
[0036] Figure 2 Schematic diagram of the appearance semantic detection process in an embodiment of the present invention;
[0037] Figure 3 Schematic diagram of the motion amplitude detection process in an embodiment of the present invention;
[0038] Figure 4 A schematic diagram of the structure of an adaptive encoder in an embodiment of the present invention;
[0039] Figure 5 A schematic diagram of the structure of a decoder in an embodiment of the present invention;
[0040] Figure 6 Schematic diagram of the structure of a classifier in an embodiment of the present invention. DETAILED DESCRIPTION
[0041] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0042] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0043] Embodiment 1
[0044] In one or more embodiments, a video anomaly detection method based on appearance semantics and motion amplitude is disclosed, combined with Figure 1 , specifically including the following process:
[0045] S101: Obtain the target foreground of the video frame to be tested, and input the target foreground into the trained image encoder to obtain image coding features ;
[0046] S102: Encoding features of the image Clustering is performed to obtain the category label of each target foreground, and text prompts corresponding to the number of categories are set based on the number of categories. The text prompts include a learnable word embedding vector, and the learnable word embedding vector is optimized to obtain an optimized text prompt for each cluster category;
[0047] S103: Calculate the similarity between the image encoding feature and the text prompt of each cluster category, and calculate the appearance anomaly score of the target foreground through the maximum similarity;
[0048] S104: while calculating the appearance anomaly score, obtaining an optical flow map corresponding to the target foreground of the video frame to be tested, inputting the optical flow map into the trained adaptive encoder to obtain encoding features, and then obtaining a motion anomaly score of the target foreground through a binary classifier;
[0049] S105: Perform a weighted summation of the appearance anomaly score and the motion anomaly score of the target foreground to obtain an anomaly score of the entire video frame, and determine whether the video frame is abnormal based on the anomaly score.
[0050] As a specific implementation method, the method of this embodiment can be mainly divided into three parts:
[0051] (1) Appearance semantic detection part
[0052] Anomaly is a high-level concept, and low-level pixel generation tasks often force the model to focus too much on pixel-level details and have difficulty paying attention to high-level semantic information. The difference between the model-generated results and the real target pixels cannot accurately detect anomalies.
[0053] Appearance anomaly means that it can be judged as abnormal only from the appearance. For example, in the public data set UCSD Ped2, the appearance of a car is an anomaly. Since the car has never appeared in the training set, it can be judged as abnormal from its appearance. This embodiment models the semantics of appearance. Since the training set does not include cars, when a car appears, its semantics are not similar to the semantic prototype modeled by the training set, and the car can be judged as abnormal.
[0054] This embodiment uses text semantics to detect appearance anomalies for the first time. It learns multiple semantic prototypes to represent normal patterns and detects anomalies from a semantic level. The specific implementation process is as follows: Figure 2 As shown, specifically including:
[0055] (1-1) Clustering to obtain pseudo labels:
[0056] The video data of the training set and the test set are obtained. The training set only includes the normal data set, and the test set includes normal frames and abnormal frames that need to be labeled with normal or abnormal category labels. The YOLOv3 neural network is used to extract all the foregrounds of the training set video frames, and the foregrounds are input into the pre-trained CLIP image encoder to obtain cross-modal semantic features. The extracted features are clustered using the DBSCAN clustering method. The distance between different samples is calculated using the cosine distance. Through clustering, foregrounds with similar semantics are classified into one category. In this way, N semantic categories are obtained through clustering, and each foreground corresponds to a semantic category.
[0057] (1-2) Set text prompt:
[0058] The cluster centers can be regarded as coarse-grained semantic prototypes. However, these cluster centers are more susceptible to noise interference and may not be well aligned with the semantic space. In order to further obtain fine-grained semantic prototype representation, this embodiment makes full use of the cross-modal description capability in CLIP to learn the semantics of the normal mode and sets N corresponding text prompts according to the number of cluster categories. This embodiment defines the text prompt as “A photo of a object"; is a learnable word embedding vector. object is determined according to the object category involved in the normal training data set. For example, if the object category involved in the normal training data set is human, the text prompt is "A photo of a person"; the rest are fixed texts, which can provide additional semantic information and assist semantic learning.
[0059] (1-3) Modeling normal mode semantic prototype:
[0060] This embodiment optimizes the learnable word embedding vector and models the semantic prototype of the normal data set by automatically learning the semantic representation of the training data set through training.
[0061] Specifically, the text prompt of any foreground corresponding category can be determined based on the pseudo-label. Input the pre-trained CLIP image encoder to obtain the encoding features The image encoder uses the ResNet50 neural network architecture. After the image passes through ResNet50, a two-dimensional attention pooling layer (AttentionPool, AP) is used to weight the features and generate the final image encoding features. :
[0062] ;
[0063] The text encoder uses the Transformer architecture to generate feature representations from sentences. The text prompt for the corresponding category is First, each word in the text prompt is converted into a 512-dimensional word embedding vector through a vocabulary. Then the learnable position is replaced with the initialized learnable word embedding vector. The fixed length of each text sequence after the final conversion is 77, including the starting [SOS] and ending [EOS] tags. Then it is input into the text encoder, and after passing through a 12-layer masked multi-head self attention module (MMHSA) model with 8 attention heads, the [EOS] tag is regarded as the feature representation of the text, layer normalization LN is performed, and then linear projection (LinearProjection, LP) is used to obtain the feature in the cross-modal embedding space. :
[0064] ;
[0065] Display pictures The semantic features of the corresponding category.
[0066] Then, the semantic similarity loss is used to align the two features to optimize the learnable word embedding vector in the text prompt, where the image encoder and text encoder parameters of CLIP are fixed during training. Since cosine similarity is suitable for measuring the distance of semantic features, the semantic similarity loss uses cosine similarity, and the formula is as follows:
[0067] ;
[0068] By optimizing the learnable word embedding vectors of N categories, we can obtain the text prompts corresponding to the N categories and regard these N text prompts as semantic prototypes of normal patterns.
[0069] (1-4) Calculation of appearance anomaly score:
[0070] During testing, the parameters of the pre-trained CLIP image encoder and text encoder as well as the learned text word embedding vector are fixed, and the target foreground image is Input into the CLIP image encoder to generate the corresponding feature image code , and then compare this feature with the semantic features of the semantic prototypes of the normal patterns of the N categories learned Calculate the cosine similarity, Represents the semantic features of N categories of text prompts after entering the text encoder.
[0071] Before calculating the similarity, the feature vector needs to be normalized, and the largest similarity among the semantic similarities with N semantic prototypes is selected and recorded as , and its calculation formula is as follows:
[0072] ;
[0073] The greater the similarity, the closer the semantics of the foreground is to a normal pattern prototype, and the more likely the foreground is a normal event. The appearance anomaly score can be obtained as follows:
[0074] ;
[0075] The closer the score is to 1, the more likely the input is an anomaly. The semantic model can accurately detect anomalies caused by different appearance semantics, and this detection method is more explainable.
[0076] (2) Motion amplitude detection part
[0077] Since the CLIP model does not have a good understanding of motion semantics, it is difficult to obtain accurate motion semantics using the CLIP model. Therefore, anomalies caused by motion (such as running, chasing, and irregular motion) are difficult to detect. In order to further solve the problem of anomalies caused by motion, according to the characteristics of motion anomalies in the data set, the present invention detects motion anomalies based on the difference in motion amplitude between normal and abnormal data. The main process is as follows: Figure 3 As shown, including:
[0078] (2-1) Movement amplitude indicates:
[0079] In order to model the difference between normal motion amplitude and abnormal motion amplitude, this embodiment uses optical flow as a representation of motion amplitude. The optical flow estimation model FlowNet is used to extract the optical flow map between two frames, and the optical flow map is cropped according to the foreground position to obtain the corresponding optical flow of the foreground. and The optical flow graph between As normal optical flow. Because normal optical flow is accurate normal data, in order to generate the boundary of normal optical flow as accurately as possible while avoiding overfitting, the foreground RGB image of the two frames is and The optical flow between As the abnormal optical flow. Representing the FlowNet network, the normal optical flow and abnormal optical flow are calculated as follows:
[0080] ;
[0081] ;
[0082] Scale the optical flow to 32×32 size, and They represent the different amplitudes of normal movement and abnormal movement respectively, and the difference between normal movement and abnormal movement is simulated by modeling the difference between the two.
[0083] (2-2) Training of adaptive encoder:
[0084] Using optical flow directly as input for binary classification may be affected by low-level noise information, causing the model to overfit easily and making it difficult to learn discriminative features. In order to obtain more representative, discriminative and robust features from the input optical flow map, this embodiment introduces an adaptive encoder that can generate different features for different optical flow size inputs. The adaptive encoder consists of a Stem layer and three downsampling layers. The Stem layer includes sequentially connected Conv (convolution), BN (batch normalization) and ReLU activation (hereinafter referred to as Conv+BN+ReLU), where the convolution kernel size is 3×3 and the step size is 1. In order to reduce information loss, when downsampling, instead of using pooling, a convolution layer with a step size of 2 is used to downsample the features. Therefore, the downsampling layer includes a layer of Conv+BN+ReLU for downsampling, where the convolution kernel size is 3×3 and the step size is 2, plus two layers of Conv+BN+ReLU, where the convolution kernel size of these two layers is 3×3 and the step size is 1. Its structure is as follows Figure 4As shown in the figure, downsampling is performed three times until the feature size is 4×4. Then the feature is gradually upsampled to restore the input size through the optical flow reconstruction decoder symmetrical to the encoder. The decoder consists of an upsampling layer and an output layer. Deconvolution is used during upsampling. The upsampling layer consists of Deconv (deconvolution), BN (batch normalization) and ReLU activation connected in sequence. The output layer also uses Conv+BN+ReLU to adjust the dimension to the size of the output target. The structure of the decoder is as follows Figure 5 shown.
[0085] Optical flow input map for normal samples It is passed through the adaptive encoder E to obtain the encoded features, and then the features are input into the optical flow reconstruction decoder To generate the reconstructed optical flow :
[0086] ;
[0087] For normal optical flow input, the training loss uses the reconstruction error between the reconstructed optical flow and the input optical flow:
[0088] ;
[0089] in, Optical flow input map representing normal samples The encoding features obtained by the adaptive encoder E; and Represents the height and width of the optical flow map, and Respectively represent the spatial positions of the input optical flow map and the reconstructed optical flow map as The optical flow value.
[0090] For the input of abnormal optical flow , passing it through the adaptive encoder E and the cross-modal picture to generate a decoder To generate the RGB image corresponding to its optical flow :
[0091] ;
[0092] in, Input representing abnormal optical flow The encoding features obtained by the adaptive encoder E; the structure of the cross-modal image generation decoder is consistent with the optical flow reconstruction decoder. Cross-modality means that the input is an optical flow map and the output is an RGB map (i.e., a video frame image). For abnormal optical flow, the generated RGB map is used. The original RGB image corresponding to the input optical flow The mean square error between pixels is used for training, and the loss is as follows:
[0093] ;
[0094] in, and Indicates the height and width of the image. and Respectively represent the original RGB image and the generated RGB image in the spatial position RGB values.
[0095] It should be noted that the original RGB image is the video frame image in the dataset. The optical flow image is obtained by using the optical flow estimation model FlowNet through the video frame image. The optical flow image is passed through an adaptive encoder and an image generation decoder to generate an RGB image corresponding to the input optical flow. The mean square error between the generated image and the original video frame image is used for training.
[0096] By alternately training the two different tasks, different generation targets during training will optimize the encoder to generate encoding features suitable for the corresponding tasks, so that for two different input encoders, different encoding features will be generated to complete different tasks; for normal optical flow, encoding features for reconstruction are generated, and for abnormal optical flow, cross-modal features for generating images are generated. This enables the encoder to dynamically adjust its encoding method according to different input data, thereby providing discriminative and robust input features for subsequent classification models.
[0097] By training the adaptive encoder, an adaptive encoder that generates different features for inputs of different optical flow sizes can be obtained.
[0098] (2-3) Calculation of movement abnormality score
[0099] In this embodiment, the optical flow map is input into a trained adaptive encoder through an adaptive encoder to obtain encoding features; the encoding features are further accurately modeled through a lightweight binary classifier to the boundary between normal motion and abnormal motion.
[0100] This embodiment uses pseudo-labeling of optical flow to train a binary classifier so that it can learn to classify normal motion optical flow and abnormal motion optical flow. Freeze the parameters of the adaptive encoder, input the original optical flow into the adaptive encoder to extract the encoding features, and then use the extracted encoding features as the input of the binary classifier. Since the encoding features have a certain degree of discrimination, a lightweight classifier architecture is required to prevent overfitting. The detailed information of the binary classifier architecture is as follows: Figure 6As shown, it includes five layers of modules: the first layer is a convolutional layer to further extract features and reduce the dimension, and then flatten the features and input them into the regularized linear activation module for processing, and then go through three layers of regularized linear activation modules. Each layer of regularized linear activation modules is a Linear (linear layer), ReLU activation layer and Dropout (random inactivation layer) connected in sequence, and finally outputs an inferred probability through Sigmod activation, which represents the probability that the input is abnormal. A random inactivation layer (Dropout layer) is added after each linear layer and activation function, which can effectively prevent the classifier from overfitting on the training set and reduce false positives for normal movements. For the training of the classifier, cross entropy loss is used for training to measure the predicted probability that the binary classifier judges the input to be abnormal. With real mark The degree of closeness, where the abnormal optical flow input is marked 1, normal optical flow input mark is 0, the loss function is expressed as follows:
[0101] ;
[0102] Obtaining abnormal probability through binary classifier , which is considered as the motion abnormality score .
[0103] (3) Video abnormality judgment part
[0104] This embodiment combines the appearance anomaly score and the motion anomaly score to obtain the target foreground anomaly score. :
[0105] ;
[0106] in, and Balance the contribution of the two parts to the total anomaly score. Since the anomaly score calculated here is based on the foreground target, it is necessary to further obtain the anomaly score of the frame. Since a frame is marked as abnormal as long as there is an abnormality in the foreground, the score of the foreground target with the largest anomaly score among all k foreground targets in a frame is selected as the anomaly score of the entire frame. :
[0107] ;
[0108] The video frames are divided into normal / abnormal frames according to the set abnormal score threshold, where frames with abnormal scores greater than the set threshold are judged as abnormal frames, and frames with abnormal scores less than the set threshold are judged as normal frames.
[0109] Embodiment 2
[0110] In one or more embodiments, a video anomaly detection system based on appearance semantics and motion amplitude is disclosed, including an appearance semantics detection module, a motion amplitude detection module and an anomaly judgment module; wherein:
[0111] (1) Appearance semantic detection module includes:
[0112] The image coding feature extraction unit is used to obtain the target foreground of the video frame to be tested, and input the target foreground into the trained image encoder to obtain the image coding feature ;
[0113] Pseudo label acquisition unit, used to encode image features Perform clustering to obtain a category label for each target foreground, and set text prompts corresponding to the number of categories based on the number of categories; the text prompts include a learnable word embedding vector;
[0114] A text prompt acquisition unit, used to optimize the learnable word embedding vector to obtain an optimized text prompt for each cluster category;
[0115] An appearance anomaly score calculation unit, used to calculate the similarity between the image encoding features of the target foreground and the text prompts of each cluster category, and calculate the appearance anomaly score of the target foreground through the maximum similarity;
[0116] (2) The motion amplitude detection module includes:
[0117] An optical flow map acquisition unit, used to acquire an optical flow map corresponding to a target foreground of a video frame to be tested;
[0118] A motion anomaly score acquisition unit is used to input the optical flow map into the trained adaptive encoder to obtain the encoding features, and then obtain the motion anomaly score of the target foreground through a binary classifier;
[0119] (3) Anomaly judgment module, used to perform weighted summation of the appearance anomaly score and the motion anomaly score of the target foreground to obtain the anomaly score of the entire video frame, and judge whether the video frame is abnormal based on the anomaly score.
[0120] The specific implementation of the above modules is the same as that in Example 1 and will not be described in detail.
[0121] Embodiment 3
[0122] In one or more embodiments, a terminal device is disclosed, including a server, the server including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the video anomaly detection method based on appearance semantics and motion amplitude in Embodiment 1 when executing the program. For the sake of brevity, it will not be described in detail here.
[0123] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0124] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.
[0125] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.
[0126] Embodiment 4
[0127] In one or more embodiments, a computer-readable storage medium is disclosed, in which a plurality of instructions are stored, wherein the instructions are suitable for being loaded by a processor of a terminal device and executing the video anomaly detection method based on appearance semantics and motion amplitude described in Embodiment 1.
[0128] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.
Claims
1. A video anomaly detection method based on appearance semantics and motion amplitude, characterized in that: include: Obtain the target foreground of the video frame to be tested, and input the target foreground into the trained image encoder to obtain the image coding feature; Clustering the image coding features to obtain the category label of each target foreground, and setting text prompts corresponding to the number of categories based on the number of categories; the text prompts include a learnable word embedding vector, and the learnable word embedding vector is optimized to obtain an optimized text prompt for each cluster category; Among them, the image encoding features are clustered to obtain the category label of each target foreground, specifically: The DBSCAN clustering method is used to cluster the image encoding features. The cosine distance is used to calculate the distance between different samples. The foreground with similar semantics is classified into one category through clustering. In this way, N semantic categories are obtained through clustering, and each foreground corresponds to a semantic category; The text prompt is defined as: A photo of a object; is a learnable word embedding vector, object is determined according to the object category involved in the normal training dataset, and the rest are fixed texts; Among them, the learnable word embedding vector is optimized as follows: prospect Input the pre-trained CLIP image encoder to obtain the encoding features ; prospect The text prompt for the corresponding category is , convert each word in the text prompt into a word embedding vector through a vocabulary, replace the learnable position with the learnable word embedding vector to obtain the text sequence; input the text sequence into the text encoder to obtain the feature ; Indicates prospects The semantic features of the corresponding category; Align features using semantic similarity loss and Features , to optimize the learnable word embedding vectors in text prompts; Calculate the similarity between the target foreground image encoding features and the text prompts of each cluster category, and calculate the appearance anomaly score of the target foreground through the maximum similarity; Obtain the optical flow map corresponding to the target foreground of the video frame to be tested, input the optical flow map into the trained adaptive encoder E to obtain the encoding features, and then obtain the motion anomaly score of the target foreground through a binary classifier; Performing a weighted summation of the appearance anomaly score and the motion anomaly score of the target foreground to obtain an anomaly score for the entire video frame, and determining whether the video frame is abnormal based on the anomaly score; When the adaptive encoder E is trained, the foreground frames of the training data are temporally adjacent. and The optical flow graph between As normal optical flow; For normal optical flow input map , pass it through the adaptive encoder E to obtain the encoding features, and then input the encoding features into the optical flow reconstruction decoder To generate the reconstructed optical flow; the training loss of the normal optical flow uses the reconstruction error between the reconstructed optical flow and the real optical flow; When the adaptive encoder E is trained, the foreground frames separated by two frames are and The optical flow between As abnormal optical flow; for the input of abnormal optical flow , through the adaptive encoder E and the picture to generate the decoder To generate the RGB image corresponding to the input optical flow ; The training loss of abnormal optical flow uses the generated RGB image The original RGB image corresponding to its input optical flow The mean square error between pixels.
2. The video anomaly detection method based on appearance semantics and motion amplitude as claimed in claim 1, characterized in that: The appearance anomaly score of the target foreground is calculated by the maximum similarity, specifically: the appearance anomaly score of the target foreground is 1 minus the maximum similarity.
3. The video anomaly detection method based on appearance semantics and motion amplitude as claimed in claim 1, characterized in that: A binary classifier is trained by using pseudo-labeling of optical flow to learn the classification of normal optical flow and abnormal optical flow; the binary classifier includes a convolutional layer, three regularized linear activation modules and a Sigmoid activation function connected in sequence; a random dropout layer is added after each linear layer and activation function in the regularized linear activation module to prevent overfitting of the classifier.
4. A video anomaly detection system based on appearance semantics and motion amplitude, characterized in that: It includes appearance semantic detection module, motion amplitude detection module and abnormality judgment module; among them, The appearance semantic detection module includes: An image coding feature extraction unit is used to obtain a target foreground of a video frame to be tested, and input the target foreground into a trained image encoder to obtain an image coding feature; A pseudo-label acquisition unit, configured to cluster the image encoding features to obtain a category label for each target foreground, and to set a text prompt corresponding to the number of categories based on the number of categories; the text prompt includes a learnable word embedding vector; A text prompt acquisition unit, used to optimize the learnable word embedding vector to obtain an optimized text prompt for each cluster category; Among them, the image encoding features are clustered to obtain the category label of each target foreground, specifically: The DBSCAN clustering method is used to cluster the image encoding features. The cosine distance is used to calculate the distance between different samples. The foreground with similar semantics is classified into one category through clustering. In this way, N semantic categories are obtained through clustering, and each foreground corresponds to a semantic category; The text prompt is defined as: A photo of a object; is a learnable word embedding vector, object is determined according to the object category involved in the normal training dataset, and the rest are fixed texts; Among them, the learnable word embedding vector is optimized as follows: prospect Input the pre-trained CLIP image encoder to obtain the encoding features ; prospect The text prompt for the corresponding category is , convert each word in the text prompt into a word embedding vector through a vocabulary, replace the learnable position with the learnable word embedding vector to obtain the text sequence; input the text sequence into the text encoder to obtain the feature ; Indicates prospects The semantic features of the corresponding category; Align features using semantic similarity loss and Features , to optimize the learnable word embedding vectors in text prompts; An appearance anomaly score calculation unit, used to calculate the similarity between the target foreground image encoding features and the text prompts of each cluster category, and calculate the appearance anomaly score of the target foreground through the maximum similarity; The motion amplitude detection module includes: An optical flow map acquisition unit, used to acquire an optical flow map corresponding to a target foreground of a video frame to be tested; The motion anomaly score acquisition unit is used to input the optical flow map into the trained adaptive encoder E to obtain the encoding features, and then obtain the motion anomaly score of the target foreground through a binary classifier; When the adaptive encoder E is trained, the foreground frames of the training data are temporally adjacent. and The optical flow graph between As normal optical flow; for the normal optical flow input map , pass it through the adaptive encoder E to obtain the encoding features, and then input the encoding features into the optical flow reconstruction decoder To generate the reconstructed optical flow; the training loss of the normal optical flow uses the reconstruction error between the reconstructed optical flow and the real optical flow; When the adaptive encoder E is trained, the foreground frames separated by two frames are and The optical flow between As abnormal optical flow; for the input of abnormal optical flow , through the adaptive encoder E and the picture to generate the decoder To generate the RGB image corresponding to the input optical flow ; The training loss of abnormal optical flow uses the generated RGB image The original RGB image corresponding to its input optical flow The mean square error between pixels; The anomaly judgment module is used to perform weighted summation of the appearance anomaly score and the motion anomaly score of the target foreground to obtain the anomaly score of the entire video frame, and judge whether the video frame is abnormal according to the anomaly score.
5. A terminal device, comprising a processor and a memory, wherein the processor is used to implement instructions; and the memory is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the video anomaly detection method based on appearance semantics and motion amplitude according to any one of claims 1 to 3.
6. A computer-readable storage medium storing a plurality of instructions, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executing the video anomaly detection method based on appearance semantics and motion amplitude as described in any one of claims 1 to 3.
