A video anomaly detection model training method and a video anomaly detection method

Abstract

The specification relates to the technical field of computer vision, and relates to a video anomaly detection model training method and a video anomaly detection method. The video anomaly detection model training method comprises obtaining a sample video frame sequence and a video anomaly detection model. The video anomaly detection model comprises an appearance encoder, a motion encoder, a consistency fusion module and a decoder. 0-t-1 frames in the sample video frame sequence are input into the video anomaly detection model, and a predicted frame of the t frame is output. A first loss function is used to perform convergence judgment on the predicted frame of the t frame and the sample frame, and model parameters are optimized. The video anomaly detection model prediction step and the convergence judgment step are repeated until the first loss function converges, and the video anomaly detection model training is completed. The video anomaly detection model training method and the video anomaly detection method improve the prediction error of anomalies, have high accuracy of video anomaly detection, and have small calculation amount.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer vision technology, and in particular to a training method for a video anomaly detection model and a video anomaly detection method. Background Technology

[0002] Video anomaly detection refers to identifying unexpected and abnormal behavioral events in videos, specifically detecting anomalous frames where such events occur. Anomaly event analysis has significant research value in areas such as traffic monitoring, smart security, smart homes, medical rehabilitation, public safety, and urban digital governance and services. Because the definition of anomalous behavioral events is somewhat vague and difficult to fully collect, most current video anomaly detection algorithms currently use training sets containing only normal data, predicting future video frames based on this normal data.

[0003] Existing prediction-based video anomaly detection methods often focus only on the overall video frame sequence features, i.e., global appearance features. When there are few moving targets (foreground objects) in the video, their background occupies the vast majority of the video area, ignoring the importance of foreground objects in video frames under surveillance. Although information about foreground objects can be explored through optical flow, it requires pre-collection and manipulation of optical flow, which involves a large amount of computation and generates additional computational overhead. Summary of the Invention

[0004] The embodiments of this specification provide a training method for a video anomaly detection model and a video anomaly detection method, which aim to solve one or more of the above-mentioned problems and other potential problems.

[0005] To achieve the above objectives, the following technical solution is provided:

[0006] According to a first aspect of this specification, a method for training a video anomaly detection model is provided, the method comprising:

[0007] Acquire sample video frame sequences and a video anomaly detection model; the video anomaly detection model includes an appearance encoder, a motion encoder, a consistency fusion module, and a decoder.

[0008] The video anomaly detection model inputs frames 0 to t-1 from the sample video frame sequence and outputs a predicted frame for frame t, specifically including:

[0009] The appearance encoder extracts appearance features from frames 0 to t-1, and the motion encoder uses an inter-frame attention mechanism to extract motion features from frames 0 to t-1. The consistency fusion module fuses the appearance features and the motion features to obtain the final features. The decoder decodes the final features and outputs the predicted frame for frame t.

[0010] The first loss function is used to determine the convergence of the predicted frame and the sample frame of frame t, and the model parameters are optimized.

[0011] Repeat the above video anomaly detection model prediction steps and convergence judgment steps until the first loss function converges, at which point the video anomaly detection model training is complete.

[0012] This specification describes a training method for a video anomaly detection model. It embeds an attention mechanism into a motion encoder to acquire motion features of foreground objects, addressing the problem of autoencoders focusing too much on global features, thus making it difficult to balance normal and abnormal modes. This method has lower computational cost and higher anomaly recognition accuracy. Simultaneously, a consistency fusion module is used to fuse appearance and motion features. The fusion addresses the low-quality prediction frames caused by semantic inconsistencies between human appearance and motion features, enabling anomalous samples to generate larger prediction errors on autoencoders trained only with normal data, further improving the sensitivity of video anomaly detection.

[0013] In some embodiments, the video anomaly detection model further includes a memory enhancement module, which stores several prototype features. After the appearance encoder extracts appearance features from frames 0 to t-1, it retrieves the prototype feature most similar to the appearance features. Based on the similarity probability between the appearance features and the retrieved prototype features, as well as all prototype features, a new appearance feature is generated, and the retrieved prototype features are updated based on the appearance features. The consistency fusion module fuses the motion features and the new appearance features. The prototype features are the appearance features of the sample video frames when no abnormal events occur.

[0014] In some embodiments, during the convergence determination of the predicted frame and sample frame of frame t using the first loss function, the method further includes using the second loss function to determine the similarity between the appearance feature and the retrieved prototype feature, and using the third loss function to determine the difference between the difference between the appearance feature and the retrieved prototype feature and the difference between the appearance feature and the second similar prototype feature; when the first loss function, the second loss function and the third loss function all converge, the video anomaly detection model is trained.

[0015] In some embodiments, the first loss function, the second loss function, and the third loss function must also satisfy the following conditions during training:

[0016] L = L fra +η c L c +η d L d

[0017] Where L represents the comprehensive loss function, L fra Let L represent the first loss function. c The second loss function, L d Let η represent the third loss function. c η represents the balance parameter of the second loss function. d The third loss function represents the balance parameter. When the comprehensive loss function converges, the video anomaly detection model is trained.

[0018] In some embodiments, updating the retrieved prototype features based on the appearance features includes: updating the retrieved prototype features using the appearance features of the predicted frame of frame t, wherein a normality score is calculated based on the predicted frame of frame t and the sample frame during the update; when the normality score is lower than a first threshold, the predicted frame of frame t is determined to be a normal frame, and the retrieved prototype features are updated to the predicted frame.

[0019] In some embodiments, the motion encoder extracts motion features from frames 0 to t-1 using an inter-frame attention mechanism, including:

[0020] Extract the appearance features from frames 0 to t-1, take frame 0 as the current frame, use any region in the appearance features of the current frame as the query region, and use the appearance features of the adjacent frames of the current frame as keywords and values ​​to generate an attention map; estimate the displacement of the query region between frames 0 and 1 based on the attention map, and weight the displacement to obtain the motion vector between frames 0 and 1; calculate the motion features from frames 0 to t based on the motion vector between frames 0 and 1.

[0021] In some embodiments, the feature maps of each layer of the appearance encoder and the motion encoder are concatenated with the feature maps of each layer of the decoder.

[0022] According to the second aspect of this specification, a video anomaly detection method is provided, the method comprising:

[0023] Acquire raw video data and extract video frame sequences from the raw video data;

[0024] A video frame sequence of a certain length is input into a video anomaly detection model, which outputs a predicted frame for the next frame of the video frame sequence; the video anomaly detection model is trained according to the training method of the video anomaly detection model described above.

[0025] The next frame of the video sequence is compared with the predicted frame. If the difference between the two is greater than a second threshold, the next frame of the video frame sequence is judged as an abnormal frame.

[0026] In some embodiments, inputting a video frame sequence of a certain length into a video anomaly detection model and outputting a predicted frame for the next frame of the video frame sequence includes:

[0027] A video frame sequence of a certain length is input into the appearance encoder and the motion encoder, and the appearance encoder and the motion encoder respectively extract and output the appearance features and motion features of the video frame sequence;

[0028] The appearance features of the video frame sequence are input into the memory enhancement module. The prototype feature that is most similar to the appearance features of the video frame sequence in the memory module is retrieved. The appearance features of the video frame sequence are adjusted based on the prototype feature in the memory enhancement module, and the adjusted appearance features are output.

[0029] The adjusted appearance features and the motion features of the video frame sequence are input into the consistency fusion module, and the final features are output.

[0030] The final features are input into the decoder, which outputs the predicted frame for the next frame of the video frame sequence.

[0031] In some embodiments, the next frame of the video sequence is compared with the predicted frame, and when the difference between the two is greater than a second threshold, the next frame of the video frame sequence is determined to be an abnormal frame, including:

[0032] The peak signal-to-noise ratio (PSNR) of the predicted frame and the next frame of the video sequence is calculated using the following formula:

[0033]

[0034] in, Indicates the predicted frame, I t N represents the next frame of the video sequence, and N represents I. t The number of pixels in This represents the peak signal-to-noise ratio between the predicted frame and the next frame of the video sequence;

[0035] The difference value between the appearance features of the video frame sequence and its most similar prototype feature is calculated using the following formula:

[0036]

[0037] Where D(q,p) represents the difference between the appearance features of the video frame sequence and its most similar prototype feature, K represents the number of appearance features, ||·||² represents the Euclidean distance, and q k Let represent the kth appearance feature, and pf represent the prototype feature that is most similar to the appearance features of the video frame sequence;

[0038] The peak signal-to-noise ratio and the difference value are normalized to calculate the anomaly score, which is calculated using the following formula:

[0039]

[0040] Where λ represents the balance coefficient, g(·) represents the normalization function, and S t Indicates abnormal scores. The peak signal-to-noise ratio of the predicted frame to the next frame of the video sequence is represented by D(q,p), which represents the difference between each of the appearance features and the retrieved prototype features.

[0041] The abnormal score and the second threshold are compared. If the abnormal score is greater than the second threshold, the next frame of the video frame sequence is judged as an abnormal frame. Attached Figure Description

[0042] The above and other objects, features and advantages of the embodiments of this specification will become readily understood by reading the following detailed description with reference to the accompanying drawings.

[0043] Several embodiments of this specification are illustrated in the accompanying drawings by way of example and not limitation.

[0044] Figure 1 A flowchart illustrating a training method for a video anomaly detection model according to an embodiment of this specification is shown.

[0045] Figure 2 A flowchart of a video anomaly detection method according to an embodiment of this specification is shown;

[0046] Figure 3 The ROC curves for training and testing a video anomaly detection model using the CUHKAvenue dataset are shown.

[0047] Figure 4 The ROC curves for training and testing a video anomaly detection model using the ShanghaiTech dataset are shown. Detailed Implementation

[0048] Preferred embodiments of this specification will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of this specification are shown in the drawings, it should be understood that this specification may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this specification will be thorough and complete, and will fully convey the scope of this specification to those skilled in the art.

[0049] The term "comprising" and its variations as used herein signify an open-ended inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". Terms such as "upper", "lower", "front", "rear", etc., indicating placement or positional relationship, are based on the orientation or positional relationship shown in the accompanying drawings and are used only for the purpose of describing the principles of this specification, and are not intended to indicate or imply that the elements referred to must have a specific orientation, be constructed or operated in a specific orientation, and therefore should not be construed as limiting this specification.

[0050] The following describes in detail, with reference to the accompanying drawings, a training method for a video anomaly detection model according to an embodiment of this specification. Figure 1 A flowchart illustrating a training method for a video anomaly detection model according to an embodiment of this specification is shown. The training method for the video anomaly detection model according to an embodiment of this specification includes:

[0051] S1. Obtain sample video frame sequences and a video anomaly detection model; the video anomaly detection model includes an appearance encoder, a motion encoder, a consistency fusion module, and a decoder.

[0052] S3. Input frames 0 to t-1 from the sample video frame sequence into the video anomaly detection model and output the predicted frame of frame t.

[0053] S5. Use the first loss function to determine the convergence of the predicted frame and the sample frame of frame t, and optimize the model parameters.

[0054] S7. Repeat steps S3 and S5 until the first loss function converges, and the video anomaly detection model is trained.

[0055] In step S1, the sample video frame sequence is extracted from videos of normal events. For example:

[0056] Fractional frame sequences were extracted from the normal event video portions of three datasets: UCSD Ped2, CUHKAvenue, and ShanghaiTech, and used as sample video frame sequences for training. The UCSD Ped2 and CUHKAvenue datasets are applicable to single scenes, with both training and test videos within the same scene. Anomalous events in the UCSD Ped2 dataset primarily refer to non-pedestrian events such as bicycles and skateboards; anomalous events in the CUHKAvenue dataset mainly include object throwing, pushing bicycles, and running; the ShanghaiTech dataset involves more scenes and anomalous types, divided into 330 training videos and 107 test videos, with anomalous behaviors in the test videos including driving cars, skateboarding, falling, and running.

[0057] Both the appearance encoder and the motion encoder are three-layer networks. The appearance encoder includes two convolutional layers and one pooling layer. The convolutional layers capture local contextual information of video frames and generate lower-dimensional feature vectors through weighted summation to obtain appearance features. The pooling layer performs downsampling to compress the image and reduce computation. The first two layers of the motion encoder consist of a CNN, and the third layer is a Transformer block with embedded inter-frame attention. The CNN layer is used to obtain multi-scale single-frame appearance features, and the single-frame appearance feature information is fused to supplement cross-scale information to obtain cross-scale appearance features. The Transformer block is used to extract motion features based on the cross-scale appearance features. The decoder corresponds to the encoder and also includes a three-layer structure. Each layer contains two convolutional layers and one upsampling layer. The feature maps of each layer of the appearance encoder and the motion encoder are concatenated with the feature maps of each layer of the decoder to alleviate the gradient vanishing problem, reduce training events and computational resource consumption, and increase the model's generalization ability and convergence speed.

[0058] In step S3, before inputting frames 0 to t-1 from the sample video frame sequence into the video anomaly detection model, the sample video frame sequence and the video anomaly detection model are preprocessed and initially set. For example:

[0059] The video frame size in all sample video frame sequences was adjusted to 256×256, and the pixel intensity was normalized to a range of [-1, 1]. The Adam optimizer was used to optimize the video anomaly detection model, with a batch size of 8. Referring to common settings for other frame-based prediction models, the length of the input video frame was set to 4, and the next frame was predicted. The initial learning rate of the video anomaly detection model was set to 2e-4. The training epochs for the three benchmark datasets—UCSD Ped2, CUHKAvenue, and ShanghaiTech—were set to 60, 60, and 10, respectively.

[0060] The sample video frame sequence includes sample frames 0 to t. Frames 0 to t-1 in the sample video frame sequence are used as input to the video anomaly detection model to obtain the predicted frame t. The sample frame t in the sample video frame sequence is compared with the predicted frame to determine the model's prediction level.

[0061] In step S3, frames 0 to t-1 of the sample video frame sequence are input into the video anomaly detection model, and the predicted frame for frame t is output. Specifically, this includes:

[0062] The appearance encoder extracts appearance features from frames 0 to t-1, and the motion encoder uses an inter-frame attention mechanism to extract motion features from frames 0 to t-1. The consistency fusion module fuses the appearance features and the motion features to obtain the final features. The decoder decodes the final features and outputs the predicted frame for frame t.

[0063] The appearance encoder and motion encoder employ ReLU activation at their ends, resulting in numerous zero-valued feature representations in the extracted appearance and motion features. These zero-valued appearance and motion feature representations exhibit high similarity in distribution. The consistency fusion module uses Sigmoid activation to activate appearance features with zero values, outputting them as a value between 0 and 1. The activated output is then multiplied by the corresponding motion feature to preserve conflict information between appearance and motion features. Finally, the previously defined appearance features are added back as input to the decoder.

[0064] The motion encoder uses an inter-frame attention mechanism to extract motion features from frames 0 to t-1, including:

[0065] Extract appearance features from frames 0 to t-1, taking frame 0 as the current frame, using any region in the appearance features of the current frame as the query region, and using the appearance features of adjacent frames as keywords and values ​​to generate an attention map; estimate the displacement of the query region between frames 0 and 1 based on the attention map, and weight the displacement to obtain a motion vector between frames 0 and 1; calculate the motion features from frames 0 to t based on the motion vector between frames 0 and 1. For example:

[0066] Let the appearance features of the two frames be denoted as I0 and I1, and any region of I0 and I1 be represented as... and and All belong to R N×N×C Where i and j represent spatial locations, N represents the size of the region window, and C represents the number of channels in the region.

[0067] The query matrix, keywords, and values ​​are generated using a linear projection matrix, as shown in the following formula:

[0068]

[0069]

[0070]

[0071] in, Represents the query matrix. Indicates keywords, Representing values, W0, W K and W V These represent the linear projection matrices corresponding to the query matrix, key, and value, respectively.

[0072] Using SoftMax to generate attention maps The formula is as follows:

[0073]

[0074] in, This represents an attention graph, where T denotes a transposition. This represents the number of channels in the feature matrix.

[0075] Construct a coordinate graph B∈R H×W×2 H and W represent the length and width of each region, and 2 represents the two numbers indicating relative position. The coordinate graph shows the relative position of each region within the entire video frame, with its center at (0, 0). Adjacent coordinates are weighted to estimate... The approximate corresponding position in I1, then Subtracting the original position from the estimated position in I1 yields... motion vector

[0076] The motion features generated after propagation through linear layers can provide local foreground motion features for the anomaly detection prediction model. Furthermore, non-abnormal behavior is linearly stationary motion, therefore... Multiplying by time t gives an approximation of the time from I0 to I. t motion vector

[0077] Finally, the motion features M generated after being passed through the linear layer will be... 0→t M 1→t and M 2→t The motion feature M, after weighted summation, is used to represent the position at time t. t The formula is as follows:

[0078] M t =α1M 0→t +α2M 1→t +α3M 2→t

[0079] Where α1, α2, and α3 represent the weights of each motion feature, M t The motion features are input to the consistency fusion module.

[0080] In step S3, the video anomaly detection model further includes a memory enhancement module, which stores several prototype features. After the appearance encoder extracts the appearance features from frames 0 to t-1, it retrieves the prototype feature most similar to the appearance features. Based on the similarity probability between the appearance features and the retrieved prototype features, as well as all prototype features, a new appearance feature is generated, and the retrieved prototype features are updated based on the appearance features. The consistency fusion module fuses the motion features and the new appearance features. The prototype features are the appearance features of the sample video frames when no abnormal events occur.

[0081] The memory enhancement module helps to better consider the diversity of normal patterns and reduces the ability of CNNs to represent anomalies. It can better distinguish between abnormal and normal patterns, increasing the accuracy of video anomaly prediction. The memory enhancement module operates by using the extracted appearance features as query features, then using the query features to retrieve and update the normal patterns in the memory module, while simultaneously generating new appearance features. The normal patterns of the new appearance features are enhanced.

[0082] The process of generating new appearance features based on the similarity probabilities of the appearance features and the retrieved prototype features, as well as all prototype features, and updating the retrieved prototype features based on the appearance features includes:

[0083] S31. Calculate the similarity probability between the query feature and the prototype feature, using the following formula:

[0084]

[0085] Where, q k Let p represent the k-th query feature. i Let w represent the i-th prototype feature. k.i q k and p i The similarity probability, where N represents the number of prototype features in the memory module, and T represents the transposition rank.

[0086] S33. Generating new appearance features includes simultaneously updating the retrieved prototype features based on the appearance features, using the following formula:

[0087]

[0088] Among them, F fin Indicates a new appearance feature, Ffin The appearance features are fused by the consistency fusion module.

[0089]

[0090] Among them, U i This represents the query feature set for updating the prototype features, where ||·||2 represents the Euclidean distance. This indicates the updated retrieved prototype features. This indicates the matching probability during the update. Standardization is required when using it.

[0091] In step S3, the retrieved prototype features are updated based on the appearance features, including: updating the retrieved prototype features using the appearance features of the predicted frame of frame t; during the update, a normality score is calculated based on the predicted frame of frame t and the sample frame; when the normality score is lower than a first threshold, the predicted frame of frame t is judged as a normal frame, and the retrieved prototype features are updated to the predicted frame. This judgment step effectively prevents the prototype features in the memory enhancement module from being updated by abnormal data during model training. The formula for the normality score is as follows:

[0092]

[0093]

[0094] Where i and j represent spatial locations. This represents the predicted frame of frame t. Let ||·||2 represent the sample frame of frame t, and let ||·||2 represent the Euclidean distance. Indicates the weight.

[0095] Based on the memory enhancement module, further, in step S5, during the convergence judgment of the predicted frame and sample frame of frame t using the first loss function, the method also includes using the second loss function to judge the similarity between the appearance feature and the retrieved prototype feature, and using the third loss function to judge the difference between the difference between the appearance feature and the retrieved prototype feature and the difference between the appearance feature and the second similar prototype feature; when the first loss function, the second loss function and the third loss function all converge, the video anomaly detection model training is complete.

[0096] In this model, the smaller the first loss function, the more similar the predicted frame and the sample frame of frame t are; the smaller the second loss function, the closer the distance between the appearance feature and the retrieved prototype feature; and the smaller the third loss function, the smaller the difference between the appearance feature and the retrieved prototype feature, and the larger the difference between the appearance feature and the second similar prototype feature. The model considers not only the prediction loss between the predicted frame and the real frame, but also the distance loss between appearance features and prototype features from the memory enhancement module, and the loss due to the distance difference between different prototype features and appearance features, thus increasing the convergence condition and improving the model's accuracy. The formulas for the first, second, and third loss functions are as follows:

[0097]

[0098]

[0099]

[0100] Among them, L fra Let L represent the first loss function. c Let L represent the second loss function. d Let ||·||2 represent the third loss function, and let ||·||2 represent the Euclidean distance. Let I represent the predicted frame of frame t, and let q represent the sample frame of frame t. k p represents the k-th appearance feature, where K represents the number of appearance features. f p represents the retrieved prototype features. s This represents the second similar prototype feature, where α is a constant.

[0101] Furthermore, the first loss function, the second loss function, and the third loss function also satisfy the following conditions during training:

[0102] L = L fra +η c L c +η d L d

[0103] Where L represents the comprehensive loss function, L fra Let L represent the first loss function. c The second loss function, L d Let η represent the third loss function. c η represents the balance parameter of the second loss function. dThe parameter represents the balance parameter of the third loss function. The video anomaly detection model is considered complete when the comprehensive loss function converges. This balance parameter is introduced to balance the proportions of the second and third loss functions in the comprehensive loss function, allowing for flexible adjustment of the importance of the first, second, and third loss functions.

[0104] This specification also provides video anomaly detection methods in its embodiments, such as... Figure 3 As shown, the method includes:

[0105] S01. Obtain the raw video data and extract the video frame sequence from the raw video data;

[0106] S03. Input a video frame sequence of a certain length into the video anomaly detection model and output the predicted frame of the next frame of the video frame sequence; the video anomaly detection model is trained according to the training method of the video anomaly detection model described above.

[0107] S05. Compare the next frame of the video sequence with the predicted frame. If the difference between the two is greater than the second threshold, then the next frame of the video frame sequence is judged as an abnormal frame.

[0108] Step S03 is the application step of the video anomaly detection model, specifically including:

[0109] S030. Input a video frame sequence of a certain length into the appearance encoder and the motion encoder, and the appearance encoder and the motion encoder respectively extract and output the appearance features and motion features of the video frame sequence;

[0110] S032. Input the appearance features of the video frame sequence into the memory enhancement module, retrieve the prototype feature in the memory module that is most similar to the appearance features of the video frame sequence, adjust the appearance features of the video frame sequence based on the prototype feature in the memory enhancement module, and output the adjusted appearance features.

[0111] S034. Input the adjusted appearance features and the motion features of the video frame sequence into the consistency fusion module, and output the final features;

[0112] S036. Input the final feature into the decoder and output the predicted frame of the next frame of the video frame sequence.

[0113] In step S05, the next frame of the video sequence is compared with the predicted frame. If the difference between the two is greater than a second threshold, the next frame of the video frame sequence is determined to be an abnormal frame. Specifically, this includes:

[0114] S050. Calculate the peak signal-to-noise ratio (PSNR) of the predicted frame and the next frame of the video sequence. The PNR formula is as follows:

[0115]

[0116] in, Indicates the predicted frame, I t N represents the next frame of the video sequence, and N represents I. t The number of pixels in This represents the peak signal-to-noise ratio between the predicted frame and the next frame of the video sequence;

[0117] S052. Calculate the difference value between the appearance features of the video frame sequence and its most similar prototype features. The formula for the difference value is as follows:

[0118]

[0119] Where D(q,p) represents the difference between the appearance features of the video frame sequence and its most similar prototype feature, K represents the number of appearance features, ||·||² represents the Euclidean distance, and q k p represents the k-th appearance feature. f The prototype feature that is most similar to the appearance features of the video frame sequence;

[0120] S054. Normalize the peak signal-to-noise ratio and the difference value, and calculate the anomaly score. The formula for the anomaly score is as follows:

[0121]

[0122] Where λ represents the balance coefficient, g(·) represents the normalization function, and S t Indicates abnormal scores. The peak signal-to-noise ratio of the predicted frame to the next frame of the video sequence is represented by D(q,p), which represents the difference between each of the appearance features and the retrieved prototype features.

[0123] S056. Determine the difference between the abnormal score and the second threshold. If the abnormal score is greater than the second threshold, then the next frame of the video frame sequence is determined to be an abnormal frame.

[0124] In step S052, the appearance features of the video frame sequence and its most similar prototype features refer to the appearance features of the video frame sequence extracted by the video detection model during application, and the prototype features most similar to the appearance features of the video frame sequence retrieved in the memory enhancement module.

[0125] The training method and evaluation criterion for the video anomaly detection model in the embodiments of this specification are based on AUC (Area Under the Curve). AUC is a metric used to evaluate the performance of a binary classifier. It represents the area between the x-axis and the line segment connecting the probability of correctly classifying a positive sample and the probability of misclassifying a negative sample, at a certain probability threshold. Classification performance is positively correlated with the AUC value; AUC = 1 indicates the best performance, with no errors. Therefore, it can be considered that the higher the AUC value, the more accurate the algorithm and model are in identifying anomalies.

[0126] Table 1 shows a comparison of the embodiments in this specification with the AUG of several recently disclosed anomaly detection methods.

[0127] Table 1: Comparison of AUG values ​​between the video anomaly detection method of this specification and several recently disclosed anomaly detection methods.

[0128]

[0129] As shown in Table 1, this invention improves anomaly detection accuracy to a certain extent. Specifically, compared with the memAE method and the Frame-pred method, the AUC of this invention is improved by 3.6%, 6.6%, and 2.8% and 2.5%, 5.4%, and 1.2% respectively on the three benchmark datasets.

[0130] To further illustrate that each part proposed in the embodiments of this specification is an effective module, ablation experiments were conducted, as shown in Table 2. Adding a motion encoder but not using the consistency fusion module resulted in a certain improvement in baseline performance. Adding a motion encoder while using the consistency fusion module significantly improved the baseline performance.

[0131] Table 2: Ablation Experiment Results

[0132]

[0133] like Figure 3 and Figure 4 As shown, the ROC curves on the CUHKAvenue and ShanghaiTech datasets also demonstrate that the video anomaly detection method of the embodiments in this specification is superior to other methods.

[0134] While several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this specification. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0135] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

[0136] Various embodiments of this specification have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technological improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A training method for a video anomaly detection model, characterized in that, include: Acquire sample video frame sequences and a video anomaly detection model; the video anomaly detection model includes an appearance encoder, a motion encoder, a consistency fusion module, and a decoder. Frames 0 to t-1 from the sample video frame sequence are input into the video anomaly detection model, and the predicted frame for frame t is output, specifically including: The appearance encoder extracts appearance features from frames 0 to t-1, and the motion encoder uses an inter-frame attention mechanism to extract motion features from frames 0 to t-1. The consistency fusion module fuses the appearance features and the motion features to obtain the final features. The decoder decodes the final features and outputs the predicted frame for frame t. The first loss function is used to determine the convergence of the predicted frame and the sample frame of frame t, and the model parameters are optimized. Repeat the above video anomaly detection model prediction steps and convergence judgment steps until the first loss function converges, and the video anomaly detection model is trained. The first two layers of the motion encoder consist of a CNN, and the third layer is a Transformer block with embedded inter-frame attention. The video anomaly detection model further includes a memory enhancement module, which stores several prototype features. After the appearance encoder extracts appearance features from frames 0 to t-1, it retrieves the prototype feature most similar to the appearance features. Based on the similarity probability between the appearance features and the retrieved prototype features, as well as all prototype features, a new appearance feature is generated, and the retrieved prototype features are updated based on the appearance features. The consistency fusion module fuses the motion features and the new appearance features. The prototype features are the appearance features of the sample video frames when no anomaly events occur. During the convergence determination of the predicted frame and sample frame of frame t using the first loss function, the method further includes using the second loss function to determine the similarity between the appearance feature and the retrieved prototype feature, and using the third loss function to determine the difference between the difference between the appearance feature and the retrieved prototype feature and the difference between the appearance feature and the second similar prototype feature; when the first loss function, the second loss function and the third loss function all converge, the video anomaly detection model is trained.

2. The method according to claim 1, characterized in that, The first loss function, the second loss function, and the third loss function must also satisfy the following conditions during training: Where L represents the comprehensive loss function, Denotes the first loss function. The second loss function, This represents the third loss function. This represents the balance parameter of the second loss function. The third loss function represents the balance parameter. When the comprehensive loss function converges, the video anomaly detection model is trained.

3. The method according to claim 1, characterized in that, Updating the retrieved prototype features based on the appearance features includes: updating the retrieved prototype features using the appearance features of the predicted frame of frame t; during the update, a normality score is calculated based on the predicted frame of frame t and the sample frame; when the normality score is lower than a first threshold, the predicted frame of frame t is judged as a normal frame, and the retrieved prototype features are updated to the predicted frame.

4. The method according to claim 1, characterized in that, The motion encoder uses an inter-frame attention mechanism to extract motion features from frames 0 to t-1, including: Extract the appearance features of frames 0 to t-1, take frame 0 as the current frame, any region in the appearance features of the current frame as the query region, and the appearance features of the adjacent frames of the current frame as keywords and values ​​to generate an attention map. Based on the attention map, the displacement of the query region between frames 0 and 1 is estimated, and the displacement is weighted to obtain the motion vector between frames 0 and 1. Based on the motion vectors between frames 0 and 1, calculate the motion features of frames 0 to t.

5. The method according to claim 1, characterized in that, The feature maps of each layer of the appearance encoder and the motion encoder are concatenated with the feature maps of each layer of the decoder.

6. A video anomaly detection method, characterized in that, include: Acquire raw video data and extract video frame sequences from the raw video data; A video frame sequence of a certain length is input into a video anomaly detection model, which outputs a predicted frame for the next frame of the video frame sequence; the video anomaly detection model is trained according to any one of the methods described in claims 1-3. The next frame of the video sequence is compared with the predicted frame. If the difference between the two is greater than a second threshold, the next frame of the video frame sequence is judged as an abnormal frame.

7. The method according to claim 6, characterized in that, Input a video frame sequence of a certain length into the video anomaly detection model, and output the predicted frame of the next frame in the video frame sequence, including: A video frame sequence of a certain length is input into the appearance encoder and the motion encoder, and the appearance encoder and the motion encoder respectively extract and output the appearance features and motion features of the video frame sequence; The appearance features of the video frame sequence are input into the memory enhancement module. The prototype feature that is most similar to the appearance features of the video frame sequence in the memory module is retrieved. The appearance features of the video frame sequence are adjusted based on the prototype feature in the memory enhancement module, and the adjusted appearance features are output. The adjusted appearance features and the motion features of the video frame sequence are input into the consistency fusion module, and the final features are output. The final features are input into the decoder, which outputs the predicted frame for the next frame of the video frame sequence.

8. The method according to claim 7, characterized in that, The next frame of the video sequence is compared with the predicted frame. If the difference between the two is greater than a second threshold, the next frame of the video frame sequence is determined to be an abnormal frame, including: The peak signal-to-noise ratio (PSNR) of the predicted frame and the next frame of the video sequence is calculated using the following formula: in, Indicates the predicted frame. N represents the next frame of the video sequence. The number of pixels in This represents the peak signal-to-noise ratio between the predicted frame and the next frame of the video sequence; The difference value between the appearance features of the video frame sequence and its most similar prototype feature is calculated using the following formula: in, K represents the difference between the appearance features of the video frame sequence and its most similar prototype feature, where K represents the number of appearance features. Represents Euclidean distance. This represents the k-th appearance feature. The prototype feature that is most similar to the appearance features of the video frame sequence; The peak signal-to-noise ratio and the difference value are normalized to calculate the anomaly score, which is calculated using the following formula: in, Represents the balance coefficient. Represents the normalization function. Indicates abnormal scores. This represents the peak signal-to-noise ratio (PSNR) between the predicted frame and the next frame of the video sequence. This represents the difference value between each of the appearance features and the retrieved prototype features; The abnormal score and the second threshold are compared. If the abnormal score is greater than the second threshold, the next frame of the video frame sequence is judged as an abnormal frame.

Images