Two-stage end memory for video anomaly detection and video anomaly detection apparatus

Abstract

The embodiment of the application discloses a two-order end memory and a video anomaly detection device for video anomaly detection, which comprises a first-order network and a second-order network; the first-order network comprises a first-order encoder, a motion memory and a first-order decoder connected in sequence, is used for motion reconstruction on a motion part, and obtains a motion prototype; the second-order network comprises a forward process network and a reverse process network connected in series; the forward process network is used for realizing a forward process, and the forward process network comprises a second-order encoder and a second-order decoder, which are used for predicting a predicted future frame of an input video; the reverse process network is cascaded with the forward process network, realizes a reverse process, and reconstructs a predicted initial frame of the input video; the predicted future frame and the predicted initial frame are used for unsupervised anomaly detection, and an anomaly detection result of the input video is output. The two-order end memory is complete in motion representation, fully utilizes the bidirectional consistency of the video, and improves the accuracy of anomaly detection.

Description

Technical Field

[0001] This invention relates to the field of video anomaly detection technology, and in particular to a two-stage terminal memory and a video anomaly detection device for video anomaly detection. Background Technology

[0002] Video anomaly detection is generally considered an unsupervised task. Existing unsupervised video anomaly detection methods fuse memory networks or dual-branch networks into unsupervised models. Fusing memory networks allows the recording of prototypes of normal events, thereby reducing the network's generalization ability to anomalies and improving the performance of video anomaly detection. However, it is usually insufficient to capture all the motion of normal events. Fusing dual-branch networks allows the use of a separate motion branch to capture motion information, but training both branches simultaneously inevitably leads to mutual interference between the motion and appearance branches, limiting the representational power of motion prototypes.

[0003] Furthermore, existing unsupervised video anomaly detection methods focus on feature extraction while neglecting the bidirectional consistency of videos. Even when bidirectional consistency is utilized, as in "Z. Fang, J. Liang, JT Zhou, Y. Xiao, and F. Yang, "Anomaly detection with bidirectional consistency in videos," IEEE Trans. Neural Netw. Learn. Syst., vol. 33, no. 3, pp. 1079–1092, 2022", multiple predicted future frames are usually required as input, which is impossible in real-time. Summary of the Invention

[0004] Based on this, the present invention provides a two-stage end-to-end memory and a video anomaly detection device for video anomaly detection, in order to solve the problems that the unsupervised models used in existing unsupervised video anomaly detection methods cannot capture all the motion of normal events and cannot flexibly utilize the bidirectional consistency of video.

[0005] Firstly, a two-order end memory for video anomaly detection is provided, comprising a first-order network and a second-order network;

[0006] The first-order network is used to receive the motion part of the input video after decomposition;

[0007] The first-order network includes a first-order encoder, a motion memory, and a first-order decoder connected in sequence, used to reconstruct motion of the motion part and obtain a motion prototype; the motion prototype is a permanent prototype that is updated once.

[0008] The second-order network includes a cascaded forward process network and a backward process network;

[0009] The forward process network is used to implement the forward process. The forward process network includes a second-order encoder and a second-order decoder. The second-order encoder is used to receive the scene part and object part after the input video is decomposed and to fuse the features of the scene part and object part. The second-order decoder is used to predict the predicted future frame of the input video.

[0010] The reverse process network is cascaded with the forward process network to realize the reverse process. The reverse process network includes a second-order reverse encoder, a spatiotemporal memory and a second-order reverse decoder connected in sequence, which are used to reconstruct the prediction initial frame of the input video based on the predicted future frame, the input video and the motion prototype.

[0011] The predicted future frame and the predicted initial frame are used for unsupervised anomaly detection, and the anomaly detection results of the input video are output.

[0012] Optionally, the forward process network has a symmetrical structure, the first-order encoder includes a forward appearance encoder and a forward motion encoder, and the second-order decoder is a forward appearance decoder.

[0013] The reverse process network has a symmetrical structure. The second-order reverse encoder includes a reverse appearance encoder and a reverse motion encoder, and the second-order reverse decoder is a reverse appearance decoder.

[0014] Optionally, the forward appearance encoder extracts fine-grained features of the scene portion, and the forward motion encoder extracts fine-grained features of the object portion.

[0015] After the fine-grained features of the scene part and the fine-grained features of the object part are fused, they are input into the forward appearance decoder to predict the predicted future frame of the input video.

[0016] Optionally, the reverse appearance encoder obtains the reverse sequence of the intermediate video, wherein the intermediate video is video data based on the original initial frame of the input video with the predicted future frames deleted and merged.

[0017] The reverse motion encoder calculates the RGB difference between the intermediate video and the reverse sequence;

[0018] The reverse appearance decoder acquires the motion prototype. The reverse appearance decoder includes a network that predicts the initial frame. The RGB difference between the intermediate video and the reverse sequence, as well as the motion prototype, are used as inputs to the network that predicts the initial frame to obtain the predicted initial frame.

[0019] Optionally, the spatiotemporal memory learns and records the video spatiotemporal prototype, and updates the video spatiotemporal prototype based on the RGB difference between the intermediate video and the reverse sequence;

[0020] The reverse appearance decoder fuses the motion features of the motion prototype and the motion features learned from the video spatiotemporal prototype, and uses them as input to the network that predicts the initial frame.

[0021] Optionally, the spatiotemporal memory is implemented as a matrix, and the updated video spatiotemporal prototype is calculated through similarity.

[0022] Optionally, the reverse appearance encoder and the reverse motion encoder are connected in parallel and then connected to the spatiotemporal memory, which is then connected to the reverse appearance decoder.

[0023] Optionally, the capacity of both the spatiotemporal memory and the motion memory is 30.

[0024] Optionally, the second-order encoder and the second-order decoder are improved structures based on the Unet structure, using stride convolutions instead of pooling layers, and hop connections are introduced between the second-order encoder and the second-order decoder;

[0025] The second-order inverse encoder and the second-order inverse decoder have the same structure as the second-order encoder and the second-order decoder.

[0026] In a second aspect, a video anomaly detection device is provided, comprising a two-stage terminal memory for video anomaly detection as described in any one of claims 1 to 9 and an unsupervised anomaly detection module;

[0027] The unsupervised anomaly detection module is used to perform unsupervised anomaly detection based on the predicted future frame and the predicted initial frame, and output the anomaly detection results of the input video.

[0028] The aforementioned two-stage end-to-end memory and video anomaly detection device for video anomaly detection divides video anomaly detection into two main stages: a first-stage network based on the motion part and a second-stage network based on the scene and object parts. This approach processes the motion part, scene part, and object part separately. The motion part-based stage separately records the motion prototype of the motion part, overcoming the problem that the motion prototype in the fused prototype is often insufficient to capture all the motion of a normal event, resulting in inadequate representation of the overall fused prototype. The scene and object part-based stage includes a forward process network and a backward process network, which are cascaded together to predict future frames and the initial frame. Finally, the anomaly scores from the predicted future frames and the predicted initial frames are used to jointly evaluate the anomaly. In the forward processing network, the predicted future frames of the input video are predicted by fusing features from the scene and object parts. Instead of encoding the original frames of the input video, it encodes the scene and object separately and then fuses them to obtain the predicted future frames. This helps extract fine-grained video features, resulting in predicted future frames containing more details. In the backward processing network, the input video is encoded based on the predicted future frames, and the initial frames are predicted through motion prototype enhancement. This fully utilizes the inherent property of video, namely bidirectional consistency, significantly improving the accuracy of video frame prediction and thus enhancing the performance of video anomaly detection. Furthermore, the cascaded processing of the forward and backward processing networks increases the accumulation of anomalies, reducing the predictive ability for anomalous frames. This increases the prediction gap between normal and anomalous frames, improving the performance of the two-stage end-memory used for video anomaly detection, including but not limited to the accuracy of video anomaly detection. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the structure of a two-stage end-to-end memory used for video anomaly detection according to an embodiment of the present invention;

[0031] Figure 2 This is an anomaly score curve and partial samples of a two-order end-memory system used for video anomaly detection in an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram illustrating the scores of four training strategies in an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram showing different locations of the spatiotemporal memory in a two-level end memory used for video anomaly detection according to an embodiment of the present invention;

[0034] Figure 5 The AUC values ​​for different memory capacities are shown in the embodiments of the present invention.

[0035] Figure 6 This is a schematic diagram illustrating the implementation process of the video anomaly detection method according to an embodiment of the present invention;

[0036] Figure 7 This is a schematic diagram of AUC curves obtained by setting different λ values ​​for three datasets in an embodiment of the present invention. Detailed Implementation

[0037] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0038] In some of the processes described in the specification, claims, and accompanying drawings of this invention, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0041] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0042] Please refer to the details. Figure 1 , Figure 1 This is a schematic diagram of the structure of the two-stage end memory used for video anomaly detection in this embodiment.

[0043] like Figure 1 As shown, a two-stage end-memory system for video anomaly detection includes a first-stage network STAGE1 and a second-stage network STAGE2; the first-stage network STAGE1 includes a first-stage encoder E connected in sequence. m Motion memory M m and first-order decoder D a The second-order network STAGE2 includes a cascaded forward process network P1 and a backward process network P2. The forward process network P1 includes a second-order encoder and a second-order decoder; the backward process network P2 includes a second-order reverse encoder and a second-order reverse decoder connected in sequence, and also includes a spatiotemporal memory M. st .

[0044] The second-order encoder and decoder are improved versions of the UNet (Convolutional Networks for Biomedical Image Segmentation) architecture, using strided convolutions instead of pooling layers and introducing skip connections between them. The second-order inverse encoder and decoder share the same structure as the second-order encoder and decoder; therefore, they are also improved versions of the UNet architecture.

[0045] To improve the performance of the two-stage end memory, a preferred solution is that the forward and reverse process networks are symmetrical structures including dual encoders and a single decoder. Specifically, the two second-order encoders of the forward process network are connected in parallel and then in series with the first-order decoder; similarly, the two second-order reverse encoders of the reverse process network are connected in parallel and then in series with the second-order reverse decoder. Figure 1 The forward process network P1 includes two second-order encoders, namely the forward appearance encoder E. s Along with the forward motion encoder Eo, the second-order decoder is the forward appearance decoder D. fa The reverse process network P2 includes two second-order reverse encoders, namely the reverse appearance encoder E. ba and reverse motion encoder E bmThe second-order inverse decoder is the inverse appearance decoder D. ba .

[0046] To improve the performance of the two-stage end-memory system, a preferred approach is to locate the spatiotemporal memory between the parallel-connected inverse appearance encoder, inverse motion encoder, and inverse appearance decoder in the inverse process network. Specifically, the inverse appearance encoder and inverse motion encoder are connected in parallel to the spatiotemporal memory, which is then connected to the inverse appearance decoder. In this embodiment, the network prediction framework in the two-stage end-memory system focuses more on appearance features, so the impact of not updating the appearance is less than the impact of not updating the motion. Therefore, the spatiotemporal memory has a greater impact on the results in the second stage than in the first stage. By placing the spatiotemporal memory in the second stage and learning appearance and motion, this embodiment can improve the video anomaly detection performance of the two-stage end-memory system.

[0047] The working principle of the two-level terminal memory used for video anomaly detection in this embodiment of the invention is as follows:

[0048] Input video decomposed into motion parts Scene S t and object part O t This step is a preprocessing step based on the input video, which can be executed by a two-stage end memory or by other video decomposition software. This embodiment of the invention does not limit it.

[0049] It should be noted that, in this embodiment of the invention, the input video is decomposed into motion, scene, and object, rather than appearance and motion. This can reduce the blurring and distortion of the appearance of objects in the prediction frame, thereby improving the ability of the two-level end memory to distinguish between normal and abnormal frames when used for unsupervised anomaly detection.

[0050] A first-order network STAGE1 is used to receive the input video I. 1:t Decomposed moving parts First-order encoder E of first-order network STAGE1 m Motion memory Mm and first-order decoder D a Used for moving parts Perform motion reconstruction to obtain the motion prototype Among them, the motion prototype is a permanent prototype that is updated once.

[0051] The detailed process of motion reconstruction is as follows: First-order encoder E m First, the movement part Encode, then motion memory M m Based on the encoded result LD, memory is read, and the encoded result LD is updated to obtain the encoded result LD', and the first-order decoder D is used. a Decode the encoded result LD' to obtain the motion prototype.

[0052] Combination Figure 1 The stages implemented by the first-order network STAGE1 described above can be described as follows:

[0053]

[0054] Where E(·), M(·), and D(·) represent encoding, memory read, and decoding operations, respectively.

[0055] It should be noted that motion reconstruction for the motion part is an update of the original motion prototype. The motion prototype is a one-time update permanent prototype, which means that when the predicted future frame and the predicted initial frame of this input video are obtained through the two-level end memory, the motion prototype obtained in the first update is no longer updated.

[0056] In this embodiment of the invention, the motion part is processed separately, which can utilize more motion information, has a strong representation capability of the motion prototype, and the motion representation is complete. Furthermore, the motion memory in the first stage is not updated, which can avoid the influence of the appearance branch on the motion memory learning prototype.

[0057] The second-order network STAGE2 is used to implement the forward and backward processes. The forward process is first implemented through the forward process network P1. During the forward process, the second-order encoder is used to receive the scene part and object part after the input video is decomposed. The features of the scene part and object part are fused by the second-order decoder, and the predicted future frames of the input video are predicted.

[0058] Based on the above preferred scheme, the detailed process of predicting future frames of the input video is as follows: Forward appearance encoder E s Receive the scene portion S after decomposing the input video. t Extracting fine-grained features from the scene portion, the forward motion encoder E o Receive the object part O after decomposition of the input video. t Extracting fine-grained features from the object portion, the forward appearance decoder D fa As a second-order decoder, it performs feature fusion and predicts future frames. The prediction, the forward appearance decoder D fa The output is the predicted future frame of the input video.

[0059] Combination Figure 1 The forward process phase implemented by the second-order network STAGE2 described above can be described as follows:

[0060]

[0061] Among them, E s (·), Eo (·) and D fa (·) represent the forward appearance encoder, forward motion encoder, and forward appearance decoder, respectively.

[0062] In the forward process shown above, the original frames of the input video are not encoded. Instead, the scene and objects are encoded separately and then fused to obtain the predicted future frames. This helps to extract fine-grained video features, so that the predicted future frames contain more details.

[0063] The reverse process network P2 is cascaded with the forward process network P1. The reverse process is implemented based on this cascaded structure. The reverse process includes reconstructing the initial frame of the input video based on the predicted future frame, the input video, and the motion prototype.

[0064] Based on the above preferred scheme, the detailed process of reconstructing the initial frame of the input video based on the predicted future frame, the input video, and the motion prototype is as follows:

[0065] Reverse appearance encoder E ba Get intermediate video reverse sequence The intermediate video is the input video I 1:t Delete the original initial frame I1 and delete and merge the predicted future frames Data; Reverse motion encoder E bm Calculate intermediate video and reverse sequence RGB difference Reverse appearance decoder D ba The motion prototype is obtained by cascading with the forward process. Reverse appearance decoder D ba This includes a network for predicting the initial frame, which will be based on intermediate video. and reverse sequence RGB difference and motion prototype As input, to obtain the initial frame for prediction Spacetime Memory M st The above process is used to learn and record the video spatiotemporal prototype, and then the video spatiotemporal prototype is updated by the RGB difference between the intermediate video and the reverse sequence. Then, the reverse appearance decoder D... ba The motion features of the motion prototype and the updated features in the video spatiotemporal prototype are fused and used as input to the network that predicts the initial frame.

[0066] Combination Figure 1 The reverse process stage implemented by the second-order network STAGE2 described above can be described as follows:

[0067]

[0068] Among them, E ba (·), E bm (·), D ba (·) represent the reverse appearance encoder, the reverse motion encoder, and the reverse appearance decoder, respectively. It is motion memory M m The motion characteristics are updated in the first stage.

[0069] In this embodiment of the invention, the spatiotemporal memory of the two-stage end-memory system updates the video spatiotemporal prototype by the RGB difference between the intermediate video and the reverse sequence. In one implementation, the spatiotemporal memory is implemented as a matrix, and the updated video spatiotemporal prototype is calculated using similarity. For example, the spatiotemporal memory is implemented as a matrix. Where N represents the number of entries in the spatiotemporal memory, and d represents the dimension m of each entry. i i = 1, ..., N. Given input features The contents of each memory entry are read sequentially, as expressed by the formula:

[0070]

[0071]

[0072]

[0073] Where d(·, ·) represents the similarity operation. Weight Let ω be the similarity between memory location M and query x. i For the i-th term of w, These are features obtained after the spatiotemporal memory M is updated, i.e., features based on the updated video spatiotemporal prototype.

[0074] It should be noted that the capacity of the memory affects the accuracy of video anomaly detection. In one embodiment, the capacity of the spatiotemporal memory and motion memory is set to 30 to obtain the optimal two-stage end memory performance.

[0075] In the reverse process described above, encoding is performed based on the predicted future frame and the input video, and the initial frame is predicted through motion prototype enhancement. This fully utilizes the inherent property of video, namely bidirectional consistency. Specifically, by inversely encoding the predicted future frame, the new video segment composed of the video segment after removing the first frame, and the RGB differences of the original video segment, and then using motion prototype enhancement to predict the initial frame, detailed spatiotemporal semantic representations and higher-level video spatiotemporal prototypes can be learned, significantly improving the accuracy of video frame prediction and thus enhancing the performance of video anomaly detection. Furthermore, the concatenated forward and reverse processing increases the accumulation of anomalies and reduces the prediction capability of the two-stage end memory for anomalous frames, thereby increasing the prediction gap between normal and anomalous frames and improving the anomaly detection capability.

[0076] Based on the working principle of the two-stage end-to-end memory for video anomaly detection described above, it can be seen that the two-stage end-to-end memory for video anomaly detection in this embodiment of the invention ultimately obtains the predicted future frame and the predicted initial frame. These predicted future frames and the predicted initial frame are used for unsupervised anomaly detection, outputting the anomaly detection result of the input video. Furthermore, the two-stage end-to-end memory for video anomaly detection provided by this embodiment of the invention processes the motion part separately, utilizing more motion information, resulting in a strong and complete representation of the motion prototype. Moreover, by not updating the first-stage motion memory, the influence of the appearance branch on the motion memory's learning prototype can be avoided. Simultaneously, by processing the motion part, scene part, and object part separately, compared to fused prototypes, the motion branch and appearance branch do not affect each other, further avoiding the problem of the two branches mutually influencing each other and limiting the representational capability of the motion prototype, such as normal motion being considered abnormal during evaluation, leading to a high false alarm rate.

[0077] In one embodiment, unsupervised anomaly detection can be achieved by:

[0078] Calculate the block-level peak signal-to-noise ratio of the true value;

[0079] The block-level peak signal-to-noise ratio of the true value is constrained by the maximum-minimum regularization function and smoothed by Gaussian filtering to obtain the anomaly score for predicting future frames and the anomaly score for predicting the initial frame.

[0080] The anomaly score is obtained by combining the anomaly score of the predicted future frame with the anomaly score of the predicted initial frame.

[0081] When the final anomaly score is greater than a preset value, the existence of an anomaly will be output as the anomaly detection result of the input video.

[0082] For example, the calculation process is as follows:

[0083] The block-level peak signal-to-noise ratio of the true value was calculated using the following formula:

[0084]

[0085] Where i = 1 and 5 represent the initial frame and the future frame, respectively. i and This represents the actual value and the predicted value. This represents the maximum prediction error at the block level.

[0086] The block-level peak signal-to-noise ratio (PSNR) of the true value is constrained to the range of [0,1] using a maximum-minimum regularization function, and then smoothed using a Gaussian filter to obtain the anomaly scores for predicting future frames and the anomaly scores for predicting the initial frame:

[0087]

[0088] Where G(·) is the Gaussian filtering operation.

[0089] The anomaly score S of the predicted future frames f With the initial frame S i Combining these, we obtain the final abnormality score: S = S f +λS i ;

[0090] λ is designed to balance the contributions of anomaly scores from future frames and the initial frame.

[0091] It is understood that unsupervised anomaly detection is not limited to the above-mentioned methods, and can also be other unsupervised anomaly detection methods based on predicting future frames and predicting initial frames. This embodiment does not limit them.

[0092] To address the aforementioned technical problems, this invention also provides a video anomaly detection device, comprising a two-stage terminal memory for video anomaly detection as described in the various embodiments above, and an unsupervised anomaly detection module. The unsupervised anomaly detection module performs unsupervised anomaly detection based on predicted future frames and predicted initial frames, and outputs the anomaly detection result of the input video. The unsupervised anomaly detection method can be the block-level peak signal-to-noise ratio detailed steps shown in the above embodiments, or other unsupervised anomaly detection methods based on predicted future frames and predicted initial frames; this embodiment does not limit such methods.

[0093] To address the aforementioned technical problems, embodiments of the present invention may also provide a video anomaly detection method, such as... Figure 6 As shown, it includes:

[0094] S101. Obtain the motion part, scene part, and object part of the input video after decomposition;

[0095] Before step S101, the input video is further decomposed into motion parts, scene parts, and object parts. This step is a preprocessing step based on the input video, which can be executed by a two-stage end memory or by other video decomposition methods. This embodiment of the invention does not limit it.

[0096] It should be noted that, in this embodiment of the invention, the input video is decomposed into motion, scene, and object, rather than appearance and motion. This can reduce the blurring and distortion of the appearance of objects in the prediction frame, thereby improving the ability to distinguish between normal and abnormal frames when performing unsupervised anomaly detection in step S104 below.

[0097] S102. Perform motion reconstruction on the moving parts to obtain the motion prototype;

[0098] The motion prototype is a permanent prototype that is updated once.

[0099] It should be noted that the motion reconstruction of the motion part in step S102 is an update of the original motion prototype. The motion prototype is a one-time update permanent prototype, which means that during the execution of steps S101 to S104 on the input video, the motion prototype obtained in the first update in step S102 will not be updated again.

[0100] In this embodiment of the invention, the motion prototype of the motion part is recorded separately. Compared with the fused prototype, it can extract comprehensive motion features and reconstruct complete motion segments. However, the motion prototype part in the fused prototype is usually insufficient to capture all the motion of a normal event, and the overall fused prototype is insufficient in representation.

[0101] S103. For the scene part and the object part, design a forward process and a backward process. In the forward process, predict the predicted future frame of the input video by fusing the features of the scene part and the object part. In the backward process, cascade with the forward process to reconstruct the predicted initial frame of the input video based on the predicted future frame, the input video and the motion prototype.

[0102] In the forward pass shown in step S103 above, the original frames of the input video are not encoded. Instead, the scene and objects are encoded separately and then fused to obtain the predicted future frames. This helps to extract fine-grained video features, resulting in the predicted future frames containing more details. In the backward pass shown in step S103 above, encoding is performed based on the predicted future frames and the input video. Motion prototype enhancement is used to predict the initial frames, fully utilizing the inherent properties of video, namely bidirectional consistency, which significantly improves the accuracy of video frame prediction and thus enhances the performance of video anomaly detection. Furthermore, the concatenated forward and backward processing increases the accumulation of anomalies, reduces the network's ability to predict anomalous frames, thereby increasing the prediction gap between normal and anomalous frames and improving the anomaly detection capability.

[0103] In steps S102 and S103 above, the motion part, scene part, and object part are processed separately. In one embodiment, the above content is implemented through two branches of a two-level terminal memory. That is, the motion reconstruction of the motion part and the forward and backward processes designed for the scene and object parts in steps S102 and S103 are executed through two branches of the two-level terminal memory. Compared with the fusion prototype, this implementation method ensures that the motion branch and appearance branch do not affect each other, further avoiding the problem of the two branches affecting each other and limiting the representation capability of the motion prototype. For example, normal motion may be considered abnormal during evaluation, leading to a high false alarm rate.

[0104] This invention processes the motion component separately, allowing for the utilization of more motion information. The motion prototype has strong representation capabilities and complete motion representation. Furthermore, it avoids updating the motion memory in the first stage, thus preventing the appearance branch from affecting the motion memory's learning prototype. Simultaneously, by processing the motion component, scene component, and object component separately, compared to a fused prototype, the motion branch and appearance branch do not interfere with each other. This further avoids the problem of the two branches influencing each other and limiting the representation capabilities of the motion prototype, such as normal motion being considered abnormal during evaluation, leading to a high false alarm rate.

[0105] In one embodiment, the reverse process of step S103 is implemented in detail as follows:

[0106] The original initial frame of the input video is deleted, and the predicted future frames are merged to obtain the intermediate video;

[0107] Obtain the reverse sequence of the intermediate video;

[0108] Calculate the RGB difference between the intermediate video and the reverse sequence;

[0109] Obtain the motion prototype;

[0110] The network that uses the RGB difference between the intermediate video and the reverse sequence, along with the motion prototype, as input to predict the initial frame is used to obtain the predicted initial frame.

[0111] The above steps, by inversely encoding the RGB differences between the predicted future frames, the new video segment composed of the video segment after removing the first frame, and the original video segment, can learn detailed spatiotemporal semantic representations and higher-level video spatiotemporal prototypes. Simultaneously, motion prototype enhancement is used to predict features for the initial frame. Based on this, in one embodiment, the RGB difference based on the intermediate video and the inverse sequence, along with the motion prototype, is used as input to the network predicting the initial frame, including:

[0112] In the reverse process, the video spatiotemporal prototype is learned and recorded;

[0113] The video spatiotemporal prototype is updated based on the RGB difference between the intermediate video and the reverse sequence;

[0114] The motion features of the motion prototype and the motion features learned from the video spatiotemporal prototype are fused and used as input to the network that predicts the initial frame.

[0115] S104. Perform unsupervised anomaly detection based on the predicted future frame and the predicted initial frame, and output the anomaly detection result of the input video.

[0116] In one embodiment, unsupervised anomaly detection is performed based on the predicted future frame and the predicted initial frame, and the anomaly detection result of the input video is output, including:

[0117] Calculate the block-level peak signal-to-noise ratio of the true value;

[0118] The block-level peak signal-to-noise ratio of the true value is constrained by the maximum and minimum regularization functions and smoothed by Gaussian filtering to obtain the anomaly score for predicting future frames and the anomaly score for predicting the initial frame.

[0119] The anomaly score is obtained by combining the anomaly score of the predicted future frame with the anomaly score of the predicted initial frame.

[0120] When the final anomaly score is greater than a preset value, the existence of an anomaly will be output as the anomaly detection result of the input video.

[0121] For example, the calculation process is as follows:

[0122] The block-level peak signal-to-noise ratio of the true value was calculated using the following formula:

[0123]

[0124] Where i = 1 and 5 represent the initial frame and the future frame, respectively. i and This represents the actual value and the predicted value. This represents the maximum prediction error at the block level.

[0125] The block-level peak signal-to-noise ratio (PSNR) of the true value is constrained to the range [0,1] using maximum and minimum regularization functions, and then smoothed using Gaussian filtering to obtain the anomaly scores for predicting future frames and the anomaly scores for predicting the initial frame:

[0126]

[0127] Where G(·) is the Gaussian filtering operation.

[0128] The anomaly score S of the predicted future frames fWith the initial frame S i Combining these, we obtain the final abnormality score: S = S f +λS i ;

[0129] In this process, incorporating the anomaly score from the initial prediction frame can improve prediction accuracy to some extent. However, the anomaly score corresponds to the predicted true value in future frames. When the anomaly score from the initial prediction frame accounts for a large proportion of the overall anomaly score, it will affect the final video anomaly detection performance. Therefore, a hyperparameter λ is set to balance the contributions of the anomaly scores from the predicted future frames and the predicted initial frame. In one embodiment, λ = 0.2.

[0130] like Figure 7 As shown in the figure, this embodiment of the invention illustrates an AUC comparison across three datasets with different λ settings. The λ range is set from 0 to 1.0, with intervals of 0.1. It can be observed that a consistent trend is present across the three datasets: the AUC values ​​initially increase or remain stable, then gradually decrease; when λ = 0.2, the AUC is relatively high in all three datasets.

[0131] This invention also provides a qualitative and quantitative comparison of the provided two-stage end-memory for video anomaly detection with existing methods. Experiments were conducted on three datasets: UCSDped2, CHUK Avenue, and ShanghaiTech. UCSDped2 is a small dataset containing 16 training videos and 12 test videos. The training videos only include normal walking activities. In addition to normal walking, the test videos include 12 anomalous events, such as cycling, trucking, and skateboarding. CHUK Avenue is another small dataset containing 16 training videos and 21 test videos. The training videos only contain normal walking events, while the test video dataset includes a wider range of scenarios, including 47 different anomalous events, such as running, throwing bags and books. ShanghaiTech is a medium-sized and challenging dataset, containing 330 training videos and 107 test videos from 13 different scenarios. The test videos include 130 anomalous items, such as fighting, running, skating, and cycling.

[0132] First, the resolution of all frames was scaled to 256×256 and their pixel intensity was normalized to [1,1]. Then the video sequence was decomposed into motion, scene and object, and the lower and upper thresholds of the decomposition were set to 0.1 and 0.9, respectively. The number and size of the memory were set to 30 and 512, respectively. The Adam optimizer with β1=0.9 and β2=0.9 was used, and the initial learning rate was set to 5e-5 by cosine annealing. The training time, error map size and step size of UCSD Ped2, CUHKAvenue and ShanghaiTech were set to (60,100,10), (32,64,64) and (4,8,4), respectively. The hyperparameter λ was set to 0.2. All models were implemented on NVIDIA RTX 4090 using the PyTorch

[40] framework.

[0133] In this embodiment of the invention, the frame-level AUC (area under the curve) is used as the evaluation metric. The higher the AUC value, the stronger the model's ability to predict positive examples and the better the video anomaly detection performance.

[0134] 1) Quantitative results:

[0135] Table 1 below shows a comparison of frame-level AUC (%) with other existing methods. Bold and underlined values ​​indicate the best and second-best results, respectively.

[0136]

[0137]

[0138]

[0139] Table 1

[0140] Table 1 above compares the frame-level AUC of the video anomaly detection method proposed in this invention with several state-of-the-art methods. Based on whether memory is used, these methods are categorized into memory-based and memory-free methods. Memory-free methods typically perform frame reconstruction, frame prediction, or event recovery without using memory, minimizing reconstruction errors between the output video frames and the ground truth. Memory-based methods, on the other hand, typically use one or more memories to learn the video spatiotemporal prototype to limit the reconstruction capability of anomalous events. As shown in Table 1, the memory-based method generally outperforms the memory-free method, especially on the CHUKAvenue and ShanghaiTec datasets. This is because the CHUKAvenue and ShanghaiTec datasets are larger than UCSD ped2, containing more anomalous events, making the memory's ability to suppress anomalous event reconstruction more significant. Among all memory-based methods, the method proposed in this invention achieves the best results on all three datasets. Specifically, embodiments of the present invention achieved AUCs of 98.2%, 90.6%, and 74.4% on UCSD ped2, CHUK Avenue, and ShanghaiTech, respectively, representing improvements of 0.1%, 0.8%, and 0.6% over the best-performing memory-based methods. USTN-DSC achieves good performance without using memory because it employs a Transformer architecture with large parameters to learn more information. However, it is not suitable for low-configuration operating platforms.

[0141] The experimental results show that the present invention performs very well on CHUKAvenue, with an AUC exceeding 90%. This is because, compared to the other two datasets, people occupy a larger proportion of the entire image in Avenue and contain more motion details.

[0142] 2) Qualitative results:

[0143] like Figure 2 The figure shows the anomaly score curve and some samples of the video anomaly detection method proposed in this embodiment of the invention. Some test segments from three datasets are shown respectively. In the anomaly score curve, light green represents the true value, and the blue line represents the actual anomaly score. Figure 2 The program outputs representative normal and abnormal samples, displaying their predicted frames and error maps. PredictedFrame represents the predicted frame, Ground Truth represents the ground truth, and Error Map represents the error map. Normal indicates a normal image, and Abnormal indicates an abnormal image. Lighter colors in the error map indicate larger errors.

[0144] from Figure 2As can be seen, when the normal and abnormal states of an event change, the trend of the anomaly curve also changes abruptly, which means that the method proposed in this embodiment can respond quickly to anomalies. Simultaneously, the anomaly curve is stable within the normal and abnormal intervals, meaning that it can continuously and stably determine whether an event is abnormal. Several samples also show that the error map of normal frames is darker, while the error map of the abnormal region of abnormal frames is significantly brighter. These visual experimental results demonstrate that the method proposed in this embodiment has effective anomaly recognition capabilities.

[0145] The embodiments of the present invention also describe the performance improvement of the provided two-stage end memory for video anomaly detection.

[0146] Firstly, the two-stage end memory used for video anomaly detection in this embodiment of the invention processes the motion part separately through a first-stage network and a second-stage network, while the scene part and the object part are processed in the second-stage network. Compared with the fused prototype, the motion branch and the appearance branch do not affect each other.

[0147] As shown in Table 2 and Figure 3 As shown, this embodiment of the invention compares different training strategies based on two basic datasets, UCSD ped2 and CHUK Avenue, and uses frame-level AUC(%) to evaluate performance, thereby illustrating the advantages of the two-stage training framework of the two-stage end-memory in this embodiment of the invention.

[0148] One-stage with update 97.1 87.2 One-stage without update 97.4 87.7 Two-stage with update 97.6 88.8 Two-stage without update 98.2 90.6

[0149] Table 2

[0150] Figure 3In the figure, AUC represents the average score of normal and abnormal frames on UCSD ped2 and CHUKAvenue. The horizontal axis represents the data type, and the vertical axis represents the average score. Normal Frames represents normal frame data, Abnormal Frames represents abnormal frame data, and Gap refers to the difference between the average score of abnormal frames and the average score of normal frames. A larger Gap indicates a stronger ability to distinguish between normal and abnormal frames. Each normal frame, abnormal frame, and Gap data partition contains data from four methods, in order: One-stage with update, One-stage without update, Two-stage with update, and Two-stage without update. It can be seen that compared to One-stage with update and One-stage without update, Two-stage with update and Two-stage without update show AUC improvements of 1.6% and 2.9%, and 0.5% and 0.8% on CHUKAvenue and UCSD ped2, respectively. Compared to updating the spatiotemporal memory in the video prediction branch, one-stage without update and two-stage without update achieve AUC improvements of 0.5% and 1.8%, and 0.3% and 0.6% on CHUK Avenue and UCSD ped2, respectively. The average difference between normal and anomalous frames on UCSD ped2 and CHUKAvenue also shows that the proposed method achieves the largest gap, demonstrating the effectiveness of the two-stage training memory in this embodiment. This effectively increases the reconstruction gap between normal and anomalous frames, thereby significantly improving the video anomaly detection level.

[0151] Secondly, the two-stage end-memory system for video anomaly detection in this embodiment of the invention encodes the predicted future frames and the input video, and uses motion prototype enhancement to predict the initial frame. This fully utilizes the inherent property of video, namely bidirectional consistency, and significantly improves the accuracy of video frame prediction. As shown in Table 3 below, this embodiment of the invention conducts ablation studies on bidirectional consistency on the UCSD ped2 and CHUKAvenue datasets, and uses frame-level AUC(%) to evaluate performance, thereby demonstrating that this embodiment of the invention fully utilizes bidirectional consistency to improve the level of video anomaly detection.

[0152]

[0153] Table 3

[0154] In Table 3 above, the method without bidirectional consistency still fuses the motion information from the first stage in frame prediction to maximize its utilization. The results in the table show that using bidirectional consistency significantly improves performance compared to not using it. Specifically, the frame-level AUC of UCSD ped2 and CHUKAvenue is improved by 0.7% and 3.4%, respectively, providing strong evidence for the effectiveness of bidirectional consistency.

[0155] Thirdly, the two-stage end-memory system for video anomaly detection in this embodiment decomposes video frames into motion, scene, and object, which can reduce the blurring and distortion of object appearance in the prediction frame, thereby improving the ability of the two-stage end-memory system to distinguish between normal and abnormal frames when used for unsupervised anomaly detection. As shown in Table 4 below, this embodiment compares the performance of the decomposition method (decomposing video frames into motion, scene, and object) and the non-decomposition method (decomposing video frames into motion and appearance) on the UCSD ped2 and CHUK Avenue datasets, and uses frame-level AUC (%) to evaluate the performance. This illustrates the improvement in video anomaly detection level achieved by decomposing video frames into motion, scene, and object in this embodiment.

[0156] no 96.1 87.5 yes 98.2 90.6

[0157] Table 4

[0158] As shown in Table 4 above, compared with the method without decomposition, the decomposition method improved the AUC on UCSD ped2 and CHUK Avenue by 2.1% and 3.1% respectively, which proves the effectiveness of decomposition.

[0159] Fourthly, in this embodiment of the invention, the two-stage end memory used for video anomaly detection separately records the motion prototype of the moving parts. Compared with the fused prototype, the motion memory learns the complete feature representation of the prototype, which can improve the video anomaly detection level. As shown in Table 5 below, this embodiment of the invention illustrates an ablation study based on motion prototype enhancement, that is, separately recording the motion prototype of the moving parts, using motion prototype enhancement, and fusion recording the motion parts without using motion prototype enhancement. The results are compared on the UCSD ped2 and CHUK Avenue datasets, and the performance is evaluated using frame-level AUC (%).

[0160] no 96.1 87.5 yes 98.2 90.6

[0161] Table 5

[0162] According to Table 5 above, in the second-stage reverse process, compared with the model without motion prototype enhancement, the network with motion prototype enhancement improved the AUC by 0.8% and 2.2% on the UCSD ped2 and CHUK Avenue datasets, respectively. Motion prototype enhancement can further enhance the prediction ability of normal frames and limit the prediction ability of abnormal frames, thereby effectively increasing the reconstruction gap between normal and abnormal frames and obtaining better anomaly detection performance.

[0163] Fifthly, in this embodiment of the invention, a two-stage end-memory system for video anomaly detection incorporates a spatiotemporal memory in the second stage and learns appearance and motion to improve the video anomaly detection performance of the two-stage end-memory system, as shown in Table 6 below. Figure 4 As shown in the embodiments of the present invention, the impact of different locations of the spatiotemporal memory and different learning content on video anomaly detection performance is illustrated, and six different comparative experiments were conducted. The different locations of the spatiotemporal memory are as follows: Figure 4 As shown in the figure, the results of the comparative experiments are shown in Table 6.

[0164]

[0165]

[0166] Table 6

[0167] Figure 4 (a) in the reverse process learns the prototype through appearance and motion. Figure 4 (b) in the reverse process learns the prototype only by its appearance. Figure 4 (c) in the reverse process learns the prototype only through motion. Figure 4 In the positive process, (d) learns the prototype through scenes and objects. Figure 4 In the positive process, (e) learns the prototype only through the scenario. Figure 4In the forward pass, (f) is located in the forward pass, where prototypes are learned solely from objects. Table 6 shows that each of the two stages contains a memory and is used to learn different content. The memory in the first stage is used to learn complete motion prototypes, while the memory in the second stage is used to learn high-level spatiotemporal prototypes. When the memory is in the forward pass, the results in Table 6 show that learning both scene and object information simultaneously yields the highest performance. This is because both scene and object are important components of visual appearance, and relying solely on either scene or object is insufficient to capture the comprehensive video appearance information necessary for memory learning. When the memory is in the backward pass, using the memory to learn both appearance and motion simultaneously yields better performance results. Furthermore, updating the memory solely through motion is more effective than updating it solely through appearance. This is because updating the memory simultaneously with both appearance and motion allows it to learn more spatiotemporal information. Since the network prediction framework forces the network to focus more on appearance features, the impact of not updating the memory through appearance is less than the impact of not updating it through motion. Overall, the memory's impact on the results is greater in the second stage than in the first stage. Compared to the memory-based approach that learns the scene and objects in the first stage, the memory-based approach that learns appearance and motion in the second stage improves the AUC by 0.2% and 1.0% on UCSD ped2 and CHUKAvenue, respectively.

[0168] Furthermore, since the input of the reverse process utilizes the output of the forward process, the memory of the reverse process can fully learn this bidirectional spatiotemporal information and record higher-level spatiotemporal normal prototypes. Moreover, the forward and reverse processes are cascaded, allowing for error accumulation between them. Adding memory during the reverse process further increases the reconstruction error of anomalous videos, thereby widening the gap between normal and anomalous reconstructions and improving the ability to detect video anomalies.

[0169] Sixthly, in this embodiment of the invention, the capacity of the two-stage end-to-end memory used for video anomaly detection affects the accuracy of video anomaly detection. Figure 5As shown in the figure, the horizontal axis represents the memory capacity, and the vertical axis represents the AUC value. This embodiment of the invention seeks the optimal value between 0 and 50, with a step size of 10. It is clear from the figure that as the memory capacity increases, the model's performance initially increases and then decreases. When the memory capacity is small, a limited number of prototypes stored can reduce the reconstruction error of normal frames. However, when the memory capacity is too large, the stored prototypes become cluttered and may contain anomalous information. This can lead to performance degradation because the model has difficulty effectively distinguishing between normal and anomalous frames. Therefore, choosing an appropriate memory capacity is crucial. Furthermore, when the memory capacity increases from zero, a significant increase in frame-level AUC can be observed on both datasets, particularly on UCSD ped2. Therefore, increasing the memory module can effectively learn normal prototypes and improve model performance, while setting the capacity of the spatiotemporal memory and motion memory to 30 yields optimal two-order end-memory performance.

[0170] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0171] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0172] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A two-stage terminal memory for video anomaly detection, characterized in that, Including first-order networks and second-order networks; The first-order network is used to receive the motion part of the input video after decomposition; The first-order network includes a first-order encoder, a motion memory, and a first-order decoder connected in sequence, used to reconstruct motion of the motion part and obtain a motion prototype; the motion prototype is a permanent prototype that is updated once. The second-order network includes a cascaded forward process network and a backward process network; The forward process network is used to implement the forward process. The forward process network includes a second-order encoder and a second-order decoder. The second-order encoder is used to receive the scene part and object part after the input video is decomposed and to fuse the features of the scene part and object part. The second-order decoder is used to predict the predicted future frame of the input video. The reverse process network is cascaded with the forward process network to realize the reverse process. The reverse process network includes a second-order reverse encoder, a spatiotemporal memory and a second-order reverse decoder connected in sequence, which are used to reconstruct the prediction initial frame of the input video based on the predicted future frame, the input video and the motion prototype. The predicted future frame and the predicted initial frame are used for unsupervised anomaly detection, and the anomaly detection results of the input video are output.

2. The two-stage end-to-end memory for video anomaly detection as described in claim 1, characterized in that, The forward process network has a symmetrical structure. The first-order encoder includes a forward appearance encoder and a forward motion encoder, and the second-order decoder is a forward appearance decoder. The reverse process network has a symmetrical structure. The second-order reverse encoder includes a reverse appearance encoder and a reverse motion encoder, and the second-order reverse decoder is a reverse appearance decoder.

3. The two-stage end-to-end memory for video anomaly detection as described in claim 2, characterized in that, The forward appearance encoder extracts fine-grained features of the scene portion, and the forward motion encoder extracts fine-grained features of the object portion. After the fine-grained features of the scene part and the fine-grained features of the object part are fused, they are input into the forward appearance decoder to predict the predicted future frame of the input video.

4. The two-stage end-to-end memory for video anomaly detection as described in claim 2, characterized in that, The reverse appearance encoder obtains the reverse sequence of the intermediate video, which is video data based on the original initial frame of the input video with the predicted future frames deleted and merged. The reverse motion encoder calculates the RGB difference between the intermediate video and the reverse sequence; The reverse appearance decoder acquires the motion prototype. The reverse appearance decoder includes a network that predicts the initial frame. The RGB difference between the intermediate video and the reverse sequence, as well as the motion prototype, are used as inputs to the network that predicts the initial frame to obtain the predicted initial frame.

5. The two-stage end-to-end memory for video anomaly detection as described in claim 3, characterized in that, The spatiotemporal memory learns and records the video spatiotemporal prototype, and updates the video spatiotemporal prototype based on the RGB difference between the intermediate video and the reverse sequence; The reverse appearance decoder fuses the motion features of the motion prototype and the motion features learned from the video spatiotemporal prototype, and uses them as input to the network that predicts the initial frame.

6. The two-stage end-to-end memory for video anomaly detection as described in claim 5, characterized in that, The spatiotemporal memory is implemented as a matrix, and the updated spatiotemporal prototype of the video is calculated using similarity.

7. The two-stage end-to-end memory for video anomaly detection as described in claim 2, characterized in that, The reverse appearance encoder and the reverse motion encoder are connected in parallel and then connected to the spatiotemporal memory, which in turn is connected to the reverse appearance decoder.

8. The two-stage end-to-end memory for video anomaly detection as described in claim 1, characterized in that, The capacity of both the spatiotemporal memory and the motion memory is 30.

9. The two-stage end-to-end memory for video anomaly detection as described in claim 1, characterized in that, The second-order encoder and the second-order decoder are improved structures based on the Unet structure, using stride convolutions instead of pooling layers, and introducing skip connections between the second-order encoder and the second-order decoder; The second-order inverse encoder and the second-order inverse decoder have the same structure as the second-order encoder and the second-order decoder.

10. A video anomaly detection device, characterized in that, Includes a two-stage end memory and an unsupervised anomaly detection module for video anomaly detection as described in any one of claims 1 to 9; The unsupervised anomaly detection module is used to perform unsupervised anomaly detection based on the predicted future frame and the predicted initial frame, and output the anomaly detection results of the input video.

Images