Information-driven dual-flow fork network architecture video anomaly detection method and device

Abstract

The application provides a double-flow fork network architecture video anomaly detection method and device based on information driving, and the method comprises the following steps: constructing a double-flow fork network architecture model, inputting a video frame to be detected into the double-flow fork network architecture model, wherein the double-flow fork network architecture model comprises an appearance fork network and a dynamic fork network; the motion fork network comprises a motion autoencoder and a motion auxiliary encoder; the appearance fork network comprises an appearance autoencoder, an information re-correction network and an appearance auxiliary encoder; a total loss function of the double-flow fork network architecture model is calculated; the video to be tested is input into the double-flow fork network architecture model, the total loss function of the appearance fork network and the motion fork network is calculated, and a video anomaly score is obtained. The double-flow fork network architecture is used to separate motion features and appearance features, the information re-correction network is used in the appearance fork network to reduce information loss, the auxiliary encoder is added to obtain deeper semantic information, and the video anomaly detection performance is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision, and particularly relates to a double-flow fork network architecture video anomaly detection method and device based on information driving. BACKGROUND

[0002] The video anomaly detection task is a hot research direction of computer vision, and has wide application prospects in video monitoring, activity recognition, scene understanding and the like. Under the premise of unsupervised, only normal training data is given in most public data sets, so the video anomaly detection algorithm attempts to capture the feature representation of normal events, and uses it to identify abnormal events in the test set. However, the video anomaly detection task still faces many difficulties and challenges, among which are: the dependence of the definition of abnormal events on specific scenes, the rarity, diversity and endlessness of abnormal events, the existence of noise and information loss in the training samples that interfere with training. There is an information loss problem in most of the current research methods.

[0003] In recent years, due to the strong and convenient characteristics of the auto-encoder network in extracting data feature representation, a large number of auto-encoder-based methods are widely used in image reconstruction, clustering, machine translation and the like. These methods mostly tend to capture hidden features in video sequences by constructing reconstruction work, but they often rely on some well-designed network architecture.

[0004] The rate distortion theory clearly points out from the perspective of information theory that data compression must be accompanied by certain information loss, and the encoding process of the auto-encoder is actually a data compression process, so the auto-encoder will inevitably lose certain features. In view of this, the Unet architecture solves the problem of information loss by introducing a skip connection part to the auto-encoder, so that the decoder can process the feature mapping of the upper layer and the encoder at the same time. In addition, some methods use the Unet architecture in the double-flow network to perform the video anomaly detection task. These methods usually use Unet to predict a specific frame of the input continuous frames, and use optical flow as an auxiliary task to obtain related motion features, but since Unet usually takes continuous video frames as input and pays more attention to motion information, it is not sensitive to some abnormal objects that slightly differ in appearance from normal objects. SUMMARY

[0005] In view of the above deficiencies of the prior art, the application provides a double-flow fork network architecture video anomaly detection method and device based on information driving, which comprises the following contents:

[0006] A two-stream forked network architecture model is constructed, comprising an appearance forked network and a dynamic forked network. The appearance forked network processes appearance information and uses a reconstruction task with information recorrection to detect objects inconsistent with the scene. The dynamic forked network processes motion information, has a spatial motion mechanism, and identifies abnormal motion patterns. The motion forked network includes a motion autoencoder and a motion-assisted encoder. ms The appearance fork network includes an appearance automatic encoder, an information recalibration network, and an appearance auxiliary encoder E. as ;

[0007] Calculate the total loss function for the two-stream cross-network architecture model;

[0008] The video to be tested is input into the dual-stream cross-network architecture model to obtain the video anomaly score.

[0009] Preferably, each encoder and decoder in the appearance forknet and motion forknet consists of three ResNet blocks, and each auxiliary encoder consists of five ResNet blocks; the ReLU layer in each ResNet block is replaced with a LeakyReLU layer; and the downsampling layer is replaced with a pixel shuffling layer in the ResNet block of the decoder.

[0010] Preferably, the motion fork network uses a U-net architecture and a 2D CNN as the backbone; the motion autoencoder includes a motion encoder E. m and motion decoder D m ;

[0011] x clips It consists of consecutive video frames, where the first video frame is x. fif The last video frame is x lif ; will x clips Dividing x lif In addition to the video frame stacking input motion encoder E m The motion representation z is then obtained. m , will x lif and x fif RGB difference x between diff As the target; motion encoder E m and motion decoder D m There are jump connections between them, representing the motion z. m The jump connection provides all levels of feature input to the motion decoder D. m Predicting RGB difference calculate With x diff The mean square loss and gradient loss are used to obtain the dynamic flow loss function of the motion autoencoder, as shown below:

[0012]

[0013] Among them, l m G represents the dynamic flow loss function. d This represents the image gradient of a video frame along the spatial x-axis and y-axis.

[0014] Preferably, the appearance automatic encoder includes an appearance encoder E. a and appearance decoder D a ; to transfer video frames x fif Input to appearance encoder E a Convert to appearance representation z a Using appearance decoder D a Representing appearance as z a Generate reconstruction results By minimizing x fif and The reconstruction error between the two values ​​is used to train the appearance autoencoder. The reconstruction error is represented by the loss function of the appearance autoencoder, and the calculation formula is as follows:

[0015]

[0016] Among them l a This is the loss function for the appearance auto encoder.

[0017] Preferably, the information recalibration network extracts the appearance encoder E. a Mapping features and appearance decoder D a The mapping features in the image are used to reduce the difference between the two and suppress information loss; the appearance encoder E a The mapping feature is the appearance encoder E a Output of the first and second ResNet blocks and Appearance Decoder D a The mapping features in the appearance decoder D a The output of the first and second ResNet blocks and The difference between feature maps is represented by a correction loss function, calculated as follows:

[0018]

[0019] Where λ0 and λ1 represent adjustable hyperparameters, l IR To correct the loss function.

[0020] Preferably, the motion-assisted encoder E ms Used to highlight motion target information that highlights RGB differences; representing motion as z m Input motion-assisted encoder E ms Obtain the target score sm ;

[0021] Appearance Auxiliary Encoder E as The reconstruction results are refined by making the appearance features continuously sparse; the appearance encoder E a The output z a Input to appearance auxiliary encoder E as In the process, the appearance target score s is obtained. a ;

[0022] Motion Assist Encoder E ms The objective function is defined as the motion feature distribution p ms (s m ||z m And the corresponding appearance feature distribution p in the appearance cross network. as (s a ||z a The KL divergence between them:

[0023] l ms =KL[p ms (s m ||z m )||p as (s a ||z a )]

[0024] Among them l ms Indicates motion-assisted encoder E ms The loss function is KL(·), which represents the KL divergence.

[0025] Appearance Auxiliary Encoder E as The objective function is expressed as the KL divergence between the appearance feature distribution and the motion feature distribution of the motion fork network:

[0026] l as =KL[p as (s a ||z a )||p ms (s m ||z m )]

[0027] Among them, l as Indicates appearance auxiliary encoder E as The loss function is KL(·), which represents the KL divergence.

[0028] Preferably, the total loss function is expressed as:

[0029] l TOTAL =l a +l m +l IR +λs (l as +l ms )

[0030] Among them, l TOTAL Let λ represent the total loss function. s It is an adjustable hyperparameter.

[0031] Preferably, the video anomaly score calculation process is as follows:

[0032] The video to be tested is input into the model, and the last video frame x is calculated. lif Prediction results in calculate With x lif The reciprocal of the mean square error between them, s lif The calculation formula is as follows:

[0033]

[0034] s lif The larger the value, the more normal the video frame. After calculating the score of each video sequence at all spatial locations, the loss is normalized to obtain the score of each video frame in the range [0,1]. The score calculation formula is as follows:

[0035]

[0036] Where S represents the video to be tested, Score(·) represents the video anomaly score, max(S) represents the maximum loss value in the video to be tested S, and min(S) represents the minimum loss value in the video to be tested S.

[0037] The present invention also proposes an electronic device comprising a memory, a processor, and program instructions stored in the memory that are executable by the processor, the processor executing the program instructions to implement the steps of the method described in the present invention.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] 1. This invention uses a dual-stream fork network architecture to separate motion features and appearance features, and utilizes a reconstruction task with information recalibration in the appearance fork network to detect objects inconsistent with the scene. The motion branch with skipped connections serves as a supplement to the appearance branch, and can effectively identify appearance anomalies while identifying motion anomalies.

[0040] 2. This invention uses an information recalibration network to enable the appearance autoencoder to suppress information loss from the source layer by layer, thereby providing a more refined representation of the test samples and producing a more significant distinction between normal and abnormal samples.

[0041] 3、The appearance auxiliary encoder and the motion auxiliary encoder are added in the appearance self-encoder and the motion self-encoder respectively in the application, so that deeper semantic information is obtained, and information loss is reduced under the premise of ensuring real-time. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description, obviously, the drawings in the following description are only some embodiments of the present application, and for the ordinary skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0043] Figure 1 The overall framework diagram of the multi-perception video abnormal event detection method based on isomeric information provided by the embodiments of the present application is shown in the figure.

[0044] Figure 2 The abnormal detection curves on the UCSD Ped2, CUHKAvenue and ShanghaiTech three data sets provided by the embodiments of the present application are shown in the figure.

[0045] Figure 3 The influence diagram of different information re-corrections on the model performance provided by the embodiments of the present application is shown in the figure.

[0046] Figure 4 The influence diagram of different channel numbers on the model performance provided by the embodiments of the present application is shown in the figure.

[0047] Figure 5 The comprehensive performance comparison diagram provided by the embodiments of the present application is shown in the figure.

[0048] The accompanying drawings are used to provide a further understanding of the present application, and constitute a part of the specification, and are used to explain the present application together with the embodiments of the present application, and do not constitute a limitation on the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application, obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by the ordinary skilled in the art without creative labor are within the protection scope of the present application.

[0050] As shown in the figure, the embodiments of the present application include the following steps: Figure 1

[0051] ​A dual-flow forked network architecture model is constructed, wherein the dual-flow forked network architecture model comprises an appearance forked network and a dynamic forked network; the appearance forked network is used to process appearance information, and uses a reconstruction task with information re-correction to detect objects inconsistent with a scene; the dynamic forked network is used to process motion information, has a spatial motion mechanism, and identifies abnormal motion patterns; the motion forked network comprises a motion auto-encoder and a motion auxiliary encoder E ms ; the appearance forked network comprises an appearance auto-encoder, an information re-correction network and an appearance auxiliary encoder E as ;

[0052] A total loss function of the dual-flow forked network architecture model is calculated.

[0053] A video to be tested is input into the dual-flow forked network architecture model to obtain a video anomaly score.

[0054] Each encoder and decoder in the appearance forked network and the motion forked network is composed of three Resnet blocks, and each auxiliary encoder is composed of five Resnet blocks; a ReLU layer in each Resnet block is replaced by a LeakyReLU layer; a pixel shuffle layer is used to replace a down-sampling layer in a Resnet block in the decoder. Figure 1 A flowchart of output of each part of the appearance (motion) auto-encoder is shown in FIG. 1, an input size is 3×256×256 (12×256×256), a latent representation size is 256×32×32, and an output size is 3×256×256; Figure 1 A flowchart of intermediate output of each part of the auxiliary encoder when the number of channels is 256, 512 and 1024 respectively is also shown in FIG. 2.

[0055] A U-net architecture and a 2D CNN are used as a backbone of the motion forked network; the motion auto-encoder comprises a motion encoder E m and a motion decoder D m ;

[0056] x clips is a continuous video frame, wherein a first video frame is x fif , and a last video frame is x lif ; video frames in x clips except x lif are stacked to input the motion encoder E m to obtain a motion representation z m ; an RGB difference x lif between x fif and x diff is taken as a target; the motion encoder E m and the motion decoder D m have a skip connection, and the motion representation z m and features of all levels provided by the skip connection are input into the motion decoder Dm Predicting RGB difference Calculate The mean square loss and gradient loss between x diff , get the dynamic flow loss function of motion autoencoder, as follows:

[0057]

[0058] Wherein, l m represents the dynamic flow loss function, g d represents the image gradient of video frame along the spatial x axis and y axis.

[0059] The appearance autoencoder comprises an appearance encoder E a and an appearance decoder D a ; the video frame x fif is input into the appearance encoder E a to be converted into an appearance representation z a , the appearance representation z a is generated by using the appearance decoder D a to generate a reconstruction result The appearance autoencoder is trained by minimizing the reconstruction error between x fif and , the reconstruction error is represented by the loss function of the appearance autoencoder, and the calculation formula is as follows:

[0060]

[0061] Wherein, l a is the loss function of the appearance autoencoder.

[0062] The information rectification network extracts the mapping features of the appearance encoder E a and the mapping features in the appearance decoder D a , and reduces the difference between them to inhibit information loss; the mapping features of the appearance encoder E a are the outputs of the first and second Resnet blocks of the appearance encoder E a and The mapping features in the appearance decoder D a are the outputs of the first and second Resnet blocks of the appearance decoder D a and The difference between the feature mappings is represented by a correction loss function, and the calculation formula is as follows:

[0063]

[0064] Wherein, λ0 and λ1 represent adjustable hyperparameters, and l IR ​​To correct the loss function.

[0065] Motion auxiliary encoder E ms Motion target information for highlighting RGB differences; the motion representation z m Input motion auxiliary encoder E ms Get motion target score s m ;

[0066] Appearance auxiliary encoder E as Refine the reconstruction result by making the appearance features persistently sparse; the output z a of the appearance encoder E a is input into the appearance auxiliary encoder E as , to get the appearance target score s a ;

[0067] Motion auxiliary encoder E ms The objective function of the motion auxiliary encoder E ms (s m ||z m ) and the corresponding appearance feature distribution p as (s a ||z a ) in the appearance cross-network between them:

[0068] l ms = KL[p ms (s m ||z m )||p as (s a ||z a )]

[0069] Where l ms represents the loss function of the motion auxiliary encoder E ms , and KL(·) represents the KL divergence;

[0070] The objective function of the appearance auxiliary encoder E as is represented as the KL divergence between the appearance feature distribution and the motion feature distribution of the motion cross-network:

[0071] l as = KL[p as (s a ||z a )||p ms (s m ||z m )]

[0072] Where, l as represents the loss function of the appearance auxiliary encoder E as , and KL(·) represents the KL divergence.

[0073] The total loss function is expressed as:

[0074] l TOTAL =l a +l m +l IR +λ s (l as +l ms )

[0075] Among them, l TOTAL Let λ represent the total loss function. s It is an adjustable hyperparameter.

[0076] The video anomaly score calculation process is as follows:

[0077] The video to be tested is input into the model, and the last video frame x is calculated. lif Prediction results in calculate With x lif The reciprocal of the mean square error between them, s lif The calculation formula is as follows:

[0078]

[0079] s lif The larger the value, the more normal the video frame. After calculating the score of each video sequence at all spatial locations, the loss is normalized to obtain the score of each video frame in the range [0,1]. The score calculation formula is as follows:

[0080]

[0081] Where S represents the video to be tested, Score(·) represents the video anomaly score, max(S) represents the maximum loss value in the video to be tested S, and min(S) represents the minimum loss value in the video to be tested S.

[0082] The effectiveness of the method of the present invention will be verified through simulation experiments below.

[0083] (1) Comparison with other detection algorithms

[0084] The method of the present application is compared with the currently existing better algorithms on the three data sets of UCSD Ped2, CUHK Avenue and Shanghai Tech, as shown in Table 1; from Table 1, it is known that in the AUC value results, the present application method can obtain better performance on the three data sets. Frame Prediction [7] performs slightly better on Shanghai Tech, but the present application method performs better on the other two data sets, especially 3.5% higher on CUHK Avenue; the performance of HF2-VAD

[11] on UCSD Ped2 is better than that of the present application method, which is mainly because the scene of UCSD Ped2 is relatively simple, and the proportion of appearance information (pedestrians, etc.) is much higher than that of motion information (bicycles, etc.), and the speed of detecting abnormal events in HF2-VAD

[11] is only 10FPS, which is lower than the speed of 34FPS of the present application method, in addition, the performance of the present application method on CUHK Avenue is better than that of HF2-VAD

[11] , which verifies that information recalibration is superior in improving the performance of auto-encoder network; the present application method only uses RGB difference as a motion clue, which reduces the calculation cost of optical flow estimation.

[0085] In terms of RBDC and TBDC results, only Liu [4] and Ramachandra [8] reported some results, compared with these two methods, the RBDC and TBDC results of the present application method on the three data sets are the best, which verifies the effectiveness of the motion fork network in the present application method using RGB difference as a motion clue to assist the encoder.

[0086] Table 1 Comparison of AUC / RBDC / TBDC results of different methods

[0087]

[0088] The comparison literature appearing in Table 1 is listed as follows:

[0089] [1] H. Park, J. Noh and B. Ham, “Learning Memory-guided Normality for Anomaly Detection,” in Proc. IEEE / CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Seattle, USA, Jun. 2020, pp. 14360-14369.

[0090] [2] W. Luo, W. Liu and S. Gao, “Remembering history with convolutional LSTM for anomaly detection,” in Proc. IEEE Int. Conf. Multimedia Expo (ICME), Hong Kong, Jul. 2017, pp. 439-444.

[0091] [3] D. Abati, A. Porrello, S. Calderara and R. Cucchiara, “Latent space autoregression for novelty detection,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Long Beach, CA, USA, Jun. 2019, pp. 481-490.

[0092] [4] W. Liu, W. Luo, D. Lian and S. Gao, “Future frame prediction for anomaly detection - A new baseline,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Salt Lake City, UT, USA, Jun. 2018, pp. 6536-6545.

[0093] [5] D. Gong, L. Liu, V. Le, B. Saha, M. R. Mansour, S. Venkatesh and A. Hengel, “Memorizing Normality to Detect Anomaly: Memory-Augmented Deep Autoencoder for Unsupervised Anomaly Detection,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Seoul, Korea (South), Oct. 2019, pp. 1705-1714.

[0094] [6] G. Yu, S. Wang, Z. Cai, E. Zhu, C. Xu, J. Yin and M. Kloft,“Cloze test helps: Effective video anomaly detection via learning to complete video events,” in Proc. 28th ACM Int. Conf. Multimedia (MM), Seattle, USA, Oct. 2020, pp. 583-591.

[0095] [7] W. Xuanzhao, C. Zhengping, Y. Ke, J. Bo, T. Jian, Y. Jieping, W. Jingyu and Q. Qi,“Robust Unsupervised Video Anomaly Detection by Multi-Path Frame Prediction,” IEEE Trans. on Neural Networks and Learn. Syst., May. 2021, doi: 10.1109 / TNNLS.2021.3083152.

[0096] [8] J. Yu, Y. Lee, K. C. Yow, M. Jeon and W. Pedrycz,“Abnormal Event Detection and Localization via Adversarial Event Prediction,” IEEE Trans. on Neural Networks and Learn. Syst. (TNNLS), vol. 99, pp. 1-15, Feb. 2021.

[0097] [9] Y. P. Chang, Z. G. Tu, W. Xie, B. Luo, S. F. Zhang and H. G. Sui,“Video anomaly detection with spatio-temporal dissociation,” Pattern Recognit., vol. 122, pp. 1-12, Feb. 2022.

[0098]

[10] M. Ye, X. Peng, W. Gan, W. Wu and Y. Qiao,“An opcn: Video anomaly detection via deep predictive coding network,” in Proc. 27th ACM Int. Conf. Multimedia (MM), Nice, France, Oct. 2019, pp. 1805-1813.

[0099]

[11] Z. Liu, Y. Nie, C. Long, Q. Zhang and G. Li,“A Hybrid Video Anomaly Detection Framework via Memory-Augmented Flow Reconstruction and Flow-Guided Frame Prediction,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Montreal, Canada, Oct. 2021, pp. 13568-13577.

[0100] (2) Conducting ablation experiments

[0101] To evaluate how different components affect the anomaly detection performance, ablation experiments were conducted on the three available datasets, and the basic model (BM) involved in Table 2 represents the model without information rectification and auxiliary encoder. First, the effectiveness of each single module was tested multiple times. As can be seen from Table 2, after using information rectification (IR), the AUC / RBDC / TBDC results have improved significantly compared to the basic model, which shows that the information loss in the encoding and decoding stages has an impact on the anomaly detection performance, and information rectification can capture the lost information as additional constraints to improve the anomaly detection performance. After adding the auxiliary encoder (SE), the AUC / RBDC / TBDC performance of the model increased by 0.7% / 5.6% / 5.4%, 1.8% / 7.8% / 6.3%, and 1.5% / 3.8% / 3.2% on the three datasets, respectively, because the auxiliary encoder (SE) makes the entire module pay more attention to the target features, which are usually directly related to abnormal events. In addition, we also conducted comparative experiments combining different modules, and on the CUHK Avenue dataset, information rectification (IR) and auxiliary encoder (SE) can increase the AUC value of the model by 4.7% and 1.8%, respectively. The combination of information rectification and secondary encoder can improve the performance of the model by 6.5%, as expected (4.7+1.8=6.5).

[0102] Table 2 Impact of different modules on model performance

[0103]

[0104] (3) Abnormal spatiotemporal location

[0105] To qualitatively analyze the anomaly detection performance of the method of this invention, anomaly detection instances on three available datasets under the method of this invention are visualized, such as... Figure 2 As shown, to more intuitively reflect the performance of our method, anomaly score curves were plotted for each frame over time. From Figure 2 As can be seen from the two curves on the left, the method of this invention performs excellently in detecting anomalous events on the UCSD Ped2. When an anomalous event (bicycle) occurs, the anomalous score increases rapidly and remains stable for a short period. Similarly, as the anomalous event disappears, the anomalous score decreases rapidly and remains stable for a very short time. Some fluctuations in the anomalous score curve are attributed to noise inherent in the UCSD Ped2 itself. Figure 2 The curves in the middle and right parts of the graphs are examples of complex datasets CUHKAvenue and Shanghai Tech. As anomalies appear (disappear), outliers tend to decrease (increase) rapidly and remain at a stable level for a short period of time. The curves also have some fluctuations, partly due to noise in the datasets, but mainly because both CUHKAvenue and Shanghai Tech contain a large number of scenes and objects, especially Shanghai Tech, which has more complex relationships and anomaly types. However, this also demonstrates the good performance of the method of this invention and its good robustness to noise and complex conditions.

[0106] (4) Further verification of information recalibration and auxiliary encoder.

[0107] Figure 3 This demonstrates the impact of information recalibration at each level on model performance, where Shallow IR represents... and Information recalibration between Deep IR representations and Information recalibration between [various sources]. Figure 3It can be observed that both Shallow IR and Deep IR play a significant role in suppressing information loss, and also indicate that encoding and decoding operations have a certain impact on the loss of both shallow visual features and deep semantic features. When using Shallow IR or Deep IR alone, regardless of the AUC / RBDC / TBDC values ​​of the dataset, Shallow IR consistently outperforms Deep IR, suggesting that the loss of visual information in Shallow IR accounts for the vast majority of the information loss. Furthermore, the model's performance is further improved when Shallow IR and Deep IR are used simultaneously.

[0108] Figure 4 The impact of each channel selection on model performance is illustrated, where Channels=0 represents a model without an auxiliary encoder. It can be seen that when Channels=1024, the model's AUC / RBDC / TBDC performance on the three datasets is at its lowest. This is because using too many channels makes the target features the model focuses on cluttered, with too many irrelevant scene features mixed in with the target features, affecting the model's attention to target information. When tested on the UCSD Ped2 dataset, the AUC / RBDC / TBDC performance peaks at Channels=256, due to the excessive background noise in the UCSD Ped2 dataset. Therefore, too many channels introduce unnecessary information. For other datasets, the AUC / RBDC / TBDC performance peaks at Channels=512. Choosing an appropriate number of channels based on the actual situation allows the model to focus on an appropriate number of target features without introducing too much irrelevant information.

[0109] (5) Running time

[0110] We used an NVIDIA GeForce 1080 Ti to compare the accuracy and efficiency of our method with other methods on the CUHK Avenue dataset. Figure 5 As shown, the method of the present invention is 1.3 times faster than Liu[4], 3.4 times faster than HF2-VAD

[11] , and 3.7 times faster than Frame Prediction[7]. The method of the present invention requires an average of 0.0294 seconds, or 34 FPS, for anomaly detection of one frame of video. Dual-Clustering[9] and MeMAE[5] can achieve 32 FPS and 38 FPS, respectively, but their AUC accuracy is much lower than that of the method of the present invention.

[0111] Finally, the present application proposes an electronic device comprising a memory, a processor and program instructions stored in the memory and executable by the processor to implement each step of the method for video anomaly detection based on scene classification proposed by the present application. It should be noted that the embodiments of the present application are described as devices, which are similar to the description of the method embodiments described above, and have similar beneficial effects as the method embodiments, and therefore will not be described again.

[0112] Program code for carrying out the methods of the present application can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, causes the machine to perform the described functions / operations specified in the flowcharts and / or block diagrams. The program code can execute entirely on a machine, partly on a machine, as a stand-alone software package, partly on a machine and partly on a remote machine or entirely on a remote machine or server.

[0113] Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some of the technical features, and any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An information-driven dual-flow forked-net architecture video anomaly detection method, characterized in that, The method comprises the following steps: A dual-flow forked-net architecture model is constructed, wherein the dual-flow forked-net architecture model comprises an appearance forked net and a dynamic forked net, the appearance forked net is used to process appearance information, uses a reconstruction task with information re-correction to detect objects inconsistent with a scene, the dynamic forked net is used to process motion information, has a spatial motion mechanism, and identifies abnormal motion patterns; the dynamic forked net comprises a motion auto-encoder and a motion auxiliary encoder , the appearance forked net comprises an appearance auto-encoder, an information re-correction network and an appearance auxiliary encoder ​ calculating a total loss function of a double-flow fork network architecture model; inputting a video to be tested into the double-flow fork network architecture model to obtain a video anomaly score; each encoder and decoder in the appearance fork network and the dynamic fork network is composed of three Resnet blocks, each auxiliary encoder is composed of five Resnet blocks, a ReLU layer in each Resnet block is replaced by a LeakyReLU layer, and a pixel shuffle layer is used to replace a down-sampling layer in the Resnet block in the decoder; The dynamic forknet uses a U-net architecture and a 2D CNN as the backbone; the motion autoencoder comprises a motion encoder and a motion decoder ​ It consists of consecutive video frames, where the first video frame is The last video frame is ;Will Except In addition to the video frame stacking input motion encoder The motion representation was then obtained. ,Will and RGB difference between As the target; motion encoder and motion decoder There are jump connections between them, representing motion. The jump connection provides all levels of feature input motion decoder Predicting RGB difference ,calculate and The mean square loss and gradient loss are used to obtain the dynamic flow loss function of the motion autoencoder, as shown below: ； wherein, represents a dynamic flow loss function, represents the image gradient of the video frame along the spatial x and y axes.

2. The information-driven double-flow fork network architecture video anomaly detection method according to claim 1, wherein: The appearance autoencoder comprises an appearance encoder and an appearance decoder ; input a video frame to the appearance encoder to convert into an appearance representation , use the appearance decoder to generate a reconstruction result from the appearance representation ; train the appearance autoencoder by minimizing a reconstruction error between and , the reconstruction error being represented by a loss function of the appearance autoencoder, the formula being as follows: ； wherein is the autoencoder reconstruction loss function.

3. The information-driven dual-flow forked-net architecture video anomaly detection method according to claim 2, characterized in that, The information recalibration network extracts the appearance encoder. Mapping features and appearance decoder The mapping features in the image are used to reduce the difference between the two and suppress information loss; appearance encoder The mapping features are for appearance encoders Output of the first and second ResNet blocks and Appearance decoder The mapping features in the appearance decoder The output of the first and second ResNet blocks and The difference between feature maps is represented by a correction loss function, calculated as follows: ； wherein and denote tunable hyperparameters, is a correction loss function.

4. The information-driven double-flow fork network architecture video anomaly detection method according to claim 3, wherein: The motion-assisted encoder Motion object information for highlighting RGB differences; motion representation Input motion-assisted encoder Obtaining motion object scores ; The appearance auxiliary encoder Refines the reconstruction result by making the appearance features persistently sparse; outputs of the appearance encoder Are input into the appearance auxiliary encoder To obtain appearance target scores ;​ Motion-augmented encoder The objective function is defined as the KL divergence between the motion feature distribution and the appearance feature distribution in the corresponding appearance fork with a KL divergence between: ； wherein representing a motion-augmented encoder a loss function of, representing a KL divergence; Appearance assisted encoder The objective function is expressed as the KL divergence between the appearance feature distribution and the motion feature distribution of the dynamic crossnet: ； wherein, represents an appearance auxiliary encoder a loss function of the appearance auxiliary encoder, represents a KL divergence.

5. The information-driven dual-flow forked-net architecture video anomaly detection method according to claim 4, characterized in that, the total loss function is expressed as: ； wherein, represents the total loss function, is a tunable hyperparameter.

6. The information-driven dual-flow forked-net architecture video anomaly detection method according to claim 5, characterized in that, the video anomaly score is calculated according to the following process: The video to be tested is input into the model, and the prediction result of the last video frame is calculated wherein the reciprocal of the mean square error between and is calculated , and the calculation formula is as follows:​ ； The larger the value of the number is, the more normal the video frame is. after calculating the score of each video sequence at all spatial positions, the loss is normalized to obtain a score of each video frame in the range [0, 1], and the score calculation formula is as follows: ； wherein denotes the video to be tested, denotes the video anomaly score, denotes the video to be tested the largest loss value in the video, denotes the video to be tested the smallest loss value in the video.

7. An electronic device comprising a memory, a processor, and program instructions stored in the memory for execution by the processor, the device being characterized by: the processor executes the program instructions to implement the steps of the method according to any one of claims 1-6.

Images