Video anomaly detection method and device, equipment, storage medium and product

Abstract

The invention discloses a video anomaly detection method and device, equipment, a storage medium and a product. The method comprises the following steps: acquiring a to-be-verified video frame sequence and a target verification video frame, and determining a to-be-verified frame difference sequence according to the to-be-verified video frame sequence; respectively inputting the to-be-verified video frame sequence and the to-be-verified frame difference sequence into a pre-trained video anomaly detection model to obtain a target prediction video frame output by the model; and performing video anomaly detection on the target verification video frame according to the target verification video frame and the target prediction video frame to obtain an anomaly detection result. According to the technical scheme of the embodiment of the invention, the video anomaly detection precision and detection efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video anomaly detection method, device, equipment, storage medium and product. Background Art

[0002] Video anomaly detection aims to identify abnormal situations that may deviate from normal behavior in the video, such as traffic accidents, natural disasters, and abnormal behavior. In practical applications, video anomaly detection usually requires detection technology to have both high accuracy and high processing speed. Therefore, an efficient video analysis model is needed as the basis for anomaly detection.

[0003] The existing video anomaly detection technology mainly detects from two aspects: one is classification-based anomaly detection technology, and the other is prediction-based video anomaly detection technology. However, the existing video anomaly detection technology ignores the characteristics of the video, and the model training process requires high computing power and a huge number of parameters, resulting in low anomaly detection accuracy and efficiency. Summary of the invention

[0004] The present invention provides a video anomaly detection method, device, equipment, storage medium and product to improve the accuracy and efficiency of video anomaly detection.

[0005] According to one aspect of the present invention, a video anomaly detection method is provided, the method comprising:

[0006] Acquire a sequence of video frames to be verified and a target verification video frame, and determine a sequence of frame difference values ​​to be verified according to the sequence of video frames to be verified;

[0007] Inputting the to-be-verified video frame sequence and the to-be-verified frame difference sequence into a pre-trained video anomaly detection model respectively to obtain a target predicted video frame output by the model;

[0008] According to the target verification video frame and the target prediction video frame, video anomaly detection is performed on the target verification video frame to obtain an anomaly detection result.

[0009] According to another aspect of the present invention, a video anomaly detection device is provided, the device comprising:

[0010] A target frame determination module is used to obtain a sequence of video frames to be verified and a target verification video frame, and determine a sequence of frame difference values ​​to be verified based on the sequence of video frames to be verified;

[0011] A frame prediction module, used to input the video frame sequence to be verified and the frame difference sequence to be verified into a pre-trained video anomaly detection model, respectively, to obtain a target predicted video frame output by the model;

[0012] The video anomaly detection module is used to perform video anomaly detection on the target verification video frame according to the target verification video frame and the target prediction video frame to obtain an anomaly detection result.

[0013] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the video anomaly detection method described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video anomaly detection method described in any embodiment of the present invention when executed.

[0018] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in the above-mentioned video anomaly detection method are implemented.

[0019] The technical solution of the embodiment of the present invention fuses the spatiotemporal features of the past frame sequence, that is, the video appearance features and the video motion features, through a two-time fusion scheme, and fuses the predicted frame with the predicted frame difference, thereby realizing the prediction of the second frame in the future to obtain the video anomaly detection result. By comparing the difference between the predicted frame and the actual frame, it is judged whether an abnormal event has occurred in the video. Only basic convolution kernels and automatic codecs are used in feature extraction and prediction, and the processing flow of the memory module is simplified. The computing power requirement is extremely low, and the high processing speed requirement can be met, thereby improving the efficiency of video anomaly detection. In addition, the method retains all the complex features of the original video to the greatest extent, and uses the memory module to obtain the relationship between the static (appearance) features and the dynamic (motion) features of the video, and then through the prediction of the second frame in the future, the accuracy of video anomaly detection is improved.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1A is a flow chart of a video anomaly detection method provided according to Embodiment 1 of the present invention;

[0023] Figure 1B is a schematic diagram of the architecture of a video anomaly detection model provided according to the first embodiment of the present invention;

[0024] Figure 2A This is an overall architecture diagram of a video anomaly detection method provided according to Embodiment 2 of the present invention;

[0025] Figure 2B This is an overall architecture diagram of an appearance encoding and decoding module provided according to Embodiment 2 of the present invention;

[0026] Figure 2C is an overall architecture diagram of a motion coding module provided according to Embodiment 2 of the present invention;

[0027] Figure 2D is an overall architecture diagram of a memory module provided according to Embodiment 2 of the present invention;

[0028] Figure 3 is a structural schematic diagram of a video anomaly detection device provided according to Embodiment 3 of the present invention;

[0029] Figure 4 It is a structural schematic diagram of an electronic device for implementing the video anomaly detection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment 1

[0033] Figure 1A This is a flowchart of a video anomaly detection method provided in the first embodiment of the present invention. This embodiment can be applied to the case where anomalies are detected in a video corresponding to a specific scene, such as a traffic accident or a natural disaster. The method can be performed by a video anomaly detection device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. Figure 1A As shown, the method includes:

[0034] S110, obtaining a sequence of video frames to be verified and a target verification video frame, and determining a sequence of frame difference values ​​to be verified according to the sequence of video frames to be verified.

[0035] S120, respectively inputting the video frame sequence to be verified and the frame difference sequence to be verified into a pre-trained video anomaly detection model to obtain a target predicted video frame output by the model.

[0036] S130. Perform video anomaly detection on the target verification video frame according to the target verification video frame and the target prediction video frame to obtain anomaly detection results.

[0037] The target verification video frame may be a video frame that needs to be detected for anomalies, and the sequence of video frames to be verified may be a video frame that has a sequence order relationship with the target verification video frame. For example, if the target verification video frame is frame t+2, then the sequence of video frames to be verified may be frames 1 to t. By performing anomaly prediction on the sequence of video frames to be verified, it is verified whether the target verification video frame is abnormal, and then it is verified whether the video in the video segment to which the target verification video frame belongs is abnormal.

[0038] The frame difference sequence to be verified may be a difference obtained by subtracting consecutive video frames. For example, if the video frame sequence to be verified is F1, F2, ..., F t+1, then subtract the pixels at the same pixel coordinate position of the two adjacent frames to obtain t consecutive frame differences X1, X2, ..., X t For example, t+1 Frame and F t Subtract the frames and you can get the Xth t Frame difference.

[0039] It should be noted that the frame sizes of each video frame in the video frame sequence to be verified and each video frame in the frame difference sequence to be verified are the same, for example, both are 256 × 256. Specifically, the frame size can be further compressed or expanded according to factors such as computing power, but it should remain consistent.

[0040] Among them, the video anomaly detection model can be an existing prediction model. In order to further improve the prediction accuracy, the video anomaly detection model can also be a network model pre-built and trained by relevant technical personnel according to actual needs.

[0041] like Figure 1B The schematic diagram of the architecture of a video anomaly detection model is shown. The video anomaly detection model 10 includes a motion codec module 11, an appearance codec module 12 and a memory module 13; the motion codec module 11 includes a motion encoder 110 and a motion decoder 111; the appearance coding module 12 includes an appearance encoder 120 and an appearance decoder 121. The output end of the motion encoder 110 and the output end of the appearance encoder 120 are respectively connected to the input end of the memory module 13; the output end of the memory module 13 is respectively connected to the input end of the motion decoder 111 and the input end of the appearance decoder 121. The motion encoder 110 is used to receive a frame difference sequence to be verified, and the appearance encoder 120 is used to receive a video frame sequence to be verified.

[0042] The video anomaly detection model constructed in this embodiment is a prediction model based on a dual-stream structure. Two encoders are used to extract and analyze the features of the frame video sequence and the frame difference sequence respectively, and a joint memory module is designed to fuse and match the features of the two sequences. Finally, the two prediction results are fused to obtain the target predicted video frame, thereby performing video anomaly detection.

[0043] Specifically, the video frame sequence to be verified and the frame difference sequence to be verified are respectively input into a pre-trained video anomaly detection model to obtain a target predicted video frame output by the model.

[0044] In a specific embodiment, the video frame sequence to be verified and the frame difference sequence to be verified are respectively input into a pre-trained video anomaly detection model to obtain a target predicted video frame output by the model, including:

[0045] Step a11: input the video frame sequence to be verified into the appearance codec module in the video anomaly detection model, and the appearance encoder in the appearance codec module performs image feature processing to obtain video appearance features.

[0046] Among them, the appearance codec module is used to extract the appearance characteristics of static scenes and objects in the video frame, and predict the next frame by inputting several consecutive frame sequences of sampling. The appearance codec module can use the U-Net network model result. The appearance encoder and appearance decoder in the appearance codec module are both constructed based on three residual convolution modules.

[0047] Specifically, the video frame sequence to be verified is input into the appearance encoder in the appearance encoding and decoding module for image feature processing, and the video appearance features output by the appearance encoder are obtained. For example, assuming that the appearance encoder adopts E a Indicates that the video frame sequence to be verified is F1, F2, …, F t , then the video appearance feature Z a The expression is as follows:

[0048]

[0049] in, Represents the parameters of the appearance encoder.

[0050] Step a12: input the frame difference sequence to be verified into the motion coding module of the video anomaly detection model, and the motion encoder of the motion coding module performs image feature processing to obtain video motion features.

[0051] The motion encoding and decoding module is used to convert the input frame difference sequence into image feature processing and convert it into video motion features.

[0052] For example, assuming that the motion encoder adopts E m Indicates that the frame difference sequence to be verified is X1, X2, …, X t , then the video motion feature Z m The expression is as follows:

[0053]

[0054] in, Represents the parameters of the motion encoder.

[0055] Step a2: input the video appearance features and the video motion features into the memory module of the video anomaly detection model for feature fusion processing to obtain the video fusion features, and perform feature decomposition on the video fusion features to obtain the target appearance features and the target motion features.

[0056] The memory module is used to efficiently learn various normal modes of appearance and motion. During the model training phase, features that are highly similar to the memory items are considered to be features corresponding to normal events, and are therefore given higher weights, resulting in a greater error between abnormal samples and the final predicted normal samples. By combining the memory module, appearance features and motion features can share information, allowing the model to more effectively detect abnormalities in videos.

[0057] In a specific implementation, the video appearance features and the video motion features are input into the memory module of the video anomaly detection model for feature fusion processing to obtain the video fusion features, and the video fusion features are feature decomposed to obtain the target appearance features and the target motion features, including:

[0058] Step a21: perform feature splicing on the video appearance features and the video motion features to obtain fused query features.

[0059] Specifically, the video appearance feature Z a and video motion feature Z m Connect the end to end and fuse them to get the fused query feature Z k .

[0060] Step a22: determine the feature similarity between the fused query feature and at least one memory item stored in the current time period.

[0061] Calculate the query feature Z based on cosine similarity and softmax operation k With each memory item m i The degree of similarity of the characteristics of k,i It should be noted that the memory item may specifically be a fusion query feature retained in a historical prediction time period, and is continuously iteratively updated based on each prediction time period.

[0062] Step a23: Generate video fusion features based on feature similarity, each memory item and fusion query feature.

[0063] Specifically, according to the feature similarity, the fusion query features are adjusted, that is, Z k With m i Get closer and generate new video fusion features Make it closer to the stored memory item, that is, the normal feature. The expression is as follows:

[0064]

[0065] Where N is the total number of memory items.

[0066] Step a24, perform feature decomposition on the video fusion features to obtain target appearance features and target motion features.

[0067] Fusion of video features Disassemble to obtain the target appearance features and target motion characteristics

[0068] Step a31: input the target appearance features into the appearance decoder of the appearance encoding and decoding module to perform video frame prediction to obtain a predicted video frame.

[0069] Specifically, the appearance decoder D a According to the target appearance characteristics Generate predicted video frames

[0070]

[0071] in, Represents the parameters of the appearance decoder.

[0072] Step a32: input the target motion feature into the motion decoder of the motion encoding and decoding module to perform difference frame prediction to obtain a predicted difference frame.

[0073] Specifically, the motion decoder D m According to the target motion characteristics Generate prediction difference frame

[0074]

[0075] in, Represents the parameters of the motion decoder.

[0076] Step a4: Obtain a target predicted video frame according to the predicted video frame and the predicted difference frame.

[0077] The above predicted video frame and predicted difference frame Directly perform frame fusion to obtain the target predicted video frame

[0078] According to the target verification video frame and the target prediction video frame, video anomaly detection is performed on the target verification video frame to obtain anomaly detection results.

[0079] Among them, the target prediction video frame at the end of the target verification video frame corresponds to the real frame, and the abnormality of the video frame is quantified by calculating the difference between the target prediction video frame and its corresponding real frame (target verification video frame). In this step, Peak Signal-to-Noise Ratio (PSNR) and its derivative forms can be used as the core indicators for evaluating abnormal events.

[0080] In a specific embodiment, according to the target verification video frame and the target prediction video frame, video anomaly detection is performed on the target verification video frame to obtain an anomaly detection result, including:

[0081] Step b1: determine the peak signal-to-noise ratio according to the target verification video frame and the target prediction video frame.

[0082] Specifically, target verification video frame and target prediction video frame Verify video frame F with target t+2 Peak signal-to-noise ratio is determined as follows:

[0083]

[0084] Where N represents the number of pixels per frame.

[0085] Step b2: determining a target memory item based on the feature similarity between at least one memory item stored in the memory module of the current video frame in the current time period and the fused query feature.

[0086] Among them, the target memory item m δ The memory module of the current video frame in the current time period stores the memory items that are related to the fusion query feature Z k The memory item with the highest feature similarity.

[0087] Step b3: Determine the L2 distance based on the target memory item and the fused query feature.

[0088] Specifically, the L2 distance D(z t ,m) is determined as follows:

[0089]

[0090] in, represents the fused query feature of the tth frame; m δ is the target memory item; K is the number of frames.

[0091] It should be noted that if the distance between the fused query feature corresponding to the input video frame and the most similar memory item is too large, it can be considered that the video has an abnormal event at the tth frame. This evaluation standard can effectively avoid the problem that when there is an abnormal event in the video frame in the input video sequence, the query feature corresponding to the abnormal event is mistakenly regarded as a normal feature by the joint memory module, resulting in a decrease in the PSNR value corresponding to the subsequent abnormal frame.

[0092] Step b4: Determine the target outlier value based on the peak signal-to-noise ratio and the L2 distance.

[0093] The target outlier value can be a weighted sum of the peak signal-to-noise ratio (PSNR) and the L2 distance, which is used to evaluate the probability of an abnormal event in a video frame. β is a preset weight parameter. A high outlier value indicates that an abnormal event is more likely to exist in the corresponding frame. The target outlier value S(t+2) of the target prediction video frame t+2 is determined as follows:

[0094] S(t+2)=β(1-P(t+2))+(1-β)D(t);

[0095] Step b5: perform video anomaly detection on the target verification video frame according to the target anomaly value to obtain an anomaly detection result.

[0096] The larger the target outlier value, the greater the possibility that the video frame is abnormal; the smaller the target outlier value, the smaller the possibility that the video frame is abnormal.

[0097] Exemplarily, an outlier threshold may be preset, which may be preset according to actual needs. If the target outlier value is greater than the preset outlier threshold, it is considered that the video sequence to which the target verification video frame belongs has an anomaly; if the target outlier value is not greater than the preset outlier threshold, it is considered that the video sequence to which the target verification video frame belongs has no anomaly.

[0098] The technical solution of the embodiment of the present invention fuses the spatiotemporal features of the past frame sequence, that is, the video appearance features and the video motion features, through a two-time fusion scheme, and fuses the predicted frame with the predicted frame difference, thereby realizing the prediction of the second frame in the future to obtain the video anomaly detection result. By comparing the difference between the predicted frame and the actual frame, it is judged whether an abnormal event has occurred in the video. Only basic convolution kernels and automatic codecs are used in feature extraction and prediction, and the processing flow of the memory module is simplified. The computing power requirement is extremely low, and the high processing speed requirement can be met, thereby improving the efficiency of video anomaly detection. In addition, the method retains all the complex features of the original video to the greatest extent, and uses the memory module to obtain the relationship between the static (appearance) features and the dynamic (motion) features of the video, and then through the prediction of the second frame in the future, the accuracy of video anomaly detection is improved.

[0099] The present invention also provides a model training method for a video anomaly detection model, and the specific steps are as follows:

[0100] Step c1, obtaining normal video frame sequence samples, and generating normal difference frame sequence samples according to the normal video frame sequence samples.

[0101] Exemplarily, a video sequence may be extracted from a video without abnormalities as a normal video frame sequence sample, and a difference calculation is performed on continuous frames in the normal video frame sequence to obtain a normal difference frame sequence sample.

[0102] Step c21: input the normal video frame sequence samples into the appearance codec module in the pre-built initial anomaly detection model, and the appearance encoder in the appearance codec module performs image feature processing to obtain predicted appearance features.

[0103] Step c22: input the normal difference frame sequence samples into the motion coding module of the initial anomaly detection model, and the motion encoder of the motion coding module performs image feature processing to obtain predicted motion features.

[0104] Step c3: input the predicted appearance features and the predicted motion features into the memory module in the initial anomaly detection model for feature fusion processing to obtain predicted fusion features, and perform feature decomposition on the predicted fusion features to obtain predicted appearance features and predicted motion features.

[0105] Step c41: input the predicted appearance features to the appearance decoder of the appearance encoding and decoding module to perform video frame prediction to obtain a reference video frame.

[0106] Step c42: input the predicted motion features into the motion decoder of the motion coding module to perform difference frame prediction to obtain a reference difference frame.

[0107] Step c5: determining a first loss value according to the predicted video frame and the corresponding normal video frame, and determining a second loss value according to the reference difference frame and the corresponding normal difference frame.

[0108] Specifically, if the predicted video frame is The corresponding normal video frame is F t+1 , then the first loss value L of the appearance codec module a is determined as follows:

[0109]

[0110] If the reference difference frame is The corresponding normal difference frame is X t+1 , then the second loss value L of the motion codec module m is determined as follows:

[0111]

[0112] Step c6: Perform model training on the initial anomaly detection model according to the first loss value and the second loss value until a preset model training end condition is met to obtain a video anomaly detection model.

[0113] The model training end condition can be that the number of iterations reaches a threshold, or that the loss value tends to be stable or reaches a preset loss value threshold.

[0114] Specifically, if the first loss value and the second loss value are both greater than the preset loss value threshold, it can be considered that the model training has reached the optimal level, and the model training is terminated to obtain a video anomaly detection model.

[0115] Embodiment 2

[0116] Figure 2A This is an overall architecture diagram of a video anomaly detection method provided in Embodiment 2 of the present invention. This embodiment provides a preferred example based on the above embodiments.

[0117] The video anomaly detection model of this embodiment is an anomaly detection model based on a dual-stream structure. The basic principle is to assume that most of the video content should be non-abnormal. Therefore, future video frames predicted based on normal video clips should also tend to be normal. Finally, by comparing the difference between the predicted frame and the actual frame, it is determined whether an abnormal event occurs in the video.

[0118] The appearance codec module is used to extract the appearance features of static scenes and objects in video sequence frames, and predict the next frame by inputting several consecutive frames of sampling. Figure 2B The overall architecture diagram of an appearance encoder-decoder module is shown in FIG. The structure adopts a U-Net structure, in which a two-dimensional (2D) convolutional neural network is used as the backbone network of the appearance encoder-decoder module. Both the appearance encoder and the appearance decoder are built based on three residual convolution modules. For the appearance encoder, only batch normalization is used after the module. For the appearance decoder, the pixel shuffle layer is used to replace the convolution layer after each module to improve the resolution. In addition, all modules of the appearance encoder and appearance decoder use LeakyReLU as the activation layer. Between the same resolution feature layers of the appearance encoder and appearance decoder, jump connections are used to suppress the gradient explosion problem.

[0119] Specifically, F1, F2, ..., F t Adjust to a 256×256 stacked video frame sequence. The specific frame size can be further compressed or expanded based on specific computing power considerations. t Input to appearance encoder E a , after being processed by the appearance encoder, the video appearance feature Z is obtained a :

[0120]

[0121] in, Represents the parameters of the appearance encoder.

[0122] The motion encoder extracts features from the video difference sequence and outputs the video motion feature Z m , video appearance feature Za and video motion feature Z m Generate fusion features through memory module Will Decompose to obtain target appearance features The detailed processing methods of the motion codec module and the memory module are described in the subsequent sections and will not be described here. a According to the target appearance characteristics Generate prediction frame The specific process is as follows:

[0123]

[0124] in, Represents the parameters of the appearance decoder.

[0125] During the training phase, the goal is to minimize the predicted frame With its true value F t+1 The L2 distance between them makes them closer, so the loss function L of the appearance encoding and decoding module is a The definition is as follows:

[0126]

[0127] Through the loss value L a The appearance encoding and decoding module is continuously trained until the loss value stabilizes or reaches the set threshold.

[0128] like Figure 2C The overall architecture of a motion codec module is shown in FIG. In the motion autoencoder, the technology firstly performs the following steps on the continuous video frames F1, F2, …, F t ,F t+1 Subtract the pixels of the t+1th frame from the tth frame to obtain t consecutive frame differences X1, X2, ..., X t The continuous frame differences are adjusted to stacked differences of 256×256. The frame size can be further compressed or expanded based on the specific computing power, but it is consistent with the frame size in the appearance codec module. The number of frames is also consistent.

[0129] Specifically, the continuous frame differences X1, X2, ..., X t Input to motion encoder E m In the video, the motion feature Z is obtained after the motion encoder is processed and output. m :

[0130]

[0131] in, Represents the parameters of the motion encoder.

[0132] Video appearance features Z a and video motion feature Z m Generate fusion features through memory module Will Decompose to obtain target appearance features Will Decompose to obtain target motion features Motion Decoder D m According to the target appearance characteristics Generate prediction frame The specific process is as follows:

[0133]

[0134] in, Represents the parameters of the motion decoder.

[0135] In the training phase, in order to make the predicted frame difference Closer to the true difference X t+1 , which controls the minimization of the L2 distance between them. Therefore, the loss function L of the motion codec module m The definition is as follows:

[0136]

[0137] Through the loss value L m The motion codec module is continuously trained until the loss value stabilizes or reaches the set threshold.

[0138] like Figure 2D The overall architecture diagram of a memory module is shown. It is mainly used to efficiently learn various normal modes of video appearance and video motion. During the training phase, features that are highly similar to memory items can be considered as features corresponding to normal events, and therefore are given higher weights, causing a larger error between abnormal samples and the final predicted normal samples. Through the memory module, appearance and motion features can share information, allowing the model to more effectively detect abnormalities in the video. This embodiment simplifies the structure of the memory module, and only uses one layer of memory modules to obtain the final fusion features.

[0139] The specific working principle of the memory module is as follows:

[0140] Query features of the memory module: The video appearance feature Z a and video motion feature Z m Connect the end to end and fuse them to get the query feature Z k .

[0141] Memory item: A memory module contains multiple memory items m n , responsible for storing different normal event characteristics.

[0142] Memory module read operation: Calculate query feature Z based on cosine similarity and softmax operation k With each memory item m i The similarity of k,i , and adjust the query features according to the similarity, that is, Z k With m i Get closer and generate new features Make it closer to the stored memory item, that is, the normal feature. That is:

[0143]

[0144] Where N is the total number of memory items.

[0145] Memory module update operation: Calculate the memory item m based on cosine similarity and softmax operation i With each query feature Z k The similarity v k,i , and adjust the memory items according to the similarity, and its operation process is opposite to reading.

[0146] In the above steps and Directly fuse to get a new prediction frame And this is taken as the final prediction result. Calculate its difference with the real frame F t+2 The difference between the two is used to quantify the abnormality of the video frame. In this step, the peak signal-to-noise ratio (PSNR) and its derivative forms are used as the core indicators for evaluating abnormal events. The specific operations are as follows:

[0147] Calculate PSNR: where N is the number of pixels per frame. A lower PSNR value means that the frame is more likely to be anomalous.

[0148]

[0149] Where N represents the number of pixels per frame.

[0150] Calculate Z k With m δ L2 distance between: The memory items in the memory module store various normal patterns. In the test phase, when the input video frame is normal, its query feature and the memory item should have a high similarity, while the opposite is true for abnormal samples. Therefore, in addition to the PSNR value, the query feature Z can also be calculated k The nearest similar item m δ The L2 distance between them is:

[0151]

[0152] in, is the query feature of the tth frame, m δ is with The most similar memory item. If the query feature corresponding to the input video frame is too far away from the most similar memory item, it can be considered that the video has an abnormal event at the tth frame. This evaluation criterion can effectively avoid the problem that when there is an abnormal event in the video frame in the input video sequence, the query feature corresponding to the abnormal event is mistakenly regarded as a normal feature by the joint memory module, resulting in a decrease in the PSNR value corresponding to the subsequent abnormal frame.

[0153] The final anomaly score is a weighted sum of PSNR and L2 distance, which is used to evaluate the probability of abnormal events in a video frame, where β is a weight parameter. A high anomaly score indicates that an abnormal event is more likely to exist in the corresponding frame. The target anomaly value S(t+2) of the predicted video frame t+2 is determined as follows:

[0154] S(t+2)=β(1-P(t+2))+(1-β)D(t);

[0155] This embodiment uses a two-time fusion scheme to fuse the spatiotemporal features (i.e., appearance features and motion features) of the past frame sequence, and fuses the predicted frame with the predicted frame difference, so as to predict the second frame in the future and obtain the video anomaly detection result. First, the video is preprocessed to obtain the sampled frame sequence and the difference sequence of the corresponding frame. Two encoders are used to extract and analyze the features of the two sequences respectively, and a joint memory module is designed to fuse and match the features of the two sequences. Then the matched features are re-decomposed and applied to the two sequences respectively to obtain a predicted frame and a predicted difference. Finally, the predicted frame and the predicted difference are fused to obtain the final predicted frame. By comparing the difference between the predicted frame and the actual frame, it is determined whether an abnormal event has occurred in the video. This method only uses basic convolution kernels and automatic codecs in feature extraction and prediction, and greatly simplifies the processing flow of the memory module, with extremely low computing power requirements, and can meet the requirements of high processing speed. In addition, this method retains all the complex features of the original video to the greatest extent, and uses a hybrid memory module to obtain the relationship between the static (appearance) features and dynamic (action) features of the video, and then improves the accuracy of anomaly detection by predicting the second frame in the future.

[0156] Compared with the traditional dual-stream prediction model, this embodiment uses frame difference instead of optical flow as input on the action feature stream, which greatly reduces the computing power requirements of the model. At the same time, the joint memory module is simplified, which reduces the cost of feature fusion and processing and speeds up the processing. Through two fusions, the fusion of predicted frames and predicted differences is added on the basis of feature fusion, achieving efficient and accurate prediction.

[0157] Embodiment 3

[0158] Figure 3 The schematic diagram of the structure of a video anomaly detection device provided in the third embodiment of the present invention. The video anomaly detection device provided in the embodiment of the present invention can be applied to the case where anomaly detection is performed on the video corresponding to a specific scene, such as a traffic accident or a natural disaster. The video anomaly detection device can be implemented in the form of hardware and / or software, such as Figure 3 As shown, the video anomaly detection device can be configured in an electronic device, and specifically includes: a target frame determination module 301, a frame prediction module 302 and a video anomaly detection module 303.

[0159] The target frame determination module 301 is used to obtain a video frame sequence to be verified and a target verification video frame, and determine a frame difference sequence to be verified according to the video frame sequence to be verified;

[0160] The frame prediction module 302 is used to input the video frame sequence to be verified and the frame difference sequence to be verified into a pre-trained video anomaly detection model to obtain a target predicted video frame output by the model;

[0161] The video anomaly detection module 303 is used to perform video anomaly detection on the target verification video frame according to the target verification video frame and the target prediction video frame to obtain an anomaly detection result.

[0162] The technical solution of the embodiment of the present invention fuses the spatiotemporal features of the past frame sequence, that is, the video appearance features and the video motion features, through a two-time fusion scheme, and fuses the predicted frame with the predicted frame difference, thereby realizing the prediction of the second frame in the future to obtain the video anomaly detection result. By comparing the difference between the predicted frame and the actual frame, it is judged whether an abnormal event has occurred in the video. Only basic convolution kernels and automatic codecs are used in feature extraction and prediction, and the processing flow of the memory module is simplified. The computing power requirement is extremely low, and the high processing speed requirement can be met, thereby improving the efficiency of video anomaly detection. In addition, the method retains all the complex features of the original video to the greatest extent, and uses the memory module to obtain the relationship between the static (appearance) features and the dynamic (motion) features of the video, and then through the prediction of the second frame in the future, the accuracy of video anomaly detection is improved.

[0163] Optionally, the video anomaly detection model includes a motion coding and decoding module, an appearance coding and decoding module and a memory module; the motion coding and decoding module includes a motion encoder and a motion decoder; and the appearance coding module includes an appearance encoder and an appearance decoder.

[0164] Optionally, the frame prediction module 302 includes:

[0165] an appearance feature prediction unit, used for inputting the video frame sequence to be verified into the appearance codec module in the video anomaly detection model, and the appearance encoder in the appearance codec module performs image feature processing to obtain video appearance features; and

[0166] A motion feature prediction unit, used for inputting the frame difference sequence to be verified into a motion codec module of a video anomaly detection model, and performing image feature processing by a motion encoder of the motion codec module to obtain video motion features;

[0167] A feature fusion unit, used for inputting the video appearance feature and the video motion feature into the memory module of the video anomaly detection model for feature fusion processing to obtain a video fusion feature, and performing feature decomposition on the video fusion feature to obtain a target appearance feature and a target motion feature;

[0168] A video frame prediction unit is used to input the target appearance feature into an appearance decoder of an appearance codec module to perform video frame prediction to obtain a predicted video frame; and

[0169] A frame difference prediction unit, used for inputting the target motion feature into a motion decoder of a motion coding module to perform difference frame prediction to obtain a predicted difference frame;

[0170] The target frame determination unit is used to obtain a target predicted video frame according to the predicted video frame and the predicted difference frame.

[0171] Optional feature fusion unit, specifically used for:

[0172] Performing feature splicing on the video appearance feature and the video motion feature to obtain a fused query feature;

[0173] Determining a feature similarity between the fused query feature and at least one memory item stored in the current time period;

[0174] Generate a video fusion feature according to the feature similarity, each of the memory items and the fusion query feature;

[0175] The video fusion features are decomposed to obtain target appearance features and target motion features.

[0176] Optionally, the video anomaly detection module 303 is specifically used to:

[0177] Determining a peak signal-to-noise ratio according to the target verification video frame and the target prediction video frame;

[0178] Determine a target memory item according to a feature similarity between at least one memory item stored in a memory module of a current video frame in a current time period and a fusion query feature;

[0179] Determining an L2 distance based on the target memory item and the fused query feature;

[0180] Determining a target outlier according to the peak signal-to-noise ratio and the L2 distance;

[0181] According to the target abnormal value, video abnormality detection is performed on the target verification video frame to obtain an abnormality detection result.

[0182] Optionally, the device further includes a model training module, and the model training module is specifically used for:

[0183] Acquire normal video frame sequence samples, and generate normal difference frame sequence samples according to the normal video frame sequence samples;

[0184] Inputting the normal video frame sequence samples into the appearance codec module in the pre-built initial anomaly detection model, the appearance encoder in the appearance codec module performs image feature processing to obtain predicted appearance features; and,

[0185] Inputting the normal difference frame sequence samples into the motion coding module of the initial anomaly detection model, and performing image feature processing by the motion encoder of the motion coding module to obtain predicted motion features;

[0186] Inputting the predicted appearance feature and the predicted motion feature into the memory module in the initial anomaly detection model for feature fusion processing to obtain a predicted fusion feature, and performing feature decomposition on the predicted fusion feature to obtain a predicted appearance feature and a predicted motion feature;

[0187] Inputting the predicted appearance features into an appearance decoder of an appearance codec module to perform video frame prediction to obtain a reference video frame; and

[0188] Inputting the predicted motion features into a motion decoder of a motion coding module to perform difference frame prediction to obtain a reference difference frame;

[0189] Determining a first loss value according to the predicted video frame and the corresponding normal video frame, and determining a second loss value according to the reference difference frame and the corresponding normal difference frame;

[0190] According to the first loss value and the second loss value, the initial anomaly detection model is trained until a preset model training end condition is met to obtain a video anomaly detection model.

[0191] The video anomaly detection device provided in the embodiment of the present invention can execute the video anomaly detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0192] Embodiment 4

[0193] Figure 4 A schematic diagram of the structure of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0194] like Figure 4 As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0195] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0196] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 performs the various methods and processes described above, such as a video anomaly detection method.

[0197] In some embodiments, the video anomaly detection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the video anomaly detection method described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the video anomaly detection method in any other suitable manner (e.g., by means of firmware).

[0198] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0199] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0200] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0201] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0202] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0203] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0204] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0205] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A video anomaly detection method, characterized in that: include: Acquire a sequence of video frames to be verified and a target verification video frame, and determine a sequence of frame difference values ​​to be verified according to the sequence of video frames to be verified; Inputting the to-be-verified video frame sequence and the to-be-verified frame difference sequence into a pre-trained video anomaly detection model respectively to obtain a target predicted video frame output by the model; According to the target verification video frame and the target prediction video frame, video anomaly detection is performed on the target verification video frame to obtain an anomaly detection result.

2. The method according to claim 1, characterized in that: The video anomaly detection model includes a motion coding module, an appearance coding module and a memory module; the motion coding module includes a motion encoder and a motion decoder; the appearance coding module includes an appearance encoder and an appearance decoder.

3. The solution according to claim 2, characterized in that: The step of inputting the to-be-verified video frame sequence and the to-be-verified frame difference sequence into a pre-trained video anomaly detection model to obtain a target predicted video frame output by the model includes: Inputting the video frame sequence to be verified into the appearance codec module in the video anomaly detection model, and performing image feature processing by the appearance encoder in the appearance codec module to obtain video appearance features; and The frame difference sequence to be verified is input into the motion codec module of the video anomaly detection model, and the motion encoder of the motion codec module performs image feature processing to obtain video motion features; Inputting the video appearance features and the video motion features into the memory module of the video anomaly detection model for feature fusion processing to obtain video fusion features, and performing feature decomposition on the video fusion features to obtain target appearance features and target motion features; Inputting the target appearance feature into the appearance decoder of the appearance codec module to perform video frame prediction to obtain a predicted video frame; and Inputting the target motion feature into a motion decoder of a motion coding module to perform difference frame prediction to obtain a predicted difference frame; A target predicted video frame is obtained according to the predicted video frame and the predicted difference frame.

4. The method according to claim 3, characterized in that The step of inputting the video appearance features and the video motion features into the memory module of the video anomaly detection model for feature fusion processing to obtain video fusion features, and performing feature decomposition on the video fusion features to obtain target appearance features and target motion features includes: Performing feature splicing on the video appearance feature and the video motion feature to obtain a fused query feature; Determining a feature similarity between the fused query feature and at least one memory item stored in the current time period; Generate a video fusion feature according to the feature similarity, each of the memory items and the fusion query feature; The video fusion features are decomposed to obtain target appearance features and target motion features.

5. The method according to claim 4, characterized in that The performing video anomaly detection on the target verification video frame according to the target verification video frame and the target prediction video frame to obtain an anomaly detection result includes: Determining a peak signal-to-noise ratio according to the target verification video frame and the target prediction video frame; Determine a target memory item according to a feature similarity between at least one memory item stored in a memory module of a current video frame in a current time period and a fusion query feature; Determining an L2 distance based on the target memory item and the fused query feature; Determining a target outlier according to the peak signal-to-noise ratio and the L2 distance; According to the target abnormal value, video abnormality detection is performed on the target verification video frame to obtain an abnormality detection result.

6. The method according to claim 1, characterized in that The model training method of the video anomaly detection model is as follows: Acquire normal video frame sequence samples, and generate normal difference frame sequence samples according to the normal video frame sequence samples; Inputting the normal video frame sequence samples into the appearance codec module in the pre-built initial anomaly detection model, the appearance encoder in the appearance codec module performs image feature processing to obtain predicted appearance features; and, Inputting the normal difference frame sequence samples into the motion coding module of the initial anomaly detection model, and performing image feature processing by the motion encoder of the motion coding module to obtain predicted motion features; Inputting the predicted appearance feature and the predicted motion feature into the memory module in the initial anomaly detection model for feature fusion processing to obtain a predicted fusion feature, and performing feature decomposition on the predicted fusion feature to obtain a predicted appearance feature and a predicted motion feature; Inputting the predicted appearance features into an appearance decoder of an appearance codec module to perform video frame prediction to obtain a reference video frame; and Inputting the predicted motion features into a motion decoder of a motion coding module to perform difference frame prediction to obtain a reference difference frame; Determining a first loss value according to the predicted video frame and the corresponding normal video frame, and determining a second loss value according to the reference difference frame and the corresponding normal difference frame; According to the first loss value and the second loss value, the initial anomaly detection model is trained until a preset model training end condition is met to obtain a video anomaly detection model.

7. A video anomaly detection device, characterized in that: include: A target frame determination module is used to obtain a sequence of video frames to be verified and a target verification video frame, and determine a sequence of frame difference values ​​to be verified based on the sequence of video frames to be verified; A frame prediction module, used to input the video frame sequence to be verified and the frame difference sequence to be verified into a pre-trained video anomaly detection model, respectively, to obtain a target predicted video frame output by the model; The video anomaly detection module is used to perform video anomaly detection on the target verification video frame according to the target verification video frame and the target prediction video frame to obtain an anomaly detection result.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the video anomaly detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video anomaly detection method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the video anomaly detection method according to any one of claims 1 to 6.

Images