Video anomaly detection method, apparatus, electronic device, and computer program product

Abstract

The present disclosure relates to a video anomaly detection method, device, electronic equipment and computer program product. The video anomaly detection method comprises: obtaining a pre-constructed video anomaly detection model, the video anomaly detection model comprising a video analysis module, an appearance anomaly detection network and a motion anomaly detection network; extracting foreground video frame images, scene images and optical flow images of a target object from multiple frames of to-be-detected video frame images through the video analysis module, obtaining future video frame images based on the foreground video frame images and the scene images through the appearance anomaly detection network, and obtaining reconstructed optical flow images based on the optical flow images and the scene images through the motion anomaly detection network; and determining a detection result of the target object by using the future video frame images and the reconstructed optical flow images. The present disclosure can comprehensively consider appearance information, motion information and scene information of the target object when detecting the anomaly of the target object in the to-be-detected video, thereby improving the accuracy of the detection result.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to video anomaly detection methods, apparatus, electronic devices, and computer program products. Background Technology

[0002] Video anomaly detection methods refer to the use of intelligent devices deployed in areas such as industrial parks and residential communities to detect abnormal events in surveillance videos, providing a strong guarantee for the orderly conduct of production and daily life.

[0003] In related technologies, when performing anomaly detection on target objects in surveillance videos, the impact of the scene environment on the detection results is not considered, resulting in low accuracy of the detection results. Summary of the Invention

[0004] In view of the above, exemplary embodiments of this disclosure provide a video anomaly detection method, apparatus, electronic device, and computer program product to address the problems existing in the related art.

[0005] A first aspect of the exemplary embodiments of this disclosure provides a video anomaly detection method, comprising:

[0006] Acquire the test video of the target object and extract multiple test video frame images from the test video;

[0007] Obtain a pre-built video anomaly detection model, which includes a video parsing module and a parallel dual-branch anomaly detection module. The parallel dual-branch anomaly detection module includes an appearance anomaly detection network and a motion anomaly detection network.

[0008] Multiple test video frames are input into the video anomaly detection model. The video parsing module extracts the foreground video frame image, scene image, and optical flow image of the target object from the multiple test video frames. The appearance anomaly detection network obtains the future video frame image of the target object based on the foreground video frame image and scene image. The motion anomaly detection network obtains the reconstructed optical flow image of the target object based on the optical flow image and scene image.

[0009] The detection results of the target object are determined by using future video frame images and reconstructed optical flow images.

[0010] A second aspect of the exemplary embodiments of this disclosure provides a video anomaly detection apparatus, comprising:

[0011] The acquisition module is used to acquire the video to be tested of the target object;

[0012] The processing module is used to extract multiple frames of video images from the video under test;

[0013] The acquisition module is also used to acquire a pre-built video anomaly detection model, which includes a video parsing module and a parallel dual-branch anomaly detection module. The parallel dual-branch anomaly detection module includes an appearance anomaly detection network and a motion anomaly detection network.

[0014] The prediction module is used to input multiple test video frame images into the video anomaly detection model. The video parsing module extracts the foreground video frame image, scene image and optical flow image of the target object from the multiple test video frame images respectively. The appearance anomaly detection network obtains the future video frame image of the target object based on the foreground video frame image and scene image. The motion anomaly detection network obtains the reconstructed optical flow image of the target object based on the optical flow image and scene image.

[0015] The processing module is also used to determine the detection result of the target object by utilizing future video frame images and reconstructing optical flow images.

[0016] A third aspect of the exemplary embodiments of this disclosure provides an electronic device, comprising:

[0017] At least one processor;

[0018] Memory used to store at least one processor-executable instruction;

[0019] At least one processor is used to execute instructions to implement the steps of the above method.

[0020] A fourth aspect of the exemplary embodiments of this disclosure provides a computer program product including a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described above.

[0021] The above-described at least one technical solution adopted in the exemplary embodiments of this disclosure can achieve the following beneficial effects: by acquiring the test video of the target object and extracting multiple test video frame images from the test video; acquiring a pre-constructed video anomaly detection model, the video anomaly detection model including a video parsing module and a parallel dual-branch anomaly detection module, the parallel dual-branch anomaly detection module including an appearance anomaly detection network and a motion anomaly detection network; inputting multiple test video frame images into the video anomaly detection model, extracting the foreground video frame image, scene image and optical flow image of the target object from the multiple test video frame images respectively through the video parsing module, obtaining the future video frame image of the target object based on the foreground video frame image and scene image through the appearance anomaly detection network, and obtaining the reconstructed optical flow image of the target object based on the optical flow image and scene image through the motion anomaly detection network; and determining the detection result of the target object using the future video frame image and the reconstructed optical flow image, which can comprehensively consider the appearance information, motion information and scene information of the target object when performing anomaly detection on the target object in the test video, thereby improving the accuracy of the detection result. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the exemplary embodiments of this disclosure, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic diagram of the network architecture of the video anomaly detection model provided in an exemplary embodiment of this disclosure is shown;

[0024] Figure 2A A schematic diagram of the structure of a traditional U-Net network in related technologies is shown;

[0025] Figure 2B A schematic diagram of the structure of a multilayer residual U-Net network provided in an exemplary embodiment of this disclosure is shown;

[0026] Figure 3 A schematic diagram of the structure of the first memory storage unit provided in an exemplary embodiment of this disclosure is shown;

[0027] Figure 4 This diagram illustrates the detection results of the video anomaly detection model provided in an exemplary embodiment of this disclosure during the testing phase.

[0028] Figure 5 A flowchart illustrating an exemplary embodiment of the video anomaly detection method provided in this disclosure is shown.

[0029] Figure 6 A schematic diagram of the structure of the video anomaly detection apparatus provided in an exemplary embodiment of this disclosure is shown;

[0030] Figure 7 A schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this disclosure is shown;

[0031] Figure 8 A schematic diagram of the structure of a computer system provided in an exemplary embodiment of this disclosure is shown. Detailed Implementation

[0032] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0033] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0034] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0035] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0036] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0037] Artificial intelligence (AI) is a technology that enables computers and machine language to exhibit intelligence and human-like thinking abilities, encompassing multiple fields such as learning and reasoning, speech recognition, autonomous driving, and object detection. With continuous technological development and innovation, AI is increasingly integrated into the national economy, injecting new growth momentum into various industries. In the current context of rapid growth in unstructured data such as images, audio, and video, computer vision has gradually become one of the most popular areas of AI, focusing on creating algorithmic models that can process, analyze, and understand visual data (images or videos) in the same way as humans.

[0038] Video anomaly detection refers to the ability of algorithmic models to analyze and understand anomalous events present in massive image or video data samples, and output easily understandable quantitative indicators. In a sense, a high-performance video anomaly detection algorithm can maximize the safety and normalcy of production and daily life, effectively improving people's work efficiency and quality of life. However, supporting the operation of such a massive algorithm presents unprecedented challenges to the network's scale, performance, reliability, and stability.

[0039] Video anomaly detection algorithms refer to the accurate detection of abnormal events in surveillance videos by intelligent devices deployed in areas such as industrial parks and residential communities, providing strong protection for the orderly operation of production and daily life. Deploying video anomaly detection algorithms to edge computing nodes, leveraging the powerful computing capabilities of intelligent servers, can more quickly detect non-compliant behaviors in areas such as industrial parks and residential communities, issuing timely and effective warnings to prevent unsafe incidents.

[0040] In related technologies, abnormal events in videos can be detected from both global and local perspectives, focusing on targets in areas where anomalies may occur, effectively mitigating the differences caused by the field of view. Alternatively, abnormal frames in the video can be identified first, and then abnormal events within those frames can be detected. However, none of these video anomaly detection methods consider the impact of the scene environment on the detection results, leading to low accuracy.

[0041] Therefore, in order to solve the above problems, the exemplary embodiments of this disclosure provide a video anomaly detection method, which detects abnormal events in the video by using a pre-built video anomaly detection model. The video anomaly detection model can combine the scene environment in the video to analyze the appearance and behavioral characteristics of the target object, thereby determining whether the behavior of the target object is abnormal and improving the accuracy of the detection results.

[0042] The video anomaly detection method provided in this exemplary embodiment can be executed by a terminal or by a chip applied to the terminal.

[0043] For example, the terminal may include one or more of the following: mobile phone, tablet computer, wearable device, in-vehicle device, laptop computer, ultra-mobile personal computer (UMPC), netbook, handheld computer (PDA), and wearable device based on augmented reality (AR) and / or virtual reality (VR) technology. The exemplary embodiments disclosed herein do not impose specific limitations on these.

[0044] With the continuous development of artificial intelligence technology, deep learning methods have gradually replaced traditional machine learning as the mainstream method in the field of video anomaly detection. Relying on the powerful neural networks they construct, deep learning algorithms can understand massive amounts of complex video data samples and make correct decisions. Based on whether data labels are needed during training, deep learning methods can be divided into supervised video anomaly detection methods, semi-supervised video anomaly detection methods, and unsupervised video anomaly detection methods. Since anomalous data samples are characterized by low frequency of occurrence, small data volume, numerous types, and difficulty in collection, the exemplary embodiment of this disclosure employs a semi-supervised anomaly detection method that uses only normal data samples during the training process.

[0045] Based on different theoretical premises, video anomaly detection methods can be categorized into reconstruction-based, prediction-based, and hybrid model-based methods. Reconstruction-based methods learn the inherent patterns of input data (in this case, normal data samples) to output results that are as similar as possible to the input data within the same time period. However, reconstruction-based methods may also reconstruct anomalous data samples very well, leading to a high false positive rate. Prediction-based methods learn from continuous normal data samples to output predicted data for the next time step. Hybrid methods integrate the above two methods into a single algorithm. Although the number of parameters and computational cost are increased to some extent, its robustness and accuracy are significantly improved. Therefore, based on the above analysis, the exemplary embodiment of this disclosure uses a semi-supervised video anomaly detection method based on a hybrid model.

[0046] Figure 1 A schematic diagram of the network architecture of the video anomaly detection model provided in an exemplary embodiment of this disclosure is shown. Figure 1 As shown, the network architecture of the video anomaly detection model 100 may include a video parsing module 110 and a parallel dual-branch anomaly detection module 120. The video parsing module 110 may include an object detection network 111, a scene extraction network 112, and an optical flow estimation network 113. The parallel dual-branch anomaly detection module 120 may include an appearance anomaly detection network 121 and a motion anomaly detection network 122. The appearance anomaly detection network 121 may include an appearance and scene-aware encoder 1211, a first memory storage unit 1212, a first scene-conditional variational autoencoder 1213, and an appearance and scene-aware decoder 1214. The motion anomaly detection network 122 may include a motion and scene-aware encoder 1221, a second memory storage unit 1222, a second scene-conditional variational autoencoder 1223, and a motion and scene-aware decoder 1224.

[0047] In this exemplary embodiment, the appearance and scene-aware encoder 1211 and the appearance and scene-aware decoder 1214 can be combined and referred to as the appearance and scene-aware autoencoder, and the motion and scene-aware encoder 1221 and the motion and scene-aware decoder 1224 can be combined and referred to as the motion and scene-aware autoencoder.

[0048] It should be noted that the appearance and scene-aware encoder 1211 and the motion and scene-aware encoder 1221 have the same structure; the first memory storage unit 1212 and the second memory storage unit 1222 have the same structure; the first scene-conditional variational autoencoder 1213 and the second scene-conditional variational autoencoder 1223 have the same structure; and the appearance and scene-aware decoder 1214 and the motion and scene-aware decoder 1224 have the same structure. In other words, the appearance anomaly detection network 121 and the motion anomaly detection network 122 have the same structure, ensuring the consistency of the overall structure of the parallel dual-branch anomaly detection module.

[0049] In practical applications, the exemplary embodiments of this disclosure can input multiple frames of original video frames over a preset time period (e.g., from time 1 to time t) into the video parsing module 110. Specifically, the target detection network 111 can be used to extract the appearance information of the target object from the multiple frames of original video frames to obtain a foreground video frame image; the scene extraction network 112 can be used to extract a scene image from the multiple frames of original video frames to obtain a scene image; and the optical flow estimation network 113 can be used to extract the motion information of the target object from the multiple frames of original video frames to obtain an optical flow image.

[0050] (1) Target detection network.

[0051] Since most abnormal events in surveillance videos originate from moving subjects, i.e., target objects in the foreground, the series of actions involved largely determine whether abnormal events exist in the original video frame image. These abnormal events stem from the appearance anomalies of the target object. Therefore, compared with the approach of directly inputting the original video frame image into a neural network model in related technologies, the exemplary embodiment of this disclosure can utilize the object detection network 111 to extract the foreground video frame image from the original video frame image, directly focusing on the target object in the foreground video frame image, making the input data of the subsequent appearance anomaly detection network 121 more consistent with the video anomaly detection task.

[0052] The target detection network 111 of this exemplary embodiment can use a multi-target tracking algorithm (Byte Track algorithm) implemented using a target tracking tool based on the PyTorch framework (such as MMTracking). The biggest difference between the Byte Track algorithm and other tracking algorithms is that the Byte Track algorithm does not simply remove low-scoring detection results, but rather utilizes the similarity between the detection boxes and the tracking trajectory to remove background from low-scoring detection results while retaining high-scoring detection results, thereby uncovering the real objects (including difficult samples such as occlusion and blurring), which can reduce missed detections and improve the coherence of the target object's motion trajectory.

[0053] (2) Extract the network for the scene.

[0054] Besides abnormalities in the moving subject itself, another cause of abnormal events in surveillance videos is the mismatch between the target object and the video scene. This mismatch refers to the target object exhibiting unexpected behavior in an inappropriate scene, such as a student running rapidly in a classroom or a pedestrian walking on a roadway. Because these abnormal events are highly scene-dependent, their behavior might be normal in one scene but abnormal in another, making accurate detection difficult. To address these characteristics, this exemplary embodiment uses a scene extraction network 112 to extract the scene from the surveillance video, which is then used to guide the generation of future video frame images and reconstructed optical flow images.

[0055] The scene extraction network 112 of this exemplary embodiment can use an image semantic segmentation model (DeeplabV3+ model). The Deeplab V3+ model mainly includes three parts: a backbone deep convolutional neural network for extracting image features, a dilated spatial convolutional pooling pyramid network, and a scene extraction decoder. The dilated spatial convolutional pooling pyramid network can include one convolutional layer with a kernel size of 1×1, three dilated convolutional layers with kernel sizes of 3×3 and different dilation coefficients, and one global pooling layer, used to process the output of the backbone deep convolutional neural network. Then, the scene extraction decoder concatenates the output of the dilated spatial convolutional pooling pyramid network at the image channel level, and performs convolution operations (such as using a 1×1 kernel size convolution operation to reduce the number of channels in the feature vector) on the output of the dilated spatial convolutional pooling pyramid network, ultimately forming a scene image of the original video frame image.

[0056] (3) For optical flow estimation networks.

[0057] Besides abnormal appearance of the target object and scene mismatch, motion abnormalities are also abnormal events in surveillance videos and may originate from abnormal motion of the target object, such as a motor vehicle driving in the wrong direction. The optical flow estimation network of the exemplary embodiment of this disclosure can use an enhanced optical flow prediction algorithm (FlowNet2.0) to extract optical flow information from the original video frame image by stacking multiple networks to obtain an optical flow image, which is then used to generate a reconstructed optical flow image.

[0058] At this time, the foreground video frame image, scene image and optical flow image processed by the video parsing module 110 can be input to the parallel dual-branch anomaly detection module 120 to obtain the future video frame image at time t+1 and the reconstructed optical flow image from time 1 to time t; wherein, the future video frame image is generated by the appearance anomaly detection network 121 based on the foreground video frame image and scene image, and the reconstructed optical flow image is generated by the motion anomaly detection network 122 based on the optical flow image and scene image.

[0059] Specifically, for the appearance anomaly detection network 121, a foreground video frame image is input into the appearance anomaly detection network 121. The appearance and scene awareness encoder 1211 is used to perform feature encoding on the foreground video frame image, learn the appearance features of the target object in the foreground video frame image, obtain the appearance feature vector of the target object, and input the appearance feature vector of the target object into the first memory storage unit 1212. The first memory storage unit 1212 is used to retrieve the memory memory through the memory addressing vector, use multiple memory items to perform linear combination to obtain the appearance query feature vector corresponding to the appearance feature vector, and input the appearance query feature vector into the first scene conditional variational autoencoder 1213. The first scene conditional variational autoencoder 1213 is used to perform feature concatenation and feature extraction operations on the appearance query feature vector and the scene image to obtain the target appearance and scene concatenation feature vector of the fused scene image, and input the target appearance and scene concatenation feature vector into the appearance and scene awareness decoder 1214. The appearance and scene awareness decoder 1214 is used to decompress the target appearance and scene concatenation feature vector to obtain the future video frame image.

[0060] (1) For appearance and scene perception encoders.

[0061] Figure 2A This diagram illustrates the structure of a traditional U-Net network in related technologies. Figure 2B A schematic diagram of the structure of a multilayer residual U-Net network provided in an exemplary embodiment of this disclosure is shown. Figure 2AThe diagram shows the structure of a traditional U-Net network in related technologies. A key feature of this traditional U-Net network is the inclusion of multiple skip connections between higher and lower-level networks to reduce information loss caused by convolutional operations. The U-Net network is a classic fully convolutional network with contraction and expansion phases. The left side of the network performs downsampling operations consisting of convolution and pooling, aiming to reduce the dimensionality of the input data and compress its feature representation. The right side performs upsampling operations consisting of deconvolution, aiming to decompress the compressed feature representation and restore its original dimensionality. Figure 2A As can be seen, the traditional U-Net network uses multiple identical small 3×3 convolutional kernels, and its nonlinear expressive power increases with the increase of network depth. In addition, the small-sized convolutional kernels also reduce the number of parameters involved in the computation, further improving the computational efficiency of the model.

[0062] As mentioned earlier, the deeper the network, the stronger its nonlinear expressive power. However, the gradient values ​​of each network layer become smaller and even disappear, eventually causing the network updates to become very slow or even stagnant. Based on this, the exemplary embodiments disclosed herein can modify the traditional U-Net network, proposing a novel multi-layer residual U-Net network for video anomaly detection.

[0063] like Figure 2B As shown, for a normal neural network, if the input data is x, then the output data can be represented as F(x), where F(·) represents the fitting function of the neural network. In a multilayer residual U-Net network, in addition to F(x), the output also includes the original input data. That is, the residual block, as a module of the neural network, outputs F(x) + x.

[0064] like Figure 2A As shown, in each convolutional layer of the traditional U-Net network, the kernel size is 3×3, and then ReLU activation is used for activation. However, in... Figure 2B In the multi-layer residual U-Net network shown, each residual block can include multiple convolutional layers with different kernel sizes, such as a 1×1 kernel layer, a 3×3 kernel layer, and a 1×1 kernel layer, for a total of three convolutional layers. In the novel multi-layer residual U-Net network, before and after the 3×3 convolution operation, 1×1 kernels are used for feature dimensionality reduction and dimensionality increase operations, respectively. By introducing residual blocks, detection accuracy can be maintained while reducing the computational cost of the model, ensuring that the output feature size remains unchanged.

[0065] As can be seen, the exemplary embodiments of this disclosure can replace the convolution operation performed by the traditional U-Net network using multiple convolution kernels of the same size with the residual convolution operation performed by multiple convolution kernels of different sizes, thereby further improving the computational efficiency and accuracy of the model.

[0066] It should be noted that in the appearance and scene-aware encoder, the input data is the foreground video frame image, and the output data is the appearance feature vector of the target object.

[0067] (2) For the first memory storage unit.

[0068] Since abnormal data samples are characterized by low occurrence rate, difficulty in collection, numerous categories, and small quantity, the method of the exemplary embodiment of this disclosure is an anomaly detection method based on a semi-supervised hybrid model. During the training phase, only normal data samples are used as training samples, which enables the model to learn the inherent regularity of video events and enable it to distinguish between normal events and abnormal events.

[0069] Because the multilayer residual U-Net network has strong generalization ability, abnormal data samples may be reconstructed very well by the multilayer residual U-Net network during the testing phase, resulting in a small loss error. This makes it difficult for the model to distinguish between normal and abnormal events, greatly increasing the false alarm rate of video anomaly detection results. Therefore, the exemplary embodiment of this disclosure sets up a first memory storage unit 1212 in the appearance anomaly detection network 121, which effectively reduces the loss error of normal data samples and increases the loss error of abnormal data samples, which is beneficial for the model to better detect abnormal events in the video.

[0070] Figure 3 A schematic diagram of the structure of a first memory storage unit provided in an exemplary embodiment of this disclosure is shown. Figure 3 As shown, the first memory storage unit mainly consists of two parts: the first memory addressing vector and the first memory memory. The first memory addressing vector is obtained by calculating the Softmax similarity (cosine similarity) between the input data (which can also be called the input feature vector) and multiple first memory storage items in the first memory memory. Here, the input feature vector f IN This can be the appearance feature vector of the target object mentioned above. The calculation process of the first memory addressing vector can be expressed as:

[0071]

[0072] Where, r i f represents the i-th first memory addressing vector, whose element sum is 1; IN Represents the input feature vector; m i d(f) represents the i-th primary memory storage item in the memory; IN ,mi ) represents the cosine similarity between the input feature vector and the i-th first memory storage item in the first memory; N represents the total number of first memory storage items in the first memory.

[0073]

[0074] Where ||·|| represents the modulus, i.e., the length of the vector.

[0075] like Figure 3 As shown, the first memory is a matrix with real values, which stores the input feature vector f. IN The input is stored in the first memory. The first memory storage unit can autonomously select the most typical first memory storage item (the memory storage item corresponding to the maximum cosine similarity) to process the input feature vector f. IN The target object's appearance query feature vector f is obtained by re-representation. OUT Among them, for the input feature vector f IN The process of re-representing can be expressed as:

[0076]

[0077] Among them, f OUT The vector represents the appearance query feature vector of the target object; r represents the first memory addressing vector; M represents the first memory memory; other parameters are described above and will not be repeated here.

[0078] Based on the input feature vector f IN The first memory storage unit can obtain multiple first memory addressing vectors through the above formula (1). Further searching is performed on the first memory storage unit by linearly combining the multiple first memory addressing vectors in the first memory storage unit with the corresponding multiple first memory items. Finally, the appearance query feature vector of the target object can be obtained through the above formula (3). Then, the appearance query feature vector of the target object is input into the first scene conditional variational autoencoder for the next step.

[0079] Based on this, the exemplary embodiment of this disclosure sets up a first memory storage unit between the appearance and scene-aware encoder and the appearance and scene-aware decoder, which can effectively increase the loss error of abnormal data samples while reducing the loss error of normal data samples, thereby widening the loss difference between the two types of data, which is beneficial for the model to accurately detect anomalies. In addition, in order to better improve the learning performance of the first memory storage unit on abnormal data samples, the exemplary embodiment of this disclosure applies cross-entropy loss to it and updates its parameters through backpropagation.

[0080] It should be noted that in the first memory storage unit, the input data is the appearance feature vector of the target object, and the output data is the appearance query feature vector of the target object.

[0081] (3) For the first scenario conditional variational autoencoder.

[0082] Deep learning models can be mainly divided into discriminative models and generative models. In recent years, with the popularity of generative tasks such as image generation and dialogue response generation, generative models have received increasing attention. In the method of the exemplary embodiment of this disclosure, the generative model used by the appearance anomaly detection network is a first scene conditional variational autoencoder, which is essentially a generative model with a deep latent space.

[0083] The first scene-conditional variational autoencoder may include a first encoder, a first latent space sampling layer, and a first decoder. Specifically, it concatenates the appearance query feature vector of the target object with a scene image to obtain a first appearance-scene concatenated feature vector. The first encoder can perform feature extraction and compression operations on the first appearance-scene concatenated feature vector to obtain a compressed appearance feature vector. The first latent space sampling layer is a representation of compressed data; its function is to learn data features and simplify data representation to find a certain pattern. It can be used to extract the appearance-scene latent space feature vector from the compressed appearance feature vector. The appearance-scene latent space feature vector and the scene image are then concatenated to obtain a second appearance-scene concatenated feature vector. The first decoder decompresses the second appearance-scene concatenated feature vector to obtain a target appearance-scene concatenated feature vector of the target object. This target appearance-scene concatenated feature vector can be a data sample similar to the appearance query feature vector of the target object.

[0084] The first scene-conditional variational autoencoder inputs label data not only during the encoding stage but also during the decoding stage, using vectors representing the label data. Based on this, the exemplary embodiment of this disclosure, by setting the first scene-conditional variational autoencoder, can use scene images as guiding conditions to flexibly control the generation process of future video frame images, making the generation results of future video frame images diverse and controllable, and achieving full-range detection of appearance anomalies and scene mismatch anomalies.

[0085] It should be noted that in the first scene-conditional variational autoencoder, the input data is the appearance query feature vector of the target object, and the output data is the concatenated feature vector of the target object's appearance and the scene.

[0086] (4) For appearance and scene perception decoders.

[0087] In the appearance and scene-aware decoder, the input data is the target appearance and scene stitched feature vector of the target object. The appearance and scene-aware decoder is used to decompress the target appearance and scene stitched feature vector of the target object to obtain the final future video frame image.

[0088] For the motion anomaly detection network 122, an optical flow image can be input into the network. The motion and scene-aware encoder 1221 is used to encode features of the optical flow image, learn the motion features of the target object in the optical flow image, obtain the motion feature vector of the target object, and input the motion feature vector of the target object into the second memory storage unit 1222. The second memory storage unit 1222 is used to retrieve memory through memory addressing vector, linearly combine multiple memory items to obtain the motion query feature vector corresponding to the motion feature vector, and input the motion query feature vector into the second scene conditional variational autoencoder 1223. The second scene conditional variational autoencoder 1223 is used to perform feature concatenation and feature extraction operations on the motion query feature vector and the scene image to obtain the target motion and scene concatenation feature vector of the fused scene image, and input the target motion and scene concatenation feature vector into the motion and scene-aware decoder 1224. The motion and scene-aware decoder 1224 is used to decompress the target motion and scene concatenation feature vector to obtain the reconstructed optical flow image.

[0089] As can be seen from the preceding text, the appearance anomaly detection network 121 and the motion anomaly detection network 122 have the same structure. For details on the structure and function of each module included in the motion anomaly detection network 122, please refer to the preceding text. Here, we will only briefly explain the differences between the two in terms of textual description.

[0090] (1) For motion and scene perception encoders.

[0091] The motion and scene-aware encoder can be the multi-layer residual U-Net network described earlier, which will not be elaborated upon here. The input data of the motion and scene-aware encoder is the optical flow image, and the output data is the motion feature vector of the target object.

[0092] (2) For the second memory storage unit.

[0093] The structure of the second memory storage unit is the same as that of the first memory storage unit described above, and will not be repeated here. The input feature vector of the second memory storage unit can be the motion feature vector of the target object described above, and the output data is the motion query feature vector of the target object. Referring to the structure of the first memory storage unit, the second memory storage unit mainly consists of two parts: the second memory addressing vector and the second memory memory. The calculation formulas involved in the second memory storage unit refer to the relevant technical formulas of the first memory storage unit, and will not be repeated here.

[0094] An exemplary embodiment of this disclosure provides a second memory storage unit between the motion and scene-aware encoder and the motion and scene-aware decoder. This effectively increases the loss error of anomalous data samples while reducing the loss error of normal data samples, thereby widening the loss difference between the two types of data and facilitating accurate anomaly detection by the model. Furthermore, to further enhance the learning performance of the second memory storage unit on anomalous data samples, an exemplary embodiment of this disclosure applies cross-entropy loss to it and updates its parameters through backpropagation.

[0095] (3) For the second scenario conditional variational autoencoder.

[0096] The structure of the second scene-conditional variational autoencoder is the same as that of the first scene-conditional variational autoencoder described above. The second scene-conditional variational autoencoder may include a second encoder, a second latent space sampling layer, and a second decoder. Specifically, it concatenates the motion query feature vector of the target object with the scene image to obtain a first motion-scene concatenated feature vector. The second encoder performs feature extraction and compression operations on the first motion-scene concatenated feature vector to obtain a motion-compressed feature vector. The second latent space sampling layer is a representation of compressed data; its function is to learn data features and simplify data representation to find a certain pattern. It can be used to extract motion-scene latent space feature vectors from the motion-compressed feature vector. The motion-scene latent space feature vectors are then concatenated with the scene image to obtain a second motion-scene concatenated feature vector. The second decoder decompresses the second motion-scene concatenated feature vector to obtain a target motion-scene concatenated feature vector of the target object. This target motion-scene concatenated feature vector can be a data sample similar to the motion query feature vector of the target object.

[0097] The second scene-conditional variational autoencoder inputs label data not only during the encoding stage but also during the decoding stage, using vectors representing the label data. Based on this, the exemplary embodiment of this disclosure, by setting a second scene-conditional variational autoencoder, can use the scene image as a guiding condition to flexibly control the generation process of the reconstructed optical flow image, making the generated result of the reconstructed optical flow image diverse and controllable, and achieving full-range detection of motion anomalies and scene mismatch anomalies.

[0098] It should be noted that in the second scene conditional variational autoencoder, the input data is the motion query feature vector of the target object, and the output data is the concatenated feature vector of the target motion and the scene of the target object.

[0099] (4) For motion and scene perception decoders.

[0100] In the motion and scene-aware decoder, the input data is the target motion and scene stitching feature vector of the target object. The motion and scene-aware decoder is used to decompress the target motion and scene stitching feature vector of the target object to obtain the final reconstructed optical flow image.

[0101] Based on this, the exemplary embodiment of this disclosure can set up two first scene conditional variational autoencoders and second scene conditional variational autoencoders with the same structure in the parallel dual-branch anomaly detection module, and use the scene image as a guiding condition to guide the generation of future video frame images and reconstructed optical flow images respectively.

[0102] In practical applications, during the training process of a video anomaly detection model, the exemplary embodiments of this disclosure can determine whether the video anomaly detection model has converged by calculating the loss of the video anomaly detection model to be constructed.

[0103] The input data of the video anomaly detection model in this exemplary embodiment consists of multiple original video frame images from time 1 to time t, and the final output data consists of a future video frame image at time t+1 and a reconstructed optical flow image from time 1 to time t. Based on this, the loss of the video anomaly detection model in this exemplary embodiment can include prediction loss and reconstruction loss. The L2 distance (i.e., mean squared error) between the input and output data is used to constrain the video anomaly detection model, defined as follows:

[0104]

[0105]

[0106] Among them, L pred Indicates the predicted loss; x t+1 Represents the actual future video frame image at time t+1; L represents the predicted future video frame image at time t+1; recon Represents the reconstruction loss; y 1:t This represents the true reconstructed optical flow image from time 1 to time t; This represents the predicted reconstructed optical flow image from time 1 to time t; the subscript 2 indicates the L2 distance; the superscript 2 indicates the square.

[0107] Because the parallel dual-branch anomaly detection module of the video anomaly detection model in this exemplary embodiment of the present disclosure is equipped with a first memory storage unit and a second memory storage unit, the feature vectors are recombinated by retrieving memory items in the memory memory through the memory addressing vector, ultimately increasing the difference in the loss function between normal data and anomaly data. The quality of the memory addressing vector directly determines the quality of the feature vectors finally output by the memory storage unit. Therefore, in order to obtain sparser memory addressing vectors, this exemplary embodiment of the present disclosure imposes a cross-entropy loss constraint on the first memory storage unit and the second memory storage unit, referred to as memory loss (including the first memory loss of the first memory storage unit and the second memory loss of the second memory storage unit). The memory loss in this exemplary embodiment of the present disclosure can be calculated by the following formula:

[0108] L memory =λ 1memory ·L 1memory +λ 2memory ·L 2memory (7)

[0109]

[0110]

[0111] Among them, L memory Indicating memory loss, r j Let L represent the j-th second memory addressing vector, whose element sum is 1, and let N' represent the total number of second memory storage items in the second memory memory. 1memory L represents the first memory loss of the first memory storage unit. 2memory λ represents the second memory loss of the second memory storage unit. 1memory λ represents the weight of the first memory loss in the first memory storage unit. 2memory This represents the weight of the second memory loss in the second memory storage unit. Other parameters are described above and will not be repeated here.

[0112] Here, cross-entropy loss describes the distance relationship between the true value and the output value, that is, the similarity between their probability distributions. In other words, the more similar the two probability distributions are, the smaller the value of the cross-entropy.

[0113] Furthermore, to avoid losing scene feature information in the video during the generation of future video frame images and reconstructed optical flow images, the parallel dual-branch anomaly detection module of the exemplary embodiment of this disclosure includes a first scene-conditional variational autoencoder (FRAE) and a second scene-conditional variational autoencoder (FRAE). As previously described, both the first and second scene-conditional variational autoencoders involve two stitching processes of the scene image during deep feature information extraction. This allows the first scene-conditional variational autoencoder to obtain the target appearance and scene stitching feature vector of the fused scene image, and the second scene-conditional variational autoencoder to obtain the target motion and scene stitching feature vector of the fused scene image. This avoids losing scene feature information during deep feature information extraction and improves the accuracy of video anomaly detection.

[0114] Based on this, the loss of the video anomaly detection model in the exemplary embodiments of this disclosure also includes a stitching loss, which may include the appearance and scene stitching loss of the first scene-conditional variational autoencoder and the motion and scene stitching loss of the second scene-conditional variational autoencoder. The calculation methods of the appearance and scene stitching loss and the motion and scene stitching loss are the same, and the calculation of the appearance and scene stitching loss will be used as an example for explanation below.

[0115] Since the time interval between the original video frames from time 1 to time t and the future video frames at time t+1 is very short within a complete video segment, and they are also very similar in content, this exemplary embodiment of the present disclosure considers that their respective content information is determined by the same latent space vector. Therefore, this exemplary embodiment of the present disclosure can represent the first scene conditional variational autoencoder as p(x t+1 |x 1:t ,s 1:t ), where x 1:t s represents multiple original video frame images from time 1 to time t. 1:t Represents a scene image, x t+1 The future video frame image at time t+1. This also applies to the generation of reconstructed optical flow images.

[0116] Given video frame images x from time 1 to time t 1:t Under these conditions, the appearance query feature vector of the original video frame image is obtained through the first memory storage unit, from which the prior distribution p(z|x) can be obtained. 1:t Subsequently, the video frame images x from time 1 to time t are... 1:t and scene images 1:t The features are concatenated and input into the first scene conditional variational autoencoder to output a new feature representation, thereby obtaining the posterior distribution q(z|x). 1:t ,s 1:tNext, z is sampled from the posterior distribution and concatenated with the conditional appearance query feature vector. This concatenation is then input into the appearance and scene-aware decoder to generate the future video frame image x. t+1 The above process applies to the optical flow image y. 1:t The same applies.

[0117] Assume p(z|x) 1:t ), q(z|x 1:t ,s 1:t ) and p(x t+1 |x 1:t ,s 1:t The parameters of the first scene conditional variational autoencoder all conform to a Gaussian distribution. The appearance and scene stitching loss of the first scene conditional variational autoencoder can be calculated using the following formula:

[0118] L 1SC-VAE =KL[q(z|x 1:t ,s 1:t )||p(z|x 1:t (10)

[0119] Among them, L 1SC-VAE The loss between appearance and scene stitching is represented by KL, and the divergence is represented by q(z|x). 1:t ,s 1:t Let p(z|x) denote the posterior distribution. 1:t ) represents the prior distribution. Similarly, the motion and scene stitching loss L of the second scene conditional variational autoencoder can be calculated. 2SC-VAE .

[0120] At this point, the stitching loss of the video anomaly detection model can be calculated using the following formula:

[0121] L SC-VAE =λ 1SC-VAE ·L 1SC-VAE +λ 2SC-VAE ·L 2SC-VAE (11)

[0122] Among them, L SC-VAE L represents the splicing loss. 1memory L represents the loss in appearance and scene splicing. 2SC-VAE λ represents the motion and scene stitching loss. 1SC-VAE λ represents the weight of the appearance and scene stitching loss. 2SC-VAE This represents the weight of the motion and scene stitching loss.

[0123] Based on this, the total loss of the video anomaly detection model provided in the exemplary embodiments of this disclosure can be calculated using the following formula:

[0124] L=λ pred ·Lpred +λ recon ·L recon +λ memory ·L memory +λ SC-VAE ·L SC-VAE (12)

[0125] Where L represents the total loss of the video anomaly detection model, λ pred λ represents the weights of the predicted loss. recon The weights λ represent the reconstruction loss. memory The weights representing memory loss, λ SC-VAE The weight represents the splicing loss. Other parameters are described in the previous text and will not be repeated here.

[0126] During the inference phase, an anomaly detection model is used to detect anomalies in the target object of the video under test. The output data of the video anomaly detection model consists of a future video frame image at time t+1 representing the appearance features of the target object and a reconstructed optical flow image from time 1 to time t representing the motion information of the target object. Based on this, an exemplary embodiment of this disclosure can calculate the anomaly score of the target object using the anomaly scores of the future video frame image and the reconstructed optical flow image to determine whether the video under test is abnormal.

[0127] For example, the anomaly score of a target object can be calculated using the following formula:

[0128] S = w fram-pred ·S fram-pred +w fram-recon ·S fram-recon (13)

[0129]

[0130]

[0131] Where S represents the anomaly score of the target object, S fram-pred S represents the appearance anomaly score of future video frame images. fram-recon w represents the motion anomaly score of the reconstructed optical flow image. fram-pred wf represents the fusion weighting coefficient for appearance anomaly scores. ram-recon This represents the fusion weighting coefficient for the motion abnormality score. Other parameters are described above and will not be repeated here.

[0132] Figure 4 This diagram illustrates the detection results of the video anomaly detection model provided in an exemplary embodiment of this disclosure during the testing phase. Figure 4As shown, the video anomaly detection model was tested using a test dataset. It can be seen that when a bicycle enters a pedestrian area in the video, the bicycle's anomaly score increases significantly, showing a large difference compared to the anomaly scores generated by normal data samples. The gray shaded area in the figure represents the true anomaly labels of the test dataset. Ultimately, the accuracy of the video anomaly detection method proposed in this exemplary embodiment of the present disclosure in the field of video anomaly detection is demonstrated.

[0133] In the video anomaly detection model proposed in this exemplary embodiment, the appearance anomaly detection network is essentially a prediction network, and the motion anomaly detection network is essentially a reconstruction network. The appearance anomaly detection network and the motion anomaly detection network are integrated to form a parallel dual-branch anomaly detection module with a hybrid structure. This parallel dual-branch anomaly detection module establishes connections between foreground video frame images, optical flow images, and scene images, learning multimodal feature information to achieve comprehensive detection of abnormal human behavior in video surveillance. Through a uniquely designed network model, the accuracy and robustness of anomaly detection are further improved.

[0134] Based on this, an exemplary embodiment of this disclosure provides a video anomaly detection method, which can be executed by a terminal or by a chip applied to the terminal. The term "terminal" can be found above and will not be repeated here.

[0135] Figure 5 A flowchart illustrating an exemplary embodiment of the video anomaly detection method provided in this disclosure is shown. Figure 5 As shown, the video anomaly detection method includes:

[0136] S501, acquire the test video of the target object, and extract multiple test video frame images from the test video;

[0137] S502, Obtain a pre-built video anomaly detection model. The video anomaly detection model includes a video parsing module and a parallel dual-branch anomaly detection module. The parallel dual-branch anomaly detection module includes an appearance anomaly detection network and a motion anomaly detection network.

[0138] S503: Input multiple test video frame images into the video anomaly detection model. The video parsing module extracts the foreground video frame image, scene image and optical flow image of the target object from the multiple test video frame images respectively. The appearance anomaly detection network obtains the future video frame image of the target object based on the foreground video frame image and scene image. The motion anomaly detection network obtains the reconstructed optical flow image of the target object based on the optical flow image and scene image.

[0139] S504 uses future video frame images and reconstructed optical flow images to determine the detection result of the target object.

[0140] Specifically, the video to be tested may include a target object. In an exemplary embodiment of this disclosure, multiple frames of video to be tested can be extracted from the video to be tested, and then anomaly detection of the target object can be performed using the multiple frames of video to be tested.

[0141] As mentioned above, anomalies of a target object can include appearance anomalies, motion anomalies, and scene mismatch anomalies. Scene mismatch anomalies can include appearance mismatch with the scene and motion mismatch with the scene. In related technologies, only appearance anomalies and motion anomalies are detected in the target object in the video under test, without considering scene mismatch anomalies. Based on this, an exemplary embodiment of this disclosure pre-constructs a video anomaly detection model. This model can include a video parsing module and a parallel dual-branch anomaly detection module. The parallel dual-branch anomaly detection module can include an appearance anomaly detection network and a motion anomaly detection network.

[0142] The video anomaly detection model inputs multiple test video frames. First, the video parsing module extracts the foreground video frame image, scene image, and optical flow image of the target object from the multiple test video frames. Then, the appearance anomaly detection network obtains the future video frame image of the target object based on the foreground video frame image and scene image, and the motion anomaly detection network obtains the reconstructed optical flow image of the target object based on the optical flow image and scene image. Finally, the detection result of the target object is determined using the future video frame image and the reconstructed optical flow image.

[0143] As can be seen, the future video frame image integrates appearance information from the foreground video frame image and scene information from the scene image, establishing a connection between the two. Similarly, the reconstructed optical flow image integrates motion information from the optical flow image and scene information from the scene image, also establishing a connection between them. Therefore, by utilizing the future video frame image and the reconstructed optical flow image, the detection result of the target object can be determined. This detection result comprehensively considers the target object's appearance, motion, and scene information, improving the accuracy of the detection.

[0144] The exemplary embodiment of this disclosure comprises a parallel dual-branch anomaly detection module consisting of an appearance anomaly detection network and a motion anomaly detection network, which can simultaneously learn the multimodal features of the target object and improve the detection efficiency of the video anomaly detection model.

[0145] According to the technical solution of the example embodiment of this disclosure, a test video of a target object is acquired, and multiple test video frame images are extracted from the test video; a pre-constructed video anomaly detection model is acquired, the video anomaly detection model including a video parsing module and a parallel dual-branch anomaly detection module, the parallel dual-branch anomaly detection module including an appearance anomaly detection network and a motion anomaly detection network; multiple test video frame images are input into the video anomaly detection model, the foreground video frame image, scene image and optical flow image of the target object are extracted from the multiple test video frame images by the video parsing module, the future video frame image of the target object is obtained by the appearance anomaly detection network based on the foreground video frame image and scene image, and the reconstructed optical flow image of the target object is obtained by the motion anomaly detection network based on the optical flow image and scene image; the detection result of the target object is determined by using the future video frame image and the reconstructed optical flow image, which can comprehensively consider the appearance information, motion information and scene information of the target object when performing anomaly detection on the target object in the test video, thereby improving the accuracy of the detection result.

[0146] In some embodiments, the appearance anomaly detection network includes an appearance and scene-aware encoder, a first memory storage unit, a first scene-conditional variational autoencoder, and an appearance and scene-aware decoder.

[0147] The motion anomaly detection network includes a motion and scene-aware encoder, a second memory storage unit, a second scene-conditional variational autoencoder, and a motion and scene-aware decoder.

[0148] The appearance and scene-aware encoder and the motion anomaly detection network have the same structure, the first memory storage unit and the second memory storage unit have the same structure, the first scene-conditional variational autoencoder and the first scene-conditional variational autoencoder have the same structure, and the appearance and scene-aware decoder and the motion and scene-aware decoder have the same structure.

[0149] Specifically, as can be seen from the preceding text, the appearance anomaly detection network may include an appearance and scene-aware encoder, a first memory storage unit, a first scene-conditional variational autoencoder, and an appearance and scene-aware decoder; the motion anomaly detection network may include a motion and scene-aware encoder, a second memory storage unit, a second scene-conditional variational autoencoder, and a motion and scene-aware decoder.

[0150] The exemplary embodiments disclosed herein can be configured such that the appearance and scene-aware encoder and the motion anomaly detection network have the same structure, the first memory storage unit and the second memory storage unit have the same structure, the first scene-conditional variational autoencoder and the first scene-conditional variational autoencoder have the same structure, and the appearance and scene-aware decoder and the motion and scene-aware decoder have the same structure. This ensures the consistency of the overall structure of the parallel dual-branch anomaly detection module composed of the appearance anomaly detection network and the motion anomaly detection network, and improves the stability of the finally constructed video anomaly detection model.

[0151] In some embodiments, obtaining future video frame images of the target object based on foreground video frame images and scene images using an appearance anomaly detection network may include:

[0152] The foreground video frame image is feature-encoded by an appearance and scene-aware encoder to obtain the appearance feature vector of the target object; the appearance query feature vector corresponding to the appearance feature vector is obtained through the first memory storage unit; the appearance query feature vector and the scene image are subjected to feature concatenation and feature extraction operations by a first scene-conditional variational autoencoder to obtain the target appearance and scene concatenation feature vector of the fused scene image; the target appearance and scene concatenation feature vector is decompressed by an appearance and scene-aware decoder to obtain the future video frame image of the target object.

[0153] Specifically, the foreground video frame image is input into the appearance anomaly detection network. The appearance and scene perception encoder can be used to encode the features of the foreground video frame image, learn the appearance features of the target object in the foreground video frame image, obtain the appearance feature vector of the target object, and input the appearance feature vector of the target object into the first memory storage unit.

[0154] The first memory storage unit can be used to retrieve memory memory through memory addressing vector, use multiple memory items to linearly combine to obtain the appearance query feature vector corresponding to the appearance feature vector, and input the appearance query feature vector into the first scene conditional variational autoencoder.

[0155] The first scene conditional variational autoencoder can be used to perform feature concatenation and feature extraction operations on appearance query feature vectors and scene images to obtain the target appearance and scene concatenation feature vector of the fused scene image, and input the target appearance and scene concatenation feature vector to the appearance and scene perception decoder.

[0156] The appearance and scene-aware decoder can be used to decompress the feature vector of the target's appearance and scene to obtain future video frame images of the target object.

[0157] Based on this, an exemplary embodiment of the present disclosure can set a first scene conditional variational autoencoder in the appearance anomaly detection network, using scene images as guiding conditions to flexibly control the generation process of future video frame images, so that the generation results of future video frame images have diversity and controllability, and realize full-range detection of appearance anomalies and appearance anomalies and scene mismatch anomalies.

[0158] In some embodiments, the first scene-conditional variational autoencoder includes a first encoder, a first latent spatial sampling layer, and a first decoder;

[0159] The first scene conditional variational autoencoder performs feature concatenation and feature extraction operations on the appearance query feature vector and the scene image to obtain the target appearance and scene concatenation feature vector of the fused scene image, which may include:

[0160] The first encoder performs feature extraction and compression operations on the appearance query feature vector and the first appearance and scene splicing feature vector of the scene image to obtain the appearance compressed feature vector; the first latent space sampling layer extracts the appearance and scene latent space feature vector from the appearance compressed feature vector; the first decoder performs decompression operations on the appearance and scene latent space feature vector and the second appearance and scene splicing feature vector of the scene image to obtain the target appearance and scene splicing feature vector of the target object.

[0161] Specifically, as can be seen from the preceding text, the first scene conditional variational autoencoder is essentially a generative model with a deep latent space, which may include a first encoder, a first latent space sampling layer, and a first decoder.

[0162] First, in an exemplary embodiment of this disclosure, the appearance query feature vector of the target object and the scene image are concatenated to obtain a first appearance and scene concatenated feature vector. Then, a first encoder can perform feature extraction and compression operations on the first appearance and scene concatenated feature vector to obtain an appearance compressed feature vector. The first latent space sampling layer is a representation of compressed data, and its function is to learn data features and simplify data representation in order to find a certain pattern. It can be used to extract the appearance and scene latent space feature vector from the appearance compressed feature vector. The appearance and scene latent space feature vector and the scene image are concatenated to obtain a second appearance and scene concatenated feature vector. The second appearance and scene concatenated feature vector is decompressed by a first decoder to obtain the target appearance and scene concatenated feature vector of the target object. This target appearance and scene concatenated feature vector can be a data sample similar to the appearance query feature vector of the target object.

[0163] As can be seen, the first scene conditional variational autoencoder includes two stitching processes of scene images when extracting deep feature information. This allows the first scene conditional variational autoencoder to obtain the target appearance and scene stitching feature vector of the fused scene image, thereby avoiding the loss of scene feature information during the deep feature information extraction process and improving the accuracy of video anomaly detection.

[0164] In some embodiments, a reconstructed optical flow image of the target object is obtained based on the optical flow image and the scene image using a motion anomaly detection network, including:

[0165] The motion feature vector of the target object is obtained by feature encoding of the optical flow image through a motion and scene-aware encoder; the motion query feature vector corresponding to the motion feature vector is obtained through a second memory storage unit; the motion query feature vector and the scene image are subjected to feature stitching and feature extraction operations by a second scene-conditional variational autoencoder to obtain the target motion and scene stitched feature vector of the fused scene image; the target motion and scene stitched feature vector is decompressed by a motion and scene-aware decoder to obtain the reconstructed optical flow image of the target object.

[0166] Specifically, the optical flow image is input into the motion anomaly detection network, and the motion and scene perception encoder performs feature encoding on the optical flow image to learn the motion features of the target object in the optical flow image, obtain the motion feature vector of the target object, and input the motion feature vector of the target object into the second memory storage unit.

[0167] The second memory storage unit retrieves the memory memory through the memory addressing vector, uses multiple memory items to perform linear combination, obtains the motion query feature vector corresponding to the motion feature vector, and inputs the motion query feature vector into the second scene conditional variational autoencoder.

[0168] The second scene conditional variational autoencoder performs feature concatenation and feature extraction operations on the motion query feature vector and the scene image to obtain the target motion and scene concatenation feature vector of the fused scene image, and then inputs the target motion and scene concatenation feature vector into the motion and scene perception decoder.

[0169] The motion and scene-aware decoder decompresses the feature vectors of target motion and scene stitching to obtain a reconstructed optical flow image.

[0170] Based on this, an exemplary embodiment of this disclosure can set a second scene-conditional variational autoencoder in the motion anomaly detection network, using the scene image as a guiding condition to flexibly control the generation process of the reconstructed optical flow image, so that the generation result of the reconstructed optical flow image has diversity and controllability, and realizes full-range detection of motion anomalies and motion anomalies and scene mismatch anomalies.

[0171] In some embodiments, the second scene-conditional variational autoencoder includes a second encoder, a second latent space sampling layer, and a second decoder;

[0172] The motion query feature vector and scene image are concatenated and extracted using a second scene conditional variational autoencoder to obtain a fused scene image feature vector of target motion and scene concatenation, including:

[0173] The second encoder performs feature extraction and compression operations on the motion query feature vector and the first motion and scene splicing feature vector of the scene image to obtain the motion compressed feature vector; the second latent space sampling layer extracts the motion and scene latent space feature vector from the motion compressed feature vector; the second decoder performs decompression operations on the motion and scene latent space feature vector and the second motion and scene splicing feature vector of the scene image to obtain the target motion and scene splicing feature vector of the target object.

[0174] Specifically, as mentioned above, the second scene conditional variational autoencoder is essentially a generative model with a deep latent space, which may include a second encoder, a second latent space sampling layer, and a second decoder.

[0175] First, in this exemplary embodiment, the motion query feature vector of the target object and the scene image are concatenated to obtain a first motion and scene concatenated feature vector. Then, a second encoder performs feature extraction and compression operations on the first motion and scene concatenated feature vector to obtain a motion compressed feature vector. The second latent space sampling layer is a representation of compressed data, and its function is to learn data features and simplify data representation in order to find a certain pattern. It can be used to extract motion and scene latent space feature vectors from the motion compressed feature vector. The motion and scene latent space feature vectors and the scene image are concatenated to obtain a second motion and scene concatenated feature vector. The second decoder performs decompression operations on the second motion and scene concatenated feature vector to obtain the target motion and scene concatenated feature vector of the target object. This target motion and scene concatenated feature vector can be a data sample similar to the motion query feature vector of the target object.

[0176] As can be seen, the second scene conditional variational autoencoder includes two stitching processes of scene images when extracting deep feature information. This allows the second scene conditional variational autoencoder to obtain the target motion and scene stitching feature vectors of the fused scene images, thereby avoiding the loss of scene feature information during the deep feature information extraction process and improving the accuracy of video anomaly detection.

[0177] In some embodiments, the appearance and scene-aware encoder and the motion anomaly detection network are both structured as multi-layer residual U-Net networks. The multi-layer residual U-Net network includes multiple residual blocks, and each residual block includes multiple convolutional layers with convolutional kernels of different sizes that perform residual convolution operations.

[0178] Specifically, both the appearance and scene-aware encoder and the motion anomaly detection network adopt the multilayer residual U-Net network proposed in the exemplary embodiments of this disclosure. Details regarding the multilayer residual U-Net network can be found above and will not be repeated here.

[0179] The multi-layer residual U-Net network in the exemplary embodiments of this disclosure may include multiple residual blocks, and each residual block may include multiple convolutional layers with convolutional kernels of different sizes performing residual convolution operations.

[0180] Compared to the traditional U-Net network, the exemplary embodiments of this disclosure can replace the convolution operation performed by the traditional U-Net network using multiple convolution kernels of the same size with the residual convolution operation performed by multiple convolution kernels of different sizes, which can further improve the computational efficiency and accuracy of the video anomaly detection model.

[0181] In some embodiments, determining the detection result of the target object using future video frame images and reconstructed optical flow images may include:

[0182] Determine the appearance anomaly score for the future video frame image and the motion anomaly score for the reconstructed optical flow image, respectively.

[0183] Calculate the anomaly score of the target object based on the appearance anomaly score and the motion anomaly score;

[0184] The detection result of the target object is determined by using the anomaly score of the target object.

[0185] Specifically, after obtaining the future video frame image and the reconstructed optical flow image, the appearance anomaly score corresponding to the future video frame image and the motion anomaly score corresponding to the reconstructed optical flow image can be determined respectively. The formula for calculating the appearance anomaly score corresponding to the future video frame image can be the formula (14) mentioned above, and the formula for calculating the motion anomaly score corresponding to the reconstructed optical flow image can be the formula (15) mentioned above.

[0186] Then, using formula (13) from the previous text, the abnormality score of the target object is calculated based on the appearance abnormality score and the motion abnormality score. The detection result of the target object is then determined using the abnormality score of the target object.

[0187] For example, a preset anomaly score can be set to determine whether a target object is abnormal. If the target object's anomaly score is greater than or equal to the preset anomaly score, the detection result of the target object is determined to be abnormal; if the target object's anomaly score is less than the preset anomaly score, the detection result of the target object is determined to be normal.

[0188] Here, the preset abnormal score can be determined according to the actual application scenario, and the exemplary embodiments disclosed herein do not impose specific limitations on it.

[0189] Based on this, the example embodiments of this disclosure can score the anomalies of future video frame images and reconstructed optical flow images of the fused scene image, and determine the detection result of the target object based on the scoring result.

[0190] In some embodiments, the video parsing module includes an object detection network, a scene extraction network, and an optical flow estimation network;

[0191] The video parsing module extracts the foreground video frame image, scene image, and optical flow image of the target object from multiple test video frames, including:

[0192] The appearance information of the target object in multiple test video frames is extracted by an object detection network to obtain a foreground video frame image; the scene information of the target object in multiple test video frames is extracted by a scene extraction network to obtain a scene image; and the motion information of the target object in multiple test video frames is extracted by an optical flow estimation network to obtain an optical flow image.

[0193] Specifically, the video parsing module may include an object detection network, a scene extraction network, and an optical flow estimation network. An exemplary embodiment of this disclosure can use the object detection network to extract the appearance information of the target object from multiple test video frames to obtain a foreground video frame image; use the scene extraction network to extract the scene information of the target object from multiple test video frames to obtain a scene image; and use the optical flow estimation network to extract the motion information of the target object from multiple test video frames to obtain an optical flow image.

[0194] For details on object detection networks, scene extraction networks, and optical flow estimation networks, please refer to the previous text; they will not be repeated here.

[0195] Based on this, the exemplary embodiment of this disclosure can use a video parsing module consisting of a target detection network, a scene extraction network, and an optical flow estimation network to preprocess multiple test video frames, extracting foreground video frame images, scene images, and optical flow images respectively, so that a parallel dual-branch anomaly detection module composed of an appearance anomaly detection network and a motion anomaly detection network can perform anomaly detection of the target object, thereby improving detection efficiency.

[0196] In some embodiments, during the training phase, the total loss of the video anomaly detection model includes prediction loss, reconstruction loss, memory loss, and stitching loss, and the total loss of the video anomaly detection model is calculated based on a weighted average of the prediction loss, reconstruction loss, memory loss, and stitching loss.

[0197] Specifically, during the training phase of the video anomaly detection model, the total loss of the video anomaly detection model can include prediction loss, reconstruction loss, memory loss, and stitching loss. The total loss of the video anomaly detection model is calculated based on the weighted average of prediction loss, reconstruction loss, memory loss, and stitching loss.

[0198] Among them, the total loss of the video anomaly detection model can be calculated using formula (12) above, the prediction loss can be calculated using formula (5) above, the reconstruction loss can be calculated using formula (6) above, the memory loss can be calculated using formulas (7) to (9) above, and the splicing loss can be calculated using formulas (10) to (11) above.

[0199] The above-described at least one technical solution adopted in the exemplary embodiments of this disclosure can achieve the following beneficial effects: by acquiring the test video of the target object and extracting multiple test video frame images from the test video; acquiring a pre-constructed video anomaly detection model, the video anomaly detection model including a video parsing module and a parallel dual-branch anomaly detection module, the parallel dual-branch anomaly detection module including an appearance anomaly detection network and a motion anomaly detection network; inputting multiple test video frame images into the video anomaly detection model, extracting the foreground video frame image, scene image and optical flow image of the target object from the multiple test video frame images respectively through the video parsing module, obtaining the future video frame image of the target object based on the foreground video frame image and scene image through the appearance anomaly detection network, and obtaining the reconstructed optical flow image of the target object based on the optical flow image and scene image through the motion anomaly detection network; and determining the detection result of the target object using the future video frame image and the reconstructed optical flow image, which can comprehensively consider the appearance information, motion information and scene information of the target object when performing anomaly detection on the target object in the test video, thereby improving the accuracy of the detection result.

[0200] The foregoing primarily describes the solutions provided by exemplary embodiments of this disclosure. It is understood that, in order to achieve the above functions, the electronic device includes corresponding hardware structures and / or software modules for performing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0201] The exemplary embodiments of this disclosure can divide the electronic device into functional units according to the above method examples. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into a single processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in the exemplary embodiments of this disclosure is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0202] By dividing each functional module according to its corresponding function, an exemplary embodiment of this disclosure provides a video anomaly detection device, which can be an electronic device or a chip applied to an electronic device. Figure 6 A schematic diagram of the structure of a video anomaly detection apparatus provided in an exemplary embodiment of this disclosure is shown. Figure 6 As shown, the device 600 includes:

[0203] Acquisition module 601 is used to acquire the video to be tested of the target object;

[0204] Processing module 602 is used to extract multiple frames of video images from the video under test;

[0205] The acquisition module 601 is also used to acquire a pre-built video anomaly detection model, which includes a video parsing module and a parallel dual-branch anomaly detection module. The parallel dual-branch anomaly detection module includes an appearance anomaly detection network and a motion anomaly detection network.

[0206] The prediction module 603 is used to input multiple test video frame images into the video anomaly detection model. The video parsing module extracts the foreground video frame image, scene image and optical flow image of the target object from the multiple test video frame images respectively. The appearance anomaly detection network obtains the future video frame image of the target object based on the foreground video frame image and scene image. The motion anomaly detection network obtains the reconstructed optical flow image of the target object based on the optical flow image and scene image.

[0207] Processing module 602 is also used to determine the detection result of the target object by utilizing future video frame images and reconstructing optical flow images.

[0208] In some embodiments, the appearance anomaly detection network includes an appearance and scene-aware encoder, a first memory storage unit, a first scene-conditional variational autoencoder, and an appearance and scene-aware decoder.

[0209] The motion anomaly detection network includes a motion and scene-aware encoder, a second memory storage unit, a second scene-conditional variational autoencoder, and a motion and scene-aware decoder.

[0210] The appearance and scene-aware encoder and the motion anomaly detection network have the same structure, the first memory storage unit and the second memory storage unit have the same structure, the first scene-conditional variational autoencoder and the first scene-conditional variational autoencoder have the same structure, and the appearance and scene-aware decoder and the motion and scene-aware decoder have the same structure.

[0211] In some embodiments, the prediction module 603 is further configured to: encode the foreground video frame image using an appearance and scene-aware encoder to obtain the appearance feature vector of the target object; obtain the appearance query feature vector corresponding to the appearance feature vector using a first memory storage unit; perform feature concatenation and feature extraction operations on the appearance query feature vector and the scene image using a first scene-conditional variational autoencoder to obtain the target appearance and scene concatenation feature vector of the fused scene image; and decompress the target appearance and scene concatenation feature vector using an appearance and scene-aware decoder to obtain the future video frame image of the target object.

[0212] In some embodiments, the first scene-conditional variational autoencoder includes a first encoder, a first latent spatial sampling layer, and a first decoder;

[0213] The prediction module 603 is further configured to perform feature extraction and compression operations on the appearance query feature vector and the first appearance and scene splicing feature vector of the scene image through the first encoder to obtain the appearance compressed feature vector; extract the appearance and scene latent space feature vector from the appearance compressed feature vector through the first latent space sampling layer; and decompress the appearance and scene latent space feature vector and the second appearance and scene splicing feature vector of the scene image through the first decoder to obtain the target appearance and scene splicing feature vector of the target object.

[0214] In some embodiments, the prediction module 603 is further configured to encode the optical flow image using a motion and scene-aware encoder to obtain a motion feature vector of the target object; obtain a motion query feature vector corresponding to the motion feature vector using a second memory storage unit; perform feature concatenation and feature extraction operations on the motion query feature vector and the scene image using a second scene-conditional variational autoencoder to obtain a target motion and scene concatenation feature vector of the fused scene image; and perform a decompression operation on the target motion and scene concatenation feature vector using a motion and scene-aware decoder to obtain a reconstructed optical flow image of the target object.

[0215] In some embodiments, the second scene-conditional variational autoencoder includes a second encoder, a second latent space sampling layer, and a second decoder;

[0216] The prediction module 603 is further configured to perform feature extraction and compression operations on the motion query feature vector and the first motion and scene concatenation feature vector of the scene image through the second encoder to obtain the motion compressed feature vector; extract the motion and scene latent space feature vector from the motion compressed feature vector through the second latent space sampling layer; and decompress the motion and scene latent space feature vector and the second motion and scene concatenation feature vector of the scene image through the second decoder to obtain the target motion and scene concatenation feature vector of the target object.

[0217] In some embodiments, the appearance and scene-aware encoder and the motion anomaly detection network are both structured as multi-layer residual U-Net networks. The multi-layer residual U-Net network includes multiple residual blocks, and each residual block includes multiple convolutional layers with convolutional kernels of different sizes that perform residual convolution operations.

[0218] In some embodiments, the processing module 602 is further configured to determine the appearance anomaly score corresponding to the future video frame image and the motion anomaly score corresponding to the reconstructed optical flow image, respectively; calculate the anomaly score of the target object based on the appearance anomaly score and the motion anomaly score; and determine the detection result of the target object using the anomaly score of the target object.

[0219] In some embodiments, the video parsing module includes an object detection network, a scene extraction network, and an optical flow estimation network;

[0220] The prediction module 603 is also used to extract the appearance information of the target object in multiple test video frame images through the target detection network to obtain the foreground video frame image; to extract the scene information of the target object in multiple test video frame images through the scene extraction network to obtain the scene image; and to extract the motion information of the target object in multiple test video frame images through the optical flow estimation network to obtain the optical flow image.

[0221] In some embodiments, during the training phase, the total loss of the video anomaly detection model includes prediction loss, reconstruction loss, memory loss, and stitching loss, and the total loss of the video anomaly detection model is calculated based on a weighted average of the prediction loss, reconstruction loss, memory loss, and stitching loss.

[0222] An exemplary embodiment of this disclosure also provides an electronic device, including: at least one processor; a memory for storing at least one processor-executable instruction; wherein the at least one processor is configured to execute the instruction to implement the steps of the method disclosed in the exemplary embodiment of this disclosure.

[0223] Figure 7 A schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this disclosure is shown. For example... Figure 7 As shown, the electronic device 700 includes at least one processor 701 and a memory 702 coupled to the processor 701, which can perform the corresponding steps in the methods disclosed in the exemplary embodiments of this disclosure.

[0224] The processor 701 described above can also be referred to as a Central Processing Unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in the exemplary embodiments of this disclosure can be implemented by the integrated logic circuitry in the hardware of the processor 701 or by instructions in software form. The processor 701 described above can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the exemplary embodiments of this disclosure can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in the memory 702, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor 701 reads information from the memory 702 and, in conjunction with its hardware, completes the steps of the above method.

[0225] Furthermore, various operations / processes according to this disclosure, implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, for example, Figure 8The computer system 800 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including those described above. Figure 8 A schematic diagram of the structure of a computer system provided in an exemplary embodiment of this disclosure is shown.

[0226] Computer system 800 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0227] like Figure 8 As shown, the computer system 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the computer system 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0228] Multiple components in the computer system 800 are connected to the I / O interface 805, including: an input unit 806, an output unit 807, a storage unit 808, and a communication unit 809. The input unit 806 can be any type of device capable of inputting information into the computer system 800. The input unit 806 can receive input numerical or character information and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 807 can be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. The storage unit 808 may include, but is not limited to, a hard disk and an optical disk. The communication unit 809 allows the computer system 800 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, a modem, network card, infrared communication device, wireless communication transceiver, and / or chipset, such as Bluetooth™ device, WiFi device, WiMax device, cellular communication device, and / or the like.

[0229] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above. For example, in some embodiments, the methods disclosed in the exemplary embodiments of this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 802 and / or communication unit 809. In some embodiments, the computing unit 801 can be configured by any other suitable means (e.g., by means of firmware) to perform the methods disclosed in the exemplary embodiments of this disclosure.

[0230] This exemplary embodiment also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the methods disclosed in this exemplary embodiment.

[0231] The computer-readable storage medium in the exemplary embodiments of this disclosure may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The aforementioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specifically, the aforementioned computer-readable storage medium may include electrical connections based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0232] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0233] This disclosure also provides an exemplary embodiment of a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements the methods disclosed in the exemplary embodiments of this disclosure.

[0234] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer.

[0235] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0236] The modules, components, or units described in the exemplary embodiments of this disclosure may be implemented in software or hardware. The names of the modules, components, or units do not, in some cases, constitute a limitation on the module, component, or unit itself.

[0237] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0238] The above description is merely an embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0239] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A video anomaly detection method, characterized in that, include: Acquire the test video of the target object, and extract multiple test video frame images from the test video; Obtain a pre-built video anomaly detection model, which includes a video parsing module and a parallel dual-branch anomaly detection module, wherein the parallel dual-branch anomaly detection module includes an appearance anomaly detection network and a motion anomaly detection network; The multi-frame video images to be tested are input into the video anomaly detection model. The video parsing module extracts the foreground video frame image, scene image, and optical flow image of the target object from the multi-frame video images to be tested. The appearance anomaly detection network obtains the future video frame image of the target object based on the foreground video frame image and the scene image. The motion anomaly detection network obtains the reconstructed optical flow image of the target object based on the optical flow image and the scene image. The detection result of the target object is determined using the future video frame image and the reconstructed optical flow image; The appearance anomaly detection network includes an appearance and scene-aware encoder, a first memory storage unit, a first scene-conditional variational autoencoder, and an appearance and scene-aware decoder; obtaining the future video frame image of the target object based on the foreground video frame image and the scene image through the appearance anomaly detection network includes: The foreground video frame image is feature-encoded by the appearance and scene-aware encoder to obtain the appearance feature vector of the target object; the appearance query feature vector corresponding to the appearance feature vector is obtained through the first memory storage unit; the appearance query feature vector and the scene image are subjected to feature concatenation and feature extraction operations by the first scene conditional variational autoencoder to obtain the target appearance and scene concatenation feature vector fused with the scene image; the target appearance and scene concatenation feature vector is decompressed by the appearance and scene-aware decoder to obtain the future video frame image of the target object. The motion anomaly detection network includes a motion and scene-aware encoder, a second memory storage unit, a second scene-conditional variational autoencoder, and a motion and scene-aware decoder; obtaining the reconstructed optical flow image of the target object based on the optical flow image and the scene image through the motion anomaly detection network includes: The motion and scene-aware encoder performs feature encoding on the optical flow image to obtain the motion feature vector of the target object; the second memory storage unit obtains the motion query feature vector corresponding to the motion feature vector; the second scene-conditional variational autoencoder performs feature concatenation and feature extraction operations on the motion query feature vector and the scene image to obtain the target motion and scene concatenation feature vector fused with the scene image; the motion and scene-aware decoder decompresses the target motion and scene concatenation feature vector to obtain the reconstructed optical flow image of the target object.

2. The method according to claim 1, characterized in that, The appearance and scene-aware encoder and the motion anomaly detection network have the same structure, the first memory storage unit and the second memory storage unit have the same structure, the first scene-conditional variational autoencoder and the first scene-conditional variational autoencoder have the same structure, and the appearance and scene-aware decoder and the motion and scene-aware decoder have the same structure.

3. The method according to claim 1, characterized in that, The first scene-conditional variational autoencoder includes a first encoder, a first latent space sampling layer, and a first decoder; The step of performing feature concatenation and feature extraction operations on the appearance query feature vector and the scene image through the first scene conditional variational autoencoder to obtain a fused target appearance and scene concatenation feature vector of the scene image includes: The first encoder performs feature extraction and compression operations on the appearance query feature vector and the first appearance and scene splicing feature vector of the scene image to obtain the appearance compressed feature vector; the first latent space sampling layer extracts the appearance and scene latent space feature vector from the appearance compressed feature vector; the first decoder performs decompression operations on the appearance and scene latent space feature vector and the second appearance and scene splicing feature vector of the scene image to obtain the target appearance and scene splicing feature vector of the target object.

4. The method according to claim 1, characterized in that, The second scene-conditional variational autoencoder includes a second encoder, a second latent space sampling layer, and a second decoder; The step of performing feature concatenation and feature extraction operations on the motion query feature vector and the scene image through the second scene conditional variational autoencoder to obtain a target motion and scene concatenation feature vector fused with the scene image includes: The second encoder performs feature extraction and compression operations on the motion query feature vector and the first motion and scene stitching feature vector of the scene image to obtain a motion compressed feature vector; the second latent space sampling layer extracts the motion and scene latent space feature vector from the motion compressed feature vector; the second decoder performs decompression operations on the motion and scene latent space feature vector and the second motion and scene stitching feature vector of the scene image to obtain the target motion and scene stitching feature vector of the target object.

5. The method according to claim 1, characterized in that, The appearance and scene perception encoder and the motion anomaly detection network are both structured as multi-layer residual U-Net networks. The multi-layer residual U-Net network includes multiple residual blocks, and each residual block includes multiple convolutional layers with convolutional kernels of different sizes performing residual convolution operations.

6. The method according to claim 1, characterized in that, The step of determining the detection result of the target object using the future video frame image and the reconstructed optical flow image includes: Determine the appearance anomaly score corresponding to the future video frame image and the motion anomaly score corresponding to the reconstructed optical flow image, respectively. Based on the appearance anomaly score and the motion anomaly score, calculate the anomaly score of the target object; The detection result of the target object is determined using the anomaly score of the target object.

7. The method according to claim 1, characterized in that, The video parsing module includes an object detection network, a scene extraction network, and an optical flow estimation network; The step of extracting the foreground video frame image, scene image, and optical flow image of the target object from the multiple test video frame images through the video parsing module includes: The appearance information of the target object in the multiple test video frames is extracted by the target detection network to obtain the foreground video frame image; the scene information of the target object in the multiple test video frames is extracted by the scene extraction network to obtain the scene image; and the motion information of the target object in the multiple test video frames is extracted by the optical flow estimation network to obtain the optical flow image.

8. The method according to any one of claims 1 to 7, characterized in that, During the training phase, the total loss of the video anomaly detection model includes prediction loss, reconstruction loss, memory loss, and splicing loss, and the total loss of the video anomaly detection model is calculated based on the weighted average of the prediction loss, reconstruction loss, memory loss, and splicing loss.

9. A video anomaly detection device, characterized in that, include: The acquisition module is used to acquire the video to be tested of the target object; The processing module is used to extract multiple frames of video images from the video under test; The acquisition module is also used to acquire a pre-built video anomaly detection model, which includes a video parsing module and a parallel dual-branch anomaly detection module. The parallel dual-branch anomaly detection module includes an appearance anomaly detection network and a motion anomaly detection network. The prediction module is used to input the multiple test video frame images into the video anomaly detection model, and the video parsing module extracts the foreground video frame image, scene image and optical flow image of the target object from the multiple test video frame images respectively. The appearance anomaly detection network obtains the future video frame image of the target object based on the foreground video frame image and the scene image, and the motion anomaly detection network obtains the reconstructed optical flow image of the target object based on the optical flow image and the scene image. The processing module is also used to determine the detection result of the target object using the future video frame image and the reconstructed optical flow image; The appearance anomaly detection network includes an appearance and scene-aware encoder, a first memory storage unit, a first scene-conditional variational autoencoder, and an appearance and scene-aware decoder. The prediction module is further configured to: encode the foreground video frame image using the appearance and scene-aware encoder to obtain the appearance feature vector of the target object; obtain the appearance query feature vector corresponding to the appearance feature vector using the first memory storage unit; perform feature concatenation and feature extraction operations on the appearance query feature vector and the scene image using the first scene-conditional variational autoencoder to obtain a target appearance and scene concatenation feature vector fused with the scene image; and decompress the target appearance and scene concatenation feature vector using the appearance and scene-aware decoder to obtain a future video frame image of the target object. The motion anomaly detection network includes a motion and scene-aware encoder, a second memory storage unit, a second scene-conditional variational autoencoder, and a motion and scene-aware decoder. The prediction module is further configured to encode the optical flow image using the motion and scene-aware encoder to obtain a motion feature vector of the target object; obtain a motion query feature vector corresponding to the motion feature vector using the second memory storage unit; perform feature concatenation and feature extraction operations on the motion query feature vector and the scene image using the second scene-conditional variational autoencoder to obtain a target motion and scene concatenation feature vector fused with the scene image; and decompress the target motion and scene concatenation feature vector using the motion and scene-aware decoder to obtain a reconstructed optical flow image of the target object.

10. An electronic device, characterized in that, include: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the steps of the method as described in any one of claims 1 to 8.

11. A computer program product, characterized in that, It includes a computer program, wherein when the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 8.

Images