Video anomaly detection method based on cross-scale multi-class complementary feature integration

Abstract

The invention belongs to the technical field of intelligent video monitoring, and particularly discloses a video anomaly detection method based on cross-scale multi-class complementary feature integration, and the method comprises the steps: extracting a frame-level general semantic feature, a target-level quantitative motion feature and a frame-level special continuity feature of a to-be-detected video; estimating normal feature distribution densities respectively corresponding to the frame-level general semantic feature, the target-level quantitative motion feature and the frame-level special continuity feature; and obtaining an abnormal result of the to-be-detected video based on the normal feature distribution density. The method can effectively improve the video anomaly detection precision.

Description

Technical Field

[0001] This application belongs to the field of intelligent video surveillance technology, and more specifically, relates to a video anomaly detection method based on cross-scale multi-class complementary feature integration. Background Technology

[0002] With the large-scale deployment of surveillance cameras, video anomaly detection has become one of the key technologies in intelligent monitoring systems. Its core task is to automatically identify events from video streams that do not conform to normal behavioral patterns, such as fighting, chasing, illegal parking, and driving against traffic. Due to the scarcity, diversity, and unpredictability of abnormal events, current mainstream methods generally adopt a single-class classification setting, that is, training only with normal samples.

[0003] Existing video anomaly detection methods can be broadly categorized into two types: generative paradigms and representation paradigms. Generative paradigm methods (such as autoencoders, generative adversarial networks, and diffusion models) reconstruct or predict video frames and use pixel-level errors as anomaly criteria. While intuitive, they are susceptible to interference from lighting conditions and jitter, and are insensitive to high-level semantic anomalies. Representation paradigm methods, on the other hand, directly estimate the density of normal samples in the feature space, making them more suitable for semantic-level anomaly detection and a current research hotspot.

[0004] However, existing representation paradigms still have significant limitations. Most methods rely on only a single type of feature or use simple feature splicing strategies, resulting in insufficient information complementarity and low anomaly detection accuracy. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this application aims to provide a video anomaly detection method based on cross-scale multi-class complementary feature integration, which addresses the problem of low anomaly detection accuracy caused by insufficient information complementarity due to existing technologies relying on single-type features or adopting simple feature splicing strategies.

[0006] To achieve the above objectives, in a first aspect, this application provides a video anomaly detection method based on cross-scale multi-class complementary feature integration, comprising: Extract frame-level general semantic features, target-level quantized motion features, and frame-level specific continuity features from the video to be detected; Estimate the normal feature distribution density corresponding to the frame-level general semantic features, the target-level quantized motion features, and the frame-level specific continuity features, respectively; Based on the normal feature distribution density, the abnormal results of the video to be detected are obtained.

[0007] This application extracts frame-level general semantic features, target-level quantized motion features, and frame-level specific continuity features from the video to be detected. Among them, the frame-level general semantic features can capture long-range spatiotemporal dependencies and global context information, and can be used to detect abnormal behaviors that do not conform to the semantics of the scene. The target-level quantized motion features can be used to detect deviations at the motion pattern level. The frame-level specific continuity features can capture subtle and logical inter-frame transition anomalies. The three features effectively complement each other in terms of semantic level, spatial scale, and temporal dynamics of the video, avoiding information redundancy or omission, and realize video anomaly detection based on the normal feature distribution density of each feature, effectively improving the accuracy of video anomaly detection.

[0008] According to the video anomaly detection method based on cross-scale multi-class complementary feature integration provided in this application, the extraction process of the frame-level general semantic features includes: The video to be tested is divided into multiple video blocks according to the number of frames. The multiple video blocks are embedded to obtain feature vectors; The frame-level general semantic features are obtained by encoding the feature vectors using a pre-trained Vit encoder.

[0009] Since the pre-trained visual Transformer encoder is usually trained on large-scale video datasets, such as action recognition datasets, the features contain general semantic information, which can bring global semantic priors to video anomaly detection, enabling the detection system to have a holistic view of the video content.

[0010] According to the video anomaly detection method based on cross-scale multi-class complementary feature integration provided in this application, the extraction process of the target-level quantized motion features includes: Obtain the optical flow information of each frame of the video to be detected; For each frame of the video to be detected, perform the following steps: Obtain the target detection bounding boxes for all moving targets in the video frame; Convert the target detection box into a target region mask; The optical flow information of the video frame is cropped using the target region mask to obtain the optical flow corresponding to each moving target; The optical flow is divided into N directions, and the optical flow amplitude of all moving targets in the corresponding N directions is obtained to form an N-dimensional optical flow amplitude vector corresponding to all moving targets, which serves as the target-level quantized motion feature of the video frame, where N is a positive integer.

[0011] Since motion is the primary indicator of anomalies in surveillance videos, this application extracts the optical flow information of the main motion source and then quantifies its direction and amplitude to explicitly indicate the dominant motion dynamics in the video, enabling the detection system to quickly understand the main motion situation in the video.

[0012] According to the video anomaly detection method based on cross-scale multi-class complementary feature integration provided in this application, the extraction process of the frame-level dedicated continuity features includes: The preset training set is input into the encoder network of the 3D CNN architecture to obtain the encoding result. The preset training set includes multiple video segments. Based on the encoding results, a dedicated continuity encoder is trained. The dedicated continuity encoder includes the encoder network of the 3D CNN architecture, the continuity judgment projection network, the missing frame localization projection network, and the missing frame estimation projection network. The video to be detected is input into a trained dedicated continuous encoder to obtain continuous backbone features and projection network features. The continuous backbone features and the projection network features are spliced ​​together to obtain the frame-level dedicated continuous features.

[0013] Since the aforementioned frame-level general semantic representation and target-level quantized motion descriptor are extracted using a general pre-trained network, and surveillance videos are characterized by relatively weak semantics and subtle inter-frame changes, these two features suffer from domain adaptation issues and are prone to overlooking local anomalies between frames. In contrast, the frame-level dedicated continuity features are networks specifically designed based on the characteristics of surveillance videos, thus effectively complementing the aforementioned two types of features and improving the overall anomaly detection performance of the method.

[0014] According to the video anomaly detection method based on cross-scale multi-class complementary feature integration provided in this application, the estimation of the normal feature distribution density corresponding to the frame-level general semantic features, the target-level quantized motion features, and the frame-level specific continuity features includes: The frame-level general semantic features, the target-level quantized motion features, and the frame-level specific continuity features are estimated using a denoising estimation method to obtain the normal feature distributions of the frame-level general semantic features, the target-level quantized motion features, and the frame-level specific continuity features. Calculate the distribution density of frame-level general semantic features, target-level quantized motion features, and frame-level specific continuity features of each video frame in the normal feature distribution of the video to be detected.

[0015] According to the video anomaly detection method based on cross-scale multi-class complementary feature integration provided in this application, the step of obtaining the anomaly result of the video to be detected based on the normal feature distribution density includes: The normal feature distribution density of frame-level general semantic features, the normal feature distribution density of target-level quantized motion features, and the normal feature distribution density of frame-level specific continuous features for each video frame to be detected are fused and integrated to obtain the outlier value corresponding to each video frame. Anomaly detection is performed on each video frame based on the outlier value corresponding to each video frame, and the anomaly result of the video to be detected is obtained.

[0016] Secondly, this application provides a video anomaly detection device based on cross-scale multi-class complementary feature integration, comprising: The extraction module is used to extract frame-level general semantic features, target-level quantized motion features, and frame-level specific continuity features from the video to be detected. The estimation module is used to estimate the normal feature distribution density corresponding to the frame-level general semantic features, the target-level quantized motion features, and the frame-level specific continuity features, respectively. The detection module is used to obtain abnormal results of the video to be detected based on the normal feature distribution density.

[0017] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the video anomaly detection method based on cross-scale multi-class complementary feature integration described in the first aspect or any possible implementation thereof.

[0018] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the video anomaly detection method based on cross-scale multi-class complementary feature integration described in the first aspect or any possible implementation of the first aspect.

[0019] Fifthly, this application provides a computer program product that, when run on a processor, causes the processor to execute the video anomaly detection method based on cross-scale multi-class complementary feature integration described in the first aspect or any possible implementation of the first aspect.

[0020] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0021] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art: This application extracts frame-level general semantic features, target-level quantized motion features, and frame-level specific continuity features from the video to be detected. Among them, the frame-level general semantic features can capture long-range spatiotemporal dependencies and global context information, and can be used to detect abnormal behaviors that do not conform to the semantics of the scene. The target-level quantized motion features can be used to detect deviations at the motion pattern level. The frame-level specific continuity features can capture subtle and logical inter-frame transition anomalies. The three features effectively complement each other in terms of semantic level, spatial scale, and temporal dynamics of the video, avoiding information redundancy or omission, and realize video anomaly detection based on the normal feature distribution density of each feature, effectively improving the accuracy of video anomaly detection. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the video anomaly detection method based on cross-scale multi-class complementary feature integration provided in the embodiments of this application; Figure 2 This is a schematic diagram of the detection results of the UCSD Ped2 dataset provided in an embodiment of this application; Figure 3 This is a schematic diagram of the detection results of the CUHK Avenue dataset provided in the embodiments of this application; Figure 4 This is a schematic diagram of the detection results of the ShanghaiTech dataset provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the video anomaly detection device based on cross-scale multi-class complementary feature integration provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0025] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0026] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0027] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0028] Next, combined Figures 1-4 This paper introduces a video anomaly detection method based on cross-scale multi-class complementary feature integration provided in the embodiments of this application.

[0029] Figure 1 This is a flowchart illustrating the video anomaly detection method based on cross-scale multi-class complementary feature integration provided in this application embodiment, as shown below. Figure 1 As shown, the method includes the following steps: Step S1: Extract frame-level general semantic features, target-level quantized motion features, and frame-level specific continuity features from the video to be detected; The complexity of video anomalies means that it is difficult to achieve accurate and reliable detection using any single-dimensional feature. In order to accurately identify anomalies, a general and high-level semantic understanding of objects and scenes in the video is required first.

[0030] Alternatively, a Vision Transformer pre-trained on a large-scale video dataset can be used to extract frame-level general semantic features of the video to be detected, in order to express the global semantic information of the video segment. By utilizing the self-attention mechanism, it is possible to capture long-range spatiotemporal dependencies and global contextual information, which can be used to detect abnormal behaviors that do not conform to the semantics of the scene.

[0031] Optionally, motion is a key indicator of anomalies in monitored scenes, but the boundary between normal and abnormal motion is often subject to interference noise. By combining optical flow networks with target detection algorithms, optical flow information of the main moving objects in the image can be extracted, the optical flow direction can be quantized into several intervals, and the average optical flow amplitude in each interval can be calculated to form a low-dimensional, noise-resistant quantized motion histogram as a target-level quantized motion feature to express the main motion dynamics of the video segment. By explicitly enhancing the sensitivity to motion anomalies while maintaining robustness to local motion noise, the dominant motion statistical patterns can be aggregated and used to detect deviations at the motion pattern level.

[0032] Optionally, normal behaviors in surveillance videos are highly repetitive, have subtle changes, and relatively low semantic richness (such as continuous walking or vehicle movement). Traditional general semantic features and quantized motion descriptors often struggle to capture the subtle but crucial logical deviations in fine-grained transition patterns between frames. Therefore, this application constructs an encoder based on a 3D CNN architecture and trains it through a dedicated self-supervised agent task to enhance the modeling ability of local spatiotemporal consistency. This enables the capture of subtle, logical anomalies in inter-frame transitions, forming frame-level dedicated continuity features to express the local consistency between video segments, effectively complementing the two features mentioned above.

[0033] The three types of features cover the complementarity of global and local semantics, general and specific semantics, high-level semantics and low-dimensional motion, and frame-level and target-level features in video. Furthermore, by leveraging the complementarity of the Transformer architecture and the CNN architecture, the complex patterns of surveillance videos can be modeled in a comprehensive manner.

[0034] Step S2: Estimate the normal feature distribution density corresponding to frame-level general semantic features, target-level quantized motion features, and frame-level specific continuity features, respectively; Alternatively, the three features can be input into independent distribution density estimation models to estimate the normal feature distribution density of each class of features.

[0035] Step S3: Based on the normal feature distribution density, obtain the abnormal results of the video to be detected.

[0036] Since features that deviate from the normal distribution have a higher probability of being abnormal, frame-level anomaly detection of the video to be detected will be achieved based on the distribution of each dimension of the features of the video to be detected relative to the normal feature distribution.

[0037] The video anomaly detection method provided in this application, based on the integration of multi-scale complementary features, extracts frame-level general semantic features, target-level quantized motion features, and frame-level specific continuity features from the video to be detected. Among them, the frame-level general semantic features can capture long-range spatiotemporal dependencies and global contextual information, and can be used to detect abnormal behaviors that do not conform to the semantics of the scene. The target-level quantized motion features can be used to detect deviations at the motion pattern level. The frame-level specific continuity features can capture subtle and logical inter-frame transition anomalies. The three features effectively complement each other in terms of semantic level, spatial scale, and temporal dynamics of the video, avoiding information redundancy or omission. Based on the normal feature distribution density of each feature, video anomaly detection is achieved, effectively improving the accuracy of video anomaly detection.

[0038] In some embodiments, the process of extracting frame-level general semantic features includes: The video to be tested is divided into multiple video blocks according to the number of frames. Embed multiple video blocks to obtain feature vectors; The feature vectors are encoded by a pre-trained Vit encoder to obtain frame-level general semantic features.

[0039] Since pre-trained visual Transformer encoders are typically trained on large-scale video datasets, such as action recognition datasets, their features contain general semantic information, which can provide global semantic priors for video anomaly detection, enabling the detection system to have a holistic view of the video content.

[0040] Optionally, a video block can contain multiple frames of video footage. For example, frames 0 to 15 form video block 0, and the frame-level general semantic features obtained from video block 0 are used as the frame-level general semantic features of frame 8.

[0041] Optionally, there are no corresponding blocks for frames 0 to 7, and similarly, there are no corresponding blocks for the last few frames of the video, because the block length is large, so the total number is less than the total number of frames.

[0042] The Vit encoder is the core component of the Vision Transformer model. Based on the Transformer architecture, it is specifically designed for computer vision tasks. Its main function is to convert a sequence of image patches into a feature representation rich in semantic information. It captures the global dependencies between different regions in an image through a self-attention mechanism.

[0043] In one embodiment of this application, the video is input into a pre-trained Hiera-L model to extract frame-level general semantic representations. The specific steps are as follows: 1a. Based on the input requirements of the Heira-L model, divide the video to be detected into multiple video blocks. ; 2a. Embed the video blocks to obtain feature vectors; 3a. The feature vectors are encoded using a pre-trained Heira-L encoder, and the encoded output is the frame-level general semantic feature. .

[0044] In some embodiments, the extraction process of target-level quantized motion features includes: Obtain optical flow information for each frame of the video to be detected; For each frame of the video to be detected, perform the following steps: Obtain the target detection bounding boxes for all moving targets in the video frame; Convert the target detection bounding box into a target region mask; The optical flow information of the video frame is cropped using the target region mask to obtain the optical flow corresponding to each moving target; The optical flow is divided into N directions, and the optical flow amplitude of all moving targets in the corresponding N directions is obtained to form an N-dimensional optical flow amplitude vector corresponding to all moving targets, which serves as the target-level quantized motion feature of the video frame, where N is a positive integer.

[0045] Since motion is the primary indicator of anomalies in surveillance videos, by extracting the optical flow information of the main motion source (i.e., the target) and then quantifying its direction and amplitude, the dominant motion dynamics in the video can be explicitly indicated, enabling the detection system to quickly understand the main motion situation in the video.

[0046] Alternatively, a pre-trained optical flow network can be used to obtain frame-level optical flow information of the video to be detected. .

[0047] Alternatively, a pre-trained object detection network can be used to obtain the object detection boxes for video frame I. Convert the target detection bounding box into a target region mask. The optical flow information of a video frame is cropped using a target region mask to obtain the target-level optical flow of that frame. .

[0048] Optionally, the optical flow direction can be divided into N regions, and the optical flow amplitude of each target in these N directions within video frame I can be statistically analyzed. This quantizes the high-dimensional optical flow information into an N-dimensional vector, and the N-dimensional vector of each target is output as the target-level quantized motion descriptor. That is, target-level quantification of motion characteristics.

[0049] Optionally, N can be set according to the resolution of the video to be detected. For example, it can be set to 8 for high-resolution videos and 1 for very low-resolution videos.

[0050] In one embodiment of this application, the video to be detected is input into a pre-trained Flownet2 or Mask R-CNN model to extract target-level quantized motion descriptors, specifically including the following steps: 1b. Frame-level optical flow information of the video to be detected is obtained using a pre-trained Flownet2 network. ; 2b. Obtain the object detection box of video frame I using a pre-trained Mask R-CNN network. ; 3b. Convert the target detection bounding box into a target region mask. ; 4b. Use the target region mask to crop the optical flow information of the video frame to obtain the target-level optical flow of the frame. ; 5b. Divide the optical flow direction into multiple regions, and count the optical flow amplitude of each target in video frame I in these multiple directions, thereby quantizing the high-dimensional optical flow information into a multi-dimensional vector; 6b. Output the multi-dimensional vector for each target, which is the target-level quantized motion descriptor. .

[0051] In some embodiments, the extraction process of frame-level specific continuity features includes: Input the preset training set into the encoder network of the 3D CNN architecture to obtain the encoding result. The preset training set includes multiple video clips. Based on the encoding results, a dedicated continuity encoder is trained. The dedicated continuity encoder includes a 3D CNN architecture encoder network, a continuity judgment projection network, a missing frame localization projection network, and a missing frame estimation projection network. The video to be detected is input into a trained dedicated continuous encoder to obtain continuous backbone features and projection network features. By splicing the continuous backbone features and the projected network features, frame-level dedicated continuous features are obtained.

[0052] Since the aforementioned frame-level general semantic representation and target-level quantized motion descriptor are extracted using a general pre-trained network, and surveillance videos are characterized by relatively weak semantics and subtle inter-frame changes, these two features suffer from domain adaptation issues and are prone to overlooking local anomalies between frames. Frame-level dedicated continuity features, on the other hand, are networks specifically designed based on the characteristics of surveillance videos. Therefore, they can effectively complement the aforementioned two types of features and improve the overall anomaly detection performance of the method.

[0053] In one embodiment of this application, the video to be detected is input into the backbone network of the I3D-RGB model, and frame-level dedicated continuity features are extracted by combining a dedicated agent task. The specific steps are as follows: 1c. Input the training set video clips into the I3D-RGB backbone network for encoding to obtain the features output by the backbone network. ; 2c, will Input the continuity judgment projection network to obtain continuity judgment features, and optimize them by binary classification loss, that is, to determine whether the input video segment has missing frames; 3c, will The missing frame localization projection network is input to obtain the missing frame localization features, and these features are optimized by multi-class classification loss, which determines the relative position of the missing frame in the input video segment. 4c, will Input the missing frame estimation projection network to obtain the missing frame estimated features, and optimize it by contrastive loss, that is, to determine which video segment the features of the missing frame are more similar to; 5c. The backbone network and the projection network constitute the entire dedicated continuous encoder. During encoder training, the three projection networks are randomly initialized, and the backbone network is initialized using the pre-trained weights of I3D-RGB. During training, each layer of the backbone network is unfrozen in stages for fine-tuning. During training, the three types of loss are used to jointly optimize the entire network. 6c. Input the video to be detected into a dedicated continuous encoder to obtain continuous backbone features and projection network features; 7c. By concatenating the continuous backbone features and the continuous judgment projection network features, the frame-level dedicated continuous features are obtained. .

[0054] In some embodiments, step S2 specifically includes: The denoising estimation method is used to estimate the feature distribution of frame-level general semantic features, target-level quantized motion features and frame-level specific continuity features, and obtain the normal feature distribution of frame-level general semantic features, target-level quantized motion features and frame-level specific continuity features. Calculate the distribution density of frame-level general semantic features, target-level quantized motion features, and frame-level specific continuity features for each video frame in the normal feature distribution.

[0055] Optionally, a feature input distribution estimation network can be used. The distribution of normal features is estimated using a denoising estimation method, as shown in the following formula:

[0056] in, For distribution estimation network The parameters, : The training data samples are noise-free and follow the true distribution. , : The training data samples with added noise follow a mean of . The covariance matrix is Gaussian distribution, The standard deviation of the noise. It is an identity matrix.

[0057] Feature distribution estimations were performed on frame-level general semantic representations, target-level quantized motion descriptors, and frame-level specific continuous features to obtain normal feature distributions. , and .

[0058] Frame-level general semantic representation of the video to be detected Target-level quantization motion descriptor Frame-level dedicated continuity features Input normal distribution respectively , , The distribution density of the video frame to be detected for these three types of complementary features is obtained. , , .

[0059] In some embodiments, step S3 specifically includes: The normal feature distribution density of frame-level general semantic features, normal feature distribution density of target-level quantized motion features, and normal feature distribution density of frame-level specific continuous features of each video frame to be detected are fused and integrated to obtain the outlier value corresponding to each video frame. Anomaly detection is performed on each video frame based on the outlier value corresponding to each video frame, and the anomaly result of the video to be detected is obtained.

[0060] Optionally, the distribution densities of the three complementary features are fused and integrated to obtain the anomaly score, calculated as follows:

[0061] in, This is the density fusion function.

[0062] Optionally, the input to the anomaly detection process is a video segment, and the obtained anomaly score is used as the anomaly score of the center frame of the segment. Then, the anomaly score of the next frame is obtained by sliding window movement, thereby realizing frame-level anomaly detection of the surveillance video. Any feature that causes the model to estimate a low log density will be assigned a higher anomaly score. The higher the score, the more likely the frame is to be an anomaly frame.

[0063] Figure 2 This is a schematic diagram of the detection results of the UCSD Ped2 dataset provided in an embodiment of this application. Figure 3 This is a schematic diagram of the detection results of the CUHK Avenue dataset provided in the embodiments of this application. Figure 4 This is a schematic diagram of the detection results of the ShanghaiTech dataset provided in the embodiments of this application, as shown below. Figures 2-4 As shown, in one embodiment of this application, the video anomaly detection method based on cross-scale multi-class complementary feature integration provided in this application was used to detect anomalies on three benchmark datasets: UCSD Ped2, CUHKAvenue, and ShanghaiTech. The Micro AUC results were 99.8%, 94.4%, and 87.3%, respectively. Here, EFC-VAD represents the result of the proposed video anomaly detection method based on cross-scale multi-class complementary feature integration, S represents the detection result using only general semantic representation, M represents the detection result using only quantized motion descriptors, C represents the detection result using only specific continuous features, and GT represents the ground truth. Comparing the normalized anomaly score curves of each feature component of CFE-VAD, CFE-VAD, which integrates all features (S+M+C), has the highest anomaly score in the anomaly interval (pink shaded area), and decreases rapidly after returning to normal, confirming the effectiveness of cross-scale multi-class complementary feature integration for video anomaly detection.

[0064] The following describes the video anomaly detection device based on cross-scale multi-class complementary feature integration provided in this application. The video anomaly detection device based on cross-scale multi-class complementary feature integration described below can be referred to in correspondence with the video anomaly detection method based on cross-scale multi-class complementary feature integration described above.

[0065] Figure 5 This is a schematic diagram of the structure of a video anomaly detection device based on cross-scale multi-class complementary feature integration provided in an embodiment of this application, as shown below. Figure 5 As shown, the device 500 includes: The extraction module 510 is used to extract frame-level general semantic features, target-level quantized motion features, and frame-level specific continuity features from the video to be detected. The estimation module 520 is used to estimate the normal feature distribution density corresponding to the frame-level general semantic features, the target-level quantized motion features, and the frame-level specific continuous features, respectively. The detection module 530 is used to obtain abnormal results of the video to be detected based on the normal feature distribution density.

[0066] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0067] Based on the methods in the above embodiments, Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown in the illustration, this application provides an electronic device that may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions from the memory 630 to execute the video anomaly detection method based on cross-scale multi-class complementary feature integration described in the above embodiment.

[0068] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the video anomaly detection method based on cross-scale multi-class complementary feature integration described in the various embodiments of this application.

[0069] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program. When the computer program is run on a processor, it causes the processor to execute the video anomaly detection method based on cross-scale multi-class complementary feature integration in the above embodiments.

[0070] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the video anomaly detection method based on cross-scale multi-class complementary feature integration in the above embodiments.

[0071] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0072] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0073] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0074] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0075] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A video anomaly detection method based on cross-scale multi-class complementary feature integration, characterized in that, The application relates to an abnormality detection method for a video. The method comprises the following steps: extracting frame-level general semantic features, target-level quantized motion features and frame-level special continuity features of a video to be detected; estimating normal feature distribution densities corresponding to the frame-level general semantic features, the target-level quantized motion features and the frame-level special continuity features respectively; 2.The video anomaly detection method based on cross-scale multi-class complementary feature integration according to claim 1, characterized in that, obtaining an abnormality result of the video to be detected based on the normal feature distribution densities. The extraction process of the frame-level general semantic features comprises the following steps: dividing the video to be detected into a plurality of video blocks; embedding the plurality of video blocks to obtain feature vectors; 3.The video anomaly detection method based on cross-scale multi-class complementary feature integration according to claim 1, characterized in that, encoding the feature vectors by using a pre-trained Vit encoder to obtain the frame-level general semantic features. The extraction process of the target-level quantized motion features comprises the following steps: obtaining optical flow information of each frame of the video to be detected; for each frame of the video to be detected, the following steps are performed: obtaining target detection boxes of all motion targets in the video frame; converting the target detection boxes into target region masks; cropping the optical flow information of the video frame by using the target region masks to obtain optical flows corresponding to all motion targets respectively; 4.The video anomaly detection method based on cross-scale multi-class complementary feature integration according to claim 1, characterized in that, dividing the optical flows into N directions, and obtaining optical flow amplitudes of all motion targets in the N directions of the corresponding optical flows to form N-dimensional optical flow amplitude vectors corresponding to all motion targets respectively, wherein N is a positive integer. The extraction process of the frame-level special continuity features comprises the following steps: inputting a preset training set into an encoder network of a 3D CNN architecture to obtain an encoding result, wherein the preset training set comprises a plurality of video clips; training a special continuity encoder based on the encoding result, wherein the special continuity encoder comprises the encoder network of the 3D CNN architecture, a continuity judgment projection network, a missing frame positioning projection network and a missing frame estimation projection network; inputting the video to be detected into the trained special continuity encoder to obtain continuity backbone features and projection network features; 5.The video anomaly detection method based on cross-scale multi-class complementary feature integration according to claim 1, characterized in that, splicing the continuity backbone features and the projection network features to obtain the frame-level special continuity features. The estimation of the normal feature distribution densities corresponding to the frame-level general semantic features, the target-level quantized motion features and the frame-level special continuity features respectively comprises the following steps: performing feature distribution estimation on the frame-level general semantic features, the target-level quantized motion features and the frame-level special continuity features by using a denoising estimation method to obtain normal feature distributions of the frame-level general semantic features, normal feature distributions of the target-level quantized motion features and normal feature distributions of the frame-level special continuity features; 6.The video anomaly detection method based on cross-scale multi-class complementary feature integration according to claim 1 or 5, characterized in that, calculating distribution densities of the frame-level general semantic features, the target-level quantized motion features and the frame-level special continuity features of each video frame of the video to be detected in the normal feature distributions. The obtaining of the abnormality result of the video to be detected based on the normal feature distribution densities comprises the following steps: integrating the normal feature distribution densities of the frame-level general semantic features, the normal feature distribution densities of the target-level quantized motion features and the normal feature distribution densities of the frame-level special continuity features of each video frame of the video to be detected respectively to obtain abnormal values corresponding to each video frame respectively. Anomaly discrimination is performed on each video frame based on the corresponding abnormal value of each video frame, and an anomaly result of the video to be detected is obtained.

7. An apparatus for video anomaly detection based on cross-scale multi-class complementary feature integration, comprising: The method comprises the steps of: extracting a frame-level general semantic feature, a target-level quantized motion feature and a frame-level special continuity feature of the video to be detected; estimating a normal feature distribution density corresponding to the frame-level general semantic feature, the target-level quantized motion feature and the frame-level special continuity feature respectively; detecting an anomaly result of the video to be detected based on the normal feature distribution density.

8. An electronic device, comprising: The method comprises the steps of: at least one memory for storing a computer program; at least one processor for executing the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute the video anomaly detection method based on cross-scale multi-class complementary feature integration according to any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. When the computer program runs on the processor, the processor is caused to execute the video anomaly detection method based on cross-scale multi-class complementary feature integration according to any one of claims 1-6.

10. A computer program product, characterised in that, When the computer program product runs on the processor, the processor is caused to execute the video anomaly detection method based on cross-scale multi-class complementary feature integration according to any one of claims 1-6.

Images