Video anomaly detection method and system based on local perception and global reasoning

Abstract

The invention provides a video anomaly detection method and system based on local perception and global reasoning, and belongs to the technical field of video anomaly detection, and the method comprises the steps: obtaining a to-be-detected continuous video frame sequence; inputting the video frame sequence into a local anomaly sensing module and a global anomaly reasoning module in parallel; wherein the local anomaly sensing module is used for detecting appearance and motion anomalies and time-space dynamic deviation anomalies of a target level and outputting a local anomaly score; the global anomaly reasoning module is used for detecting frame-level scene semantic anomaly and event-level sequential logic anomaly and outputting a global anomaly score; and carrying out standardization, time sequence smoothing and adaptive weighted fusion on the local abnormal score and the global abnormal score to generate a final frame-level abnormal score, and judging whether the video sequence is abnormal or not according to the frame-level abnormal score. According to the method, the comprehensiveness, the accuracy and the robustness of anomaly detection in a complex real scene are effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of video anomaly detection technology, and particularly relates to a video anomaly detection method and system based on local perception and global reasoning. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Video anomaly detection (VAD) is one of the core research directions in the fields of computer vision and intelligent security. It aims to automatically identify events that do not conform to normal behavioral patterns, such as chasing, trespassing into restricted areas, and abnormal gatherings, from massive amounts of surveillance video. With the large-scale deployment of urban security systems, traditional manual inspection methods are no longer sufficient to meet the demands for real-time and efficient anomaly detection. Therefore, achieving automated and intelligent video anomaly detection has significant application value.

[0004] Early video anomaly detection methods largely relied on hand-designed features or traditional machine learning models. While simple to implement, these methods struggled to capture the complex spatiotemporal semantic information within videos and had limited ability to model dynamic changes and contextual relationships. In recent years, with the development of deep learning technology, reconstruction- or prediction-based video anomaly detection (VAD) methods have gradually become mainstream. These methods learn normal video patterns using models such as autoencoders (AEs), variational autoencoders (VAEs), or generative adversarial networks (GANs), and identify anomalies during the testing phase based on reconstruction errors or prediction biases.

[0005] However, these methods have the following drawbacks. First, reconstruction-based methods are prone to producing low reconstruction errors even for anomalous samples due to overly strong model generalization capabilities, leading to missed detections. Second, prediction-based methods lack sensitivity to static or low-dynamic appearance anomalies, making it difficult to effectively identify subtle anomalies. Furthermore, single-mechanism methods often cannot simultaneously address the detection needs of both local target-level anomalies and global scene-level anomalies, resulting in insufficient detection comprehensiveness in complex real-world scenarios. To alleviate these problems, researchers have attempted to introduce memory modules to enhance the modeling ability of normal patterns. However, the design of memory modules still faces issues such as incomplete memory coverage, low retrieval efficiency, and feature redundancy. Simultaneously, existing methods often focus on a single modality or single granularity, lacking the systematic modeling ability for multi-scale, multi-dimensional anomalies, especially showing significant shortcomings in local fine-grained perception and global semantic reasoning. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention proposes a video anomaly detection method and system based on local perception and global reasoning. It aims to address the core challenges of incomplete detection dimensions and global-local perception imbalance caused by the reliance on a single mechanism in existing technologies. Specifically, this method systematically optimizes both local perception and global reasoning: at the local perception level, fine-grained feature extraction and dynamic modeling enable accurate capture of target-level anomalies. At the global reasoning level, semantic understanding and temporal logic analysis enable deep identification of scene-level anomalies. Through the synergistic effect of local and global approaches, this method achieves comprehensive anomaly coverage from microscopic behavior to macroscopic scenes, significantly improving detection accuracy, comprehensiveness, and generalization ability in complex real-world scenarios.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: In a first aspect, the present invention discloses a video anomaly detection method based on local perception and global reasoning, comprising: Obtain the sequence of consecutive video frames to be detected; The video frame sequence is input in parallel to the local anomaly perception module and the global anomaly inference module; wherein, the local anomaly perception module is used to detect target-level appearance and motion anomalies as well as spatiotemporal dynamic deviation anomalies, and outputs a local anomaly score; the global anomaly inference module is used to detect frame-level scene semantic anomalies as well as event-level temporal logic anomalies, and outputs a global anomaly score. The local anomaly scores and global anomaly scores are standardized, temporally smoothed, and adaptively weighted to generate the final frame-level anomaly score, and the presence of anomalies in the video sequence is determined based on the frame-level anomaly score.

[0008] Secondly, this invention discloses a video anomaly detection system based on local perception and global reasoning, comprising: The data acquisition module is used to acquire the continuous video frame sequence to be detected; A multi-branch scoring module is used to input the video frame sequence in parallel to a local anomaly perception module and a global anomaly inference module; wherein, the local anomaly perception module is used to detect target-level appearance and motion anomalies as well as spatiotemporal dynamic deviation anomalies, and output local anomaly scores; the global anomaly inference module is used to detect frame-level scene semantic anomalies as well as event-level temporal logic anomalies, and output global anomaly scores. The anomaly detection module is used to standardize, temporally smooth, and adaptively weight and fuse the local anomaly scores and global anomaly scores to generate the final frame-level anomaly score, and to determine whether there are anomalies in the video sequence based on the frame-level anomaly score.

[0009] Thirdly, the present invention discloses an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when run by the processor, complete the steps of the video anomaly detection method based on local perception and global reasoning described above.

[0010] Fourthly, the present invention discloses a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the video anomaly detection method based on local perception and global reasoning described above.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention features network innovation, achieving comprehensive coverage. For the first time, it systematically integrates four complementary detection mechanisms—target-level appearance-motion memory, spatiotemporal dynamic deviation modeling, frame-level scene semantic memory, and event-level normal knowledge retrieval—into a unified framework. Through a multi-branch collaborative architecture, it can simultaneously capture both local fine-grained anomalies and global macroscopic anomalies, achieving comprehensive coverage of multi-dimensional and multi-scale anomalies and significantly improving the comprehensiveness of detection.

[0012] This invention achieves more precise local perception. By constructing a dual memory database of appearance and motion and combining it with target-level feature extraction, fine-grained monitoring of the static and dynamic attributes of individual targets is realized. At the same time, by utilizing an improved spatiotemporal prediction network and a maximum local error mechanism, the sensitivity and localization capability for anomalies that violate short-term physical laws are effectively enhanced, avoiding interference from normal areas in anomaly detection and improving the accuracy of local detection.

[0013] This invention achieves deeper global reasoning. By introducing semantic priors from a visual-language model and temporal pyramid pooling, it enhances the model's ability to understand the contextual rules of a scene. By utilizing a hash twin network to encode long-term event logic into compact knowledge representations and perform fast retrieval, it effectively identifies semantic and logical anomalies that are difficult to detect using traditional methods, thus achieving a deep semantic understanding of complex scenes.

[0014] This invention achieves adaptive fusion, enhancing robustness. By adaptively determining the optimal fusion weights for each branch through a grid search strategy, the model can automatically adjust its decision focus based on the data characteristics of different scenarios, thus enhancing its adaptability and robustness in various real-world situations.

[0015] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0016] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0017] Figure 1 This is a flowchart illustrating the overall framework of the video anomaly detection method based on local perception and global reasoning as described in Embodiment 1 of the present invention.

[0018] Figure 2 Figure 1 is a schematic diagram of the memory bank partitioning construction described in Embodiment 1 of the present invention; wherein, Figure (a) shows the patch partitioning process of appearance features, Figure (b) shows the patch partitioning process of motion features, and Figure (c) shows the patch partitioning process of scene semantic features.

[0019] Figure 3 This is a schematic diagram of the improved spatiotemporal U-Net structure described in Embodiment 1 of the present invention.

[0020] Figure 4 This is a schematic diagram of the hash twin network structure described in Embodiment 1 of the present invention.

[0021] Figure 5 This is a schematic diagram of the improved spatiotemporal U-Net architecture described in Embodiment 1 of the present invention. Detailed Implementation

[0022] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0023] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0024] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0025] Example 1 In one or more embodiments, a video anomaly detection method based on local perception and global reasoning is disclosed, such as... Figure 1 As shown, it includes: Step S1: Obtain the sequence of continuous video frames to be detected.

[0026] The system obtains the original video sequence to be detected from monitoring equipment or video databases, and extracts consecutive frame images using a video decoder. The frame images are then standardized in size and pixel size, and scaled to a preset resolution. Based on the temporal modeling requirements of each branch, the consecutive frames are organized into fixed-length video segments, providing standardized input data for subsequent parallel processing.

[0027] Step S2: Input the video frame sequence in parallel to the local anomaly perception module and the global anomaly inference module; the local anomaly perception module is used to detect target-level appearance and motion anomalies as well as spatiotemporal dynamic deviation anomalies, and output local anomaly scores; the global anomaly inference module is used to detect frame-level scene semantic anomalies as well as event-level temporal logic anomalies, and output global anomaly scores.

[0028] The local perception module achieves fine-grained anomaly detection through a two-branch collaborative architecture. This includes a target-level appearance-motion memory branch that extracts target features using YOLOv5 and CLIP models, generating target-level appearance motion anomaly scores through dual-memory comparison; and a spatiotemporal dynamic deviation branch that performs frame prediction based on an improved spatiotemporal U-Net, employing a maximum local error mechanism to extract spatiotemporal dynamic deviation anomaly scores from prediction errors. These two branches achieve accurate perception of local anomalies from both static attribute and dynamic behavior dimensions. The global reasoning module, through a two-branch architecture, achieves macroscopic semantic and logical anomaly detection. The frame-level scene semantic memory branch extracts global features using the CLIP model, constructs a scene memory bank through temporal pyramid pooling, and calculates scene semantic anomaly scores based on feature distance. The event-level normal knowledge retrieval branch encodes the video into compact hash codes using a hash twin network, establishes a hash knowledge base, and determines event-level logical anomaly scores through minimum matching distance. These two branches achieve deep recognition of global anomalies from both scene layout and event logic dimensions.

[0029] Specifically, the preprocessed video frames are input in parallel into the local anomaly perception module and the global anomaly inference module to generate multi-dimensional anomaly scores respectively. This step is the core of the invention to realize multi-dimensional anomaly detection. Its data processing flow consists of four branches working in parallel, which together constitute the system's perception and inference capabilities.

[0030] The local anomaly perception module includes a target-level appearance-motion memory branch for detecting target-level appearance and motion anomalies, and a spatiotemporal dynamic deviation branch for detecting spatiotemporal dynamic deviation anomalies. The global anomaly reasoning module includes a frame-level scene semantic memory branch for detecting frame-level scene semantic anomalies, and an event-level normal knowledge retrieval branch for detecting event-level temporal logic anomalies.

[0031] First, the target-level appearance-motion memory branch of the local anomaly perception module is responsible for detecting appearance distortions and abnormal motion behaviors of individual targets in the video. Its memory bank construction process is as follows: Figure 2 As shown in (a) and (b) in the figure.

[0032] First, a pre-trained object detection model, YOLOv5, is used to detect objects in a continuous video frame sequence. After obtaining the target regions, all object regions are uniformly adjusted to a standard size and then input into the ResNet-101 visual encoder of the CLIP model for feature extraction. During the feature extraction process, the intermediate layers of the network output features. and The visual features of the target region are enhanced, and a normal appearance memory bank and a normal motion memory bank are constructed by sampling the greedy core set, which solves the problems of high redundancy and low retrieval efficiency of traditional memory banks.

[0033] In this embodiment, ResNet-101 is a deep residual network with two intermediate layers, corresponding to the second and third residual blocks of ResNet-101. The receptive field of the second layer is small, and the output features of the second residual block retain fine-grained features such as the local texture and contour of the target. The receptive field of the third layer is large, and the output features of the third residual block can capture the overall structure of the target and global semantic features related to the category.

[0034] To adapt to target-level fine-grained detection tasks, a method based on local detail features is adopted. With the core as the basis, and the pooled global semantic features The strategy of splicing together forms an enhanced visual feature representation, and the calculation method can be referred to formula (1): (1) in, To enhance the visual features of the target region, For average pooling operation, The input target region image (an image that has been detected by YOLOv5 and resized to a standard size). This is a feature splicing operation.

[0035] To construct a feature representation that incorporates temporal information, the enhanced visual features of the target regions across all targets in consecutive d frames are used. The stacking along the time dimension forms a local spatiotemporal feature tensor. .in, This represents the total number of feature channels after splicing. , Here, represents the height and width of the feature map, respectively, and n is the total number of targets extracted by YOLOv5. Since... Due to the high channel dimension, Split Pooling is used for channel compression to reduce storage and computational overhead. Each channel is evenly divided into The channels are grouped, and the average value of each group is calculated to obtain the compressed local spatiotemporal features. ,in, This represents the number of channels after compression.

[0036] In constructing the appearance and motion memory databases, two independent databases are built to store normal appearance and motion patterns respectively, providing a reliable reference benchmark for anomaly detection. The database construction process is as follows: Figure 2 As shown in (a) and (b) in the figure.

[0037] Appearance memory The structure is as follows: First, the compressed local spatiotemporal features are analyzed. Global average pooling is performed along the time dimension to integrate appearance information from multiple frames and eliminate time dimension variations, as shown in formula (2): (2) in, The feature tensor after time pooling. It is a time-based global pooling operation.

[0038] Next, spatial average pooling is used to further compress the spatial size and reduce dimensional redundancy, resulting in spatial feature blocks, the mathematical expression of which is shown in formula (3): (3) in, The feature tensor after spatial average pooling. and These represent the height and width of the feature map after pooling, respectively.

[0039] The feature tensor after spatial average pooling Reconstructing it into a two-dimensional matrix, the dimensional transformation process is shown in formula (4): (4) The characteristic tensor of a two-dimensional matrix Stored in the space appearance memory bank .

[0040] Motion patch partitioning generates motion memory library The steps to adapt the patch are as follows: First, calculate the characteristic tensor. The difference between adjacent frames yields a motion feature tensor of dimension 1. To reflect the motion changes of an object between consecutive frames, the calculation process is shown in formula (5): (5) in, The motion features are between frames k and k+1. Let be the local spatiotemporal feature tensor between frames k and k+1.

[0041] The memory phase uses global average pooling to extract the stationary central trend of the motion, and the generation process can be described by formula (6): (6) in, The characteristic tensor of the steady-state tendency of the object's motion. For global average pooling, Let be the motion feature tensor.

[0042] The same appearance patch partition is flattened into a two-dimensional matrix by Reshape, and its dimensional transformation process is shown in formula (7): (7) After sampling and optimization using a greedy core set, the data is stored in the motion memory bank. .

[0043] The logic of the greedy core set sampling optimization should be understood as follows: without losing the ability to represent normal patterns, select the most representative minimum subset from the massive amount of original normal features and replace the full set of features in the memory. Specifically: Step 1: Initialization: Determine the original feature pool and initial core points. The flattened two-dimensional matrix in formula (7) The original motion feature pool is used to randomly select one feature vector as the initial core point (the first row of features is selected from Tp) to form the initial core set, and the remaining features form the candidate set.

[0044] Step 2: Select the feature with the broadest coverage and add it to the core set. Repeat the following steps until the core set size reaches a preset threshold (e.g., 10% of the original feature pool): 1. For each candidate feature, calculate its minimum Euclidean distance to all features in the core set. The larger the distance, the less the motion pattern represented by that feature is covered by the core set. 2. Select the candidate feature with the largest distance and add it to the core set, while removing it from the candidate set.

[0045] Step 3: Replacement and Storage. A feature memory is built by replacing the entire feature pool with the core set. After the iteration, the size of the core set is only 10% of the original feature pool.

[0046] Preferably, in the motion memory bank, global average pooling is used in the memory stage to store typical stationary states of normal motion, aiming to capture its central trend. In the inference stage, global max pooling is used to sensitively extract local peak responses in the motion sequence to be tested. This mismatched pooling strategy enhances the detection capability of abnormal motion. The calculation process is shown in formula (8): (8) During the inference phase, a most anomalous patch selection mechanism is used for the test samples. First, the set of appearance feature blocks of the test samples is selected. Select and memory The farthest feature block Then calculate the feature block and The distance between the nearest neighbors is used as the appearance anomaly score, and the calculation methods are shown in formulas (9) and (10) respectively: (9) (9) in, The most unusual appearance feature block, For its in The nearest neighbor in A score is given for appearance abnormalities.

[0047] Similarly, the calculation methods for motion characteristics are shown in formulas (11) and (12), respectively: (11) (12) in, The most abnormal motion feature block, For its in The nearest neighbor in The score is for abnormal movement.

[0048] The final target-level appearance-motor memory branch comprehensive abnormality score is obtained by weighted summation of appearance and motor scores, as shown in formula (13): (13) in, , This is an adjustable weighting parameter used to balance the contribution of appearance and motion abnormalities to the final score.

[0049] In modern intelligent surveillance scenarios, anomalies in individual targets often manifest as sudden changes in appearance attributes or abnormal changes in movement patterns. This branch constructs a dual-memory mechanism to model appearance and movement features separately, achieving accurate perception of target-level anomalies.

[0050] Second, the spatiotemporal dynamic deviation branch of the local anomaly perception module addresses the issues of low sensitivity to static anomalies and significant interference in normal regions in existing prediction methods by innovatively improving the spatiotemporal U-Net architecture and designing a time-dimensional compression layer. Its structural diagram is shown below. Figure 3 As shown, the encoder part is designed for video frame prediction tasks, and uses the I3D network based on ResNet-50 as the backbone.

[0051] The improved spatiotemporal U-Net architecture is based on a 5-level encoder-decoder architecture, including a five-level encoder, a five-level decoder, and a temporal compression layer that skips between the encoders and decoders at the corresponding levels. By preserving the temporal dimension, using the temporal compression layer, optimizing the convolutional block design, and residual connections, it achieves efficient modeling of video motion information. The specific connection relationships of each layer are as follows: The input of the spatiotemporal U-Net is 8 consecutive video segments (resolution 256×256, feature dimension represented as (3,8,256), where 3 is the number of channels, 8 is the time step, and 256 is the spatial size). The encoder gradually reduces the spatial and temporal resolution to extract high-level semantic features, while the decoder gradually restores the spatial resolution through transposed convolution, finally generating a single-frame prediction result. The corresponding encoder and decoder layers achieve information fusion through temporal compression layers and feature concatenation. The encoder includes an input layer, an L1 level encoder, an L2 level encoder, an L3 level encoder, an L4 level encoder, and an L5 level encoder connected in sequence. The L1 level encoder includes a 3D convolutional layer, a max pooling layer, three fn1 (64) convolutional blocks, and nine convolutional layers connected in sequence. The L2 level encoder includes a max pooling layer, two sets of fn1 (128) convolutional blocks and convolutional layer structures connected in sequence, and ten convolutional layers. The L3 level encoder includes three fn2 (256) convolutional blocks, a 3D convolutional layer, and sixteen convolutional layers connected in sequence. The L4 level encoder includes an fn2 (512) convolutional block, a convolutional layer, an fn1 (512) convolutional block, and seven convolutional layers. The L5 level encoder includes an average pooling layer. The decoder includes an L1-level decoder, an L2-level decoder, an L3-level decoder, an L4-level decoder, an L5-level decoder, and an output layer. The L1-level decoder includes a 3D convolutional layer, a transposed convolutional layer, and a feature concatenation layer with the L5-level encoder after TSL processing, connected in sequence. The L4-level decoder includes an fn3 (512) convolutional block and a feature concatenation layer with the L4-level encoder after TSL processing. The L3-level decoder includes an fn3 (256) convolutional block and a feature concatenation layer with the L3-level encoder after TSL processing. The L2-level decoder includes an fn3 (128) convolutional block and a feature concatenation layer with the L2-level encoder after TSL processing. The L1-level decoder includes an fn3 (64) convolutional block and a feature concatenation layer with the L1-level encoder after TSL processing. The output layer includes an fn3 (32) convolutional block and two convolutional layers.

[0052] The fn1(n) convolutional block consists of a (n,3,1²) convolutional layer, a (n,1,3²) convolutional layer, and a (4n,1,1²) convolutional layer. The fn2(n) convolutional block consists of a (n,1,1²) convolutional layer, a (n,1,3²) convolutional layer, and a (4n,1,1²) convolutional layer connected in sequence. The fn3(n) convolutional block consists of a (n,1,3²) convolutional layer, a (n,1,3²) convolutional layer, a (4n,1,3²) convolutional layer, and a (2n,1,2²) transposed convolutional layer connected in sequence, where n is the feature dimension, n=32,64,128,256,512.

[0053] In the encoder section, it is built based on the I3D network (ResNet-50 backbone), and the connection logic of each layer is as follows: The input layer first connects to a 3D convolutional layer with a kernel of (64,5,7²) and a stride of (1,2²), outputting a first intermediate feature map of (64,8,128²). After passing through a max pooling layer (kernel (1,3²), stride (1,2²)), it becomes a second intermediate feature map of (64,8,64²). Then, it passes through 3 fn1 (64) convolutional blocks and 9 (1,1²) convolutional layers to output a third intermediate feature map of (256,8,64²), completing the L1 layer construction. From L1 to L2, a max pooling layer (kernel (2,1²), stride (2,1²)) is used to compress the time dimension to 4, outputting a first encoded feature map of (256,4,64²), which is then connected to 2 sets of fn1 (128) blocks. A convolutional block and a (1,1²) convolutional layer (stride (1,2²)) are combined with 10 (1,1²) convolutional layers to obtain a second encoded feature map of (512, 4, 32²). L2 to L3 use 3 fn2(256) convolutional blocks, 3D convolutions with a stride of (1, 2²), and 16 (1,1²) convolutional layers to output a third encoded feature map of (1024, 4, 16²). L3 to L4 connects the fn2(512) convolutional block, the (1,1²) convolutional layer, the fn1(512) convolutional block (stride (1,2²)), and a 7 (1,1²) convolutional layer fn2(512) block to output (2048, 4, 32²). The fourth encoded feature map is (8²). From L4 to L5, the time dimension is compressed to 1 by an average pooling layer (kernel (4,7²) and stride (1,1²)) to output the fifth encoded feature map (2048,1,2²). At the same time, each layer of the encoder skips the convolutional block through residual connections and directly superimposes the input and output features to alleviate gradient vanishing. All convolutional layers are followed by batch normalization (BN) and ReLU activation (the last ReLU layer is removed to preserve feature diversity).

[0054] Temporal compression layers are key to resolving inconsistencies in the temporal dimensions of encoding and decoding. They are added to the residual connections between the encoder and the corresponding decoder layers. Each TSL contains a convolutional layer with a kernel of (C,4,1²) and a stride of (1,1²) (where C is the number of channels in the encoder layer). It can fuse features from different time steps and compress the temporal dimension of the encoder features to 1, allowing the encoding and decoding features to be concatenated along the channel dimension as input to the next level of the decoder.

[0055] The decoder has a custom structure, and the connection logic of each layer is as follows: The input layer receives the fifth encoded feature map (2048,1,2²) output by encoder L5, and outputs the fourth intermediate feature map (512,1,4²) through 3D convolution with kernel (512,1,1²). Then, it outputs the fifth intermediate feature map (512,1,8²) through a transposed convolutional layer (kernel (512,1,2²) and stride (1,2²)). Subsequently, the fifth intermediate feature map is concatenated with the features processed by encoder L5 after TSL to obtain the fifth decoded feature map (2048,1,8²), thus completing the L5 layer construction. L5 to L4 is connected to the fn3 (512) convolutional block, which outputs the fourth decoded feature map (1024,1,16²), and concatenates it with the features processed by encoder L4 after TSL. The L4 to L3, L3 to L2, and L2 to L1 are connected to the fn3(256), fn3(128), and fn3(64) convolutional blocks respectively, each outputting a third decoding feature map of (512,1,32²), a second decoding feature map of (256,1,64²), and a first decoding feature map of (128,1,128²), and all of them are concatenated with the features of the corresponding layer of the encoder after TSL processing; the L1 to the output layer is connected to the fn3(32) convolutional block outputting a sixth intermediate feature map of (64,1,256²), and the sixth intermediate feature map is finally passed through two 3D convolutional layers with kernels of (3,1,3²) to output a predicted frame (3,1,256²) with the same resolution as the input frame. At the same time, except for the last layer, all convolutional layers of the decoder are followed by BN and Leaky ReLU activation function layers, and the transposed convolution stride is fixed at (1,2²) to double the spatial resolution.

[0056] The above structure is as follows Figure 5As shown, Conv is a convolutional layer; MaxPool is a max pooling layer; AvgPool is an average pooling layer; TransConv is a transposed convolutional layer; fn1(n) includes (n,3,1²), (n,1,3²), and (4n,1,1²) convolutional layers connected in sequence; fn2(n) includes (n,1,1²), (n,1,3²), and (4n,1,1²) convolutional layers connected in sequence; fn3(n) includes two (n,1,3²), (4n,1,3²), and the transposed (2n,1,2²) convolutional layers connected in sequence; the time dimension of each level of the unified encoder of TSL (Temporal Compression Layer) is 1, realizing the matching of the encoder and decoder feature dimensions; the features of each level of the decoder and the corresponding level of the encoder after TSL processing are concatenated along the channel dimension to supplement the detailed information.

[0057] Let the spatiotemporal feature map output by the encoder be... ,in, This is the number of input channels. It is the input time dimension. The feature map space size.

[0058] The time-dimensional compression layer uses convolution kernel parameters Step size set to Output feature map The time dimension is compressed to 1. The mathematical expression of this process can be described by formula (14): (14) in, For convolution kernel weights, For bias terms, For output channel index, For spatial location index, The time dimension index of the input feature map X (corresponding to the time step of X, with a value ranging from 1 to T). in ), The channel dimension index of the input feature map X (corresponding to the channels of X, with values ​​ranging from 1 to D). in ).

[0059] The training process aims to reduce the deviation between the predicted frames output by the model and the actual real frames in the video, and adopts a joint loss function that combines mean squared error (MSE) and L1 loss. The specific definition of this loss function is shown in formula (15): (15) in, Represents frame-level error loss, used to quantify the overall bias in single-frame prediction; This represents the prediction frame output by the spatiotemporal U-net for the t-th frame. The value of each pixel, Represents the t-th frame of the real image. The actual value of each pixel. Represents the total number of pixels contained in a single frame of an image. The weight coefficient of the L1 loss term is set to 1 by default in this embodiment. It is used to balance the sensitivity of the mean square error term to the overall error and the robustness of the L1 loss term to local outliers, so as to achieve joint optimization of spatial details and overall consistency of the predicted frame.

[0060] Based on the MSE, a frame-level error map covering the entire frame is generated, and the frame is predicted. With real frames The pixel-level absolute error map is shown in formula (16): (16) in, This is a pixel-level absolute error map.

[0061] To accurately locate abnormal regions and avoid interference from prediction errors in normal regions, this branch introduces a maximum local error mechanism, using the maximum error in a local region instead of the traditional whole-frame error as the anomaly. Specifically, this involves using a window size... Square sliding window for error plot Perform max pooling to obtain a local maximum error map. Among them, window size Using normal videos from the training set to generate pseudo-anomaly samples, different... Select the optimal AUC for detection on pseudo-anomaly samples. This avoids the subjectivity of manual parameter adjustment.

[0062] Take the local maximum error map The maximum value is used as the outlier score of the spatiotemporal dynamic deviation. This calculation process can be represented by formula (17): (17) This design ensures that a high score is output only when there are local high error regions within the frame, effectively suppressing errors in normal regions.

[0063] Third, the frame-level scene semantic memory branch of the global anomaly inference module aims to detect anomalies that violate the overall semantics and layout of the scene, inputting the entire frame image to the CLIP encoder. In feature level selection, global semantic features are used. The core strategy, supplemented by detailed features As a supplement, frame-level visual features are formed through formula (18). .

[0064] (18) in, As frame-level visual features, This is the input full-frame image.

[0065] To enhance the global feature representation capability, Simultaneously perform global average pooling and global max pooling, and add the results of the two pooling methods to obtain the frame-level global feature vector. ,in The calculation process is shown in formula (19): (19) in, Indicates the first The global feature vector of the frame. Where is the number of feature channels in the global branch, and D is the number of input frames. For global average pooling, This is a global max pooling algorithm. The feature vector contains both the overall context of the scene and retains information from key regions, providing a high-quality feature foundation for subsequent correlation analysis.

[0066] Stack the frame-level global feature vectors on the time dimension in chronological order of the frames. Remodeling into temporal global feature tensor It includes global features and temporal sequence information of D-frames, capturing multi-scale contextual information of the scene in the time dimension.

[0067] For temporal global feature tensor Execution time-series pyramid pooling, with different numbers of max pooling iterations denoted as... , For the number of pooling operations, ( Extract multi-scale temporal features using the formulas =0,1,2) to generate scene semantic feature blocks. The specific partitioning process is as follows: Figure 2 As shown in (c), the generation process can be referred to formula (20): (20) in, This indicates that the input tensor is processed. Second max pooling operation. This is a scene semantic feature block that integrates multi-scale temporal information.

[0068] Without altering the data itself, simply swapping the order of the feature channel dimension C and the video frame count dimension D will... Remodeling Subsequently, the size of the memory bank is optimized through greedy core set sampling and stored in the frame-level scene semantic memory bank. This memory bank can cover short, medium, and long-term multi-scale scene association patterns, comprehensively representing the temporal logic of normal scenes.

[0069] During the inference phase, the scene semantics and frame-level scene semantic memory are computed. The scene anomaly score is obtained by calculating the Euclidean distance between the nearest neighbor samples in all feature blocks. The specific calculation methods are shown in formulas (21) and (22): (twenty one) (twenty two) in, This is the semantic feature block for the most unusual scenario. For the most unusual scene semantic feature block in The nearest neighbor in Score the semantic anomalies in the scene.

[0070] Fourth, the event-level normal knowledge retrieval branch of the global anomaly reasoning module adopts a hash twin network architecture, the structure of which is shown in the diagram below. Figure 4 As shown, the network comprises two branches with identical topologies and shared parameters. Each branch takes temporally close video clips as input, which the network learns to map to a compact binary hash space. The backbone of the Hasi Twin network employs a ResNet-50-based I3D network with five coding layers, capable of transforming the input video clips into high-dimensional spatiotemporal semantic feature vectors. The network introduces eight completely independent parallel hash layers, each consisting of a linear layer, layer normalization, and a cascaded sigmoid activation function, ultimately outputting a 32-dimensional real-valued hash code.

[0071] Specifically, the hash twin network has a symmetrical dual-branch structure with identical topology and shared parameters between the two branches. Each branch consists of a concatenated event encoder and a hash encoder. The event encoder uses a ResNet-50-based I3D network (containing 5 coding layers) to transform the input video clips into high-dimensional semantic feature vectors that incorporate spatiotemporal information, providing a high-quality foundation for hash encoding. The hash encoder consists of 8 independent parallel hash layers, each comprising a sequentially connected linear layer, layer normalization, and sigmoid activation function, mapping high-dimensional features to 32-dimensional real-valued hash codes.

[0072] During training, this embodiment introduces two complementary loss functions to fuse the similarity loss and the cross-difference loss with fixed weights to obtain the total hash loss, balancing the two constraint objectives. This fusion process can be represented by formula (23): (twenty three) in, For the total hash loss, The weighting coefficients for cross-difference loss. For hash layer code difference loss, This is for similarity loss.

[0073] Among them, hash code difference loss Calculate the L2 distance between the outputs of any two hash layers, sum them, and then take the average. The larger the loss value, the greater the difference between the hash layers. The calculation method is shown in formula (24): (twenty four) in, , ∈ They represent the first and the The real-valued hash code output by each hash layer. The number of hash layers. This represents the hash code length.

[0074] Similarity loss The similarity of the hash codes output by the two branches is constrained. The similarity between the two hash codes is measured by cosine distance; the smaller the loss value, the higher the similarity. Specifically, it can be calculated using formula (25): (25) in, ∈ These represent the concatenated hash codes of the outputs from the two branches. This represents the time offset between the two input video clips.

[0075] Training a hash twin network allows similar normal events to generate similar hash codes, while maximizing the differences in hash codes output by different hash layers, thus ensuring the distinguishability of knowledge representations.

[0076] For each normal video segment in the training set The hash encoder, after training, outputs... Real-valued hash codes Then, each real-valued hash code is binarized to obtain a binary key. Each hash table uses binary keys and mean vectors as basic storage entries, with the specific storage rules as follows: If binary key Already exists In each hash table, update the mean vector and count under the entry. The new mean vector is calculated as shown in formula (26), and the count is incremented by 1.

[0077] (26) If binary key If the video segment does not exist in the hash table, add a new storage entry, set its count to 1, and its mean vector is the real-value hash code corresponding to the current video segment. .

[0078] When the video segment to be detected Real-value hash codes are generated by the hash encoder. And the normal knowledge representation of the corresponding binary key is retrieved in the b-th hash table. At that time, the matching degree between the two is calculated by L2 distance, as shown in formula (27): (27) After traversing the L2 distances of all B hash layers, the minimum value is selected as the initial anomaly score for the segment to be detected. If no matching binary key is found in a certain hash layer, the distance of that layer is assigned to the preset maximum distance. Since the hash code elements take values ​​in the range [0,1], the maximum L2 distance is... It can be described by formula (28): (28) The final event-level logical exception score can be calculated using formula (29): (29) in, This represents the total number of hash layers. For a single hash layer index (1≤b≤B).

[0079] Step S3: Standardize, temporally smooth, and adaptively weight the local anomaly score and the global anomaly score to generate the final frame-level anomaly score, and determine whether there is anomaly in the video sequence based on the frame-level anomaly score.

[0080] First, Z-score standardization is applied to the scores of each branch to eliminate dimensional differences, and then temporal Gaussian filtering is used to smooth temporal fluctuations. Next, the information from the four branches is integrated through weighted fusion, with the weights determined by grid search optimization. Finally, the scores are normalized to the [0,1] interval, and the optimal parameter combination is adaptively selected based on the AUROC metric to output a unified frame-level anomaly detection result. This part effectively improves the robustness and accuracy of the system's decision-making.

[0081] Step S3-1: Standardize the scores of each branch and map them to a unified distribution space.

[0082] The standardization process employs the classic Z-score method, using a linear transformation based on the mean and standard deviation of each branch score on the validation set. The branch scores are then mapped to a uniform distribution, as shown in formula (30). (30) Where μ is the mean and σ is the standard deviation. Scoring for anomalies in each branch, These are the standardized outlier scores.

[0083] The abnormal scores of adjacent frames should exhibit a smooth transition. However, due to various noise factors during the detection process, the original score sequence often contains unreasonable local fluctuations. Therefore, this embodiment introduces a one-dimensional Gaussian filter to perform temporal smoothing on the standardized score sequence. This improves temporal consistency, as shown in formula (31): (31) in, For one-dimensional Gaussian filtering, Scoring for time-series smoothness anomalies. The standard deviation parameter of the Gaussian filter is used to control the smoothing strength of the time-series outlier score sequences. This is the variable name used to store the specific standard deviation value.

[0084] Step S3-2: The scores of each branch after standardization and smoothing can now be meaningfully fused. This embodiment adopts a linear weighted fusion strategy, adjusting the contribution of each branch to the final result through weight coefficients. The fusion formula is shown in (32): (32) in, , , , These are respectively the scores for smooth target-level appearance motion anomalies, spatiotemporal dynamic deviation anomalies, scene semantic anomalies, and event-level logic anomalies. For the fusion score; The contribution of controlling target-level appearance motion anomalies. Adjusting for the effects of spatiotemporal dynamic deviations The importance of determining semantic anomalies in a scene Adjust the weights of event-level logical exceptions. Weight coefficients. arrive The value range is [0, 1.0], and it satisfies the normalization condition.

[0085] To facilitate subsequent anomaly threshold judgment and result interpretation, the weighted fusion score needs to be mapped to a unified [0,1] interval for anomaly threshold judgment. The calculation method is shown in formula (33): (33) in, This is the final score.

[0086] Preferably, the selection of fusion parameters is crucial to system performance. Traditional methods typically employ empirical settings or simple grid searches, which are difficult to adapt to the characteristics of different datasets. This invention is based on a performance-oriented adaptive parameter optimization strategy, using grid search to find the optimal combination of fusion parameters. (Branch weights) Gaussian filter parameters with a step size of 0.1 The integer step size determines the optimal parameter combination by maximizing the frame-level AUROC. This optimization process can be represented by formula (34): (34) in,( ) represents the optimal combination of parameters found. This is a real label.

[0087] Example 2 In one or more embodiments, a video anomaly detection system based on local perception and global reasoning is disclosed, specifically including: The data acquisition module is used to acquire the continuous video frame sequence to be detected; A multi-branch scoring module is used to input the video frame sequence in parallel to a local anomaly perception module and a global anomaly inference module; wherein, the local anomaly perception module is used to detect target-level appearance and motion anomalies as well as spatiotemporal dynamic deviation anomalies, and output local anomaly scores; the global anomaly inference module is used to detect frame-level scene semantic anomalies as well as event-level temporal logic anomalies, and output global anomaly scores. The anomaly detection module is used to standardize, temporally smooth, and adaptively weight and fuse the local anomaly scores and global anomaly scores to generate the final frame-level anomaly score, and to determine whether there are anomalies in the video sequence based on the frame-level anomaly score.

[0088] Example 3 This embodiment provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, they complete the steps of the video anomaly detection method based on local perception and global reasoning described above.

[0089] Example 4 This embodiment provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the video anomaly detection method based on local perception and global reasoning described above.

[0090] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0091] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0092] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, whereby a series of operational steps are performed to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0093] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0094] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A video anomaly detection method based on local perception and global reasoning, characterized in that, include: Obtain the sequence of consecutive video frames to be detected; The video frame sequence is input in parallel to the local anomaly perception module and the global anomaly inference module; wherein, the local anomaly perception module is used to detect target-level appearance and motion anomalies as well as spatiotemporal dynamic deviation anomalies, and outputs a local anomaly score; the global anomaly inference module is used to detect frame-level scene semantic anomalies as well as event-level temporal logic anomalies, and outputs a global anomaly score. The local anomaly scores and global anomaly scores are standardized, temporally smoothed, and adaptively weighted to generate the final frame-level anomaly score, and the presence of anomalies in the video sequence is determined based on the frame-level anomaly score.

2. The video anomaly detection method based on local perception and global reasoning as described in claim 1, characterized in that, The local anomaly perception module includes a target-level appearance-motion memory branch that uses YOLOv5 and CLIP models to extract target features and generates a target-level appearance motion anomaly score through dual memory comparison; and a spatiotemporal dynamic deviation branch that performs frame prediction based on an improved spatiotemporal U-Net and uses the maximum local error mechanism to extract spatiotemporal dynamic deviation anomaly score from the prediction error. The global reasoning module includes a frame-level scene semantic memory branch that uses the CLIP model to extract global features, constructs a scene memory library through temporal pyramid pooling, and calculates scene semantic anomaly scores based on feature distance. The event-level normal knowledge retrieval branch encodes videos into compact hash codes using a hash twin network, establishes a hash knowledge base, and determines the event-level logical anomaly score by using the minimum matching distance.

3. The video anomaly detection method based on local perception and global reasoning as described in claim 2, characterized in that, The pre-trained object detection model YOLOv5 is used to detect objects in a continuous video frame sequence. After obtaining the target region, all object regions are uniformly adjusted to the standard size and then input into the ResNet-101 visual encoder of the CLIP model for feature extraction. During feature extraction, a strategy of combining local detail features and global semantic features output from the intermediate layers of the network is used to obtain enhanced visual features of the target region. Furthermore, appearance memory and motion memory are constructed through greedy core set sampling.

4. The video anomaly detection method based on local perception and global reasoning as described in claim 2, characterized in that, During the inference phase, a most anomalous patch selection mechanism is employed to generate a target-level appearance motion anomaly score through dual-memory comparison. Specifically: Select the feature block that is farthest from the appearance memory from the set of appearance feature blocks of the test sample, and then calculate the distance between the feature block and the nearest neighbor in the appearance memory as the appearance anomaly score. Select the feature block that is farthest from the motion memory bank from the set of motion feature blocks of the test sample, and then calculate the distance between the feature block and the nearest neighbor in the motion memory bank as the motion anomaly score; The final target-level appearance motion anomaly score is obtained by weighted summation of appearance and motion anomaly scores.

5. The video anomaly detection method based on local perception and global reasoning as described in claim 2, characterized in that, The improved spatiotemporal U-Net includes a five-layer encoder, a five-layer decoder, and a temporal compression layer that skips between the encoders and decoders at the corresponding layers. The expression for the time dimension compression layer is as follows: in, This is the number of input channels. It is the input time dimension. For convolution kernel weights, For bias terms, For output channel index, For spatial location index, The temporal dimension index of the input feature map. This is the channel dimension index of the input feature map.

6. The video anomaly detection method based on local perception and global reasoning as described in claim 2, characterized in that, The frame-level scene semantic memory branch is specifically as follows: Input the entire frame image into the CLIP encoder, concatenate the global semantic features and detail features after average pooling, and then perform global average pooling and global max pooling. Add the two pooling results to obtain the frame-level global feature vector. Frame-level global feature vectors are stacked on the time dimension in chronological order of frames, and the frame-level global feature vectors are reshaped into temporal global feature tensors. Temporal pyramid pooling is then performed on the temporal global feature tensors. Multi-scale temporal features are extracted through max pooling of different numbers to generate scene semantic feature blocks. The scene semantic feature blocks are then resized and stored in the scene semantic memory. During the inference phase, the Euclidean distance between the scene semantics and the nearest neighbor sample in all feature blocks in the scene semantic memory is calculated to obtain the scene anomaly score.

7. The video anomaly detection method based on local perception and global reasoning as described in claim 2, characterized in that, The method of determining the event-level logical anomaly score by using the minimum matching distance is as follows: When the video segment to be detected generates a real-valued hash code through a hash encoder, and the normal knowledge representation of the corresponding binary key is found in the hash table, the matching degree between the two is calculated by L2 distance. After traversing the L2 distances of all hash layers, the minimum value is selected as the initial anomaly score for the segment to be detected; If a matching binary key is not found in a certain hash layer, the distance of that layer is assigned to the preset maximum distance; The final event-level logic exception score is calculated as follows: in, To preset the maximum distance, The degree of matching between the real-valued hash code and the normal knowledge representation. This represents the total number of hash layers. An index for a single hash layer.

8. A video anomaly detection system based on local perception and global reasoning, characterized in that, include: The data acquisition module is used to acquire the continuous video frame sequence to be detected; A multi-branch scoring module is used to input the video frame sequence in parallel to a local anomaly perception module and a global anomaly inference module; wherein, the local anomaly perception module is used to detect target-level appearance and motion anomalies as well as spatiotemporal dynamic deviation anomalies, and output local anomaly scores; the global anomaly inference module is used to detect frame-level scene semantic anomalies as well as event-level temporal logic anomalies, and output global anomaly scores. The anomaly detection module is used to standardize, temporally smooth, and adaptively weight and fuse the local anomaly scores and global anomaly scores to generate the final frame-level anomaly score, and to determine whether there are anomalies in the video sequence based on the frame-level anomaly score.

9. An electronic device, characterized in that, The method includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the video anomaly detection method based on local perception and global reasoning as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the video anomaly detection method based on local perception and global reasoning as described in any one of claims 1-7.

Images