Frame-centered video anomaly detection method and system based on physical motion prior

Abstract

The application belongs to the technical field of intelligent video monitoring, and specifically discloses a frame center type video anomaly detection method and system based on physical motion prior, which comprises the following steps: based on a video to be detected, calculating dense optical flow between adjacent frames to obtain motion information of the video to be detected; performing foreground awareness filtering on the motion information to suppress background motion noise, obtaining foreground motion features, and then extracting global macro motion features; based on the foreground motion features, performing perspective adaptive normalization by using a scene scale diagram to obtain corrected motion features; based on the corrected motion features, extracting local micro motion features; based on the global macro motion features, performing conditional density estimation on the local micro motion features to obtain motion anomaly scores; and obtaining an anomaly detection result of the video to be detected. Through the application, the detection accuracy and robustness of the frame center type video anomaly detection in a complex motion scene can be improved without relying on a target detector.

Description

Technical Field

[0001] This application belongs to the field of intelligent video surveillance technology, and more specifically, relates to a frame-centered video anomaly detection method and system based on physical motion priors. Background Technology

[0002] Video anomaly detection aims to automatically identify events that deviate from normal patterns in surveillance videos, and is widely used in scenarios such as urban security, park monitoring, and campus security. Based on different processing granularities, video anomaly detection methods can be broadly classified into two categories: target-centered methods and frame-centered methods.

[0003] The target-centered method first uses a target detector to locate targets in the image, and then analyzes the appearance and motion trajectory of each target. The frame-centered method does not rely on a target detector, but directly extracts appearance or motion information on a whole frame basis, and uses reconstruction, prediction or density estimation to establish a normal pattern model, and then makes anomaly judgment based on the degree of deviation between the frame to be detected and the normal pattern.

[0004] In terms of utilizing motion information, the frame-centered method usually first calculates the dense optical flow between adjacent frames of the video, and then inputs the overall optical flow of the whole image or the motion features obtained from the statistics of the whole image optical flow into the subsequent model, which then models and discriminates the motion patterns.

[0005] How to improve the detection accuracy and robustness of frame-centered video anomaly detection in complex motion scenarios without relying on a target detector is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this application is to suppress the interference of background motion noise on motion features, correct motion scale distortion caused by perspective projection, and model the dependency between local and global motion without relying on a target detector, so as to improve the detection accuracy and robustness of frame-centered video anomaly detection in complex motion scenes.

[0007] To achieve the above objectives, in a first aspect, this application provides a frame-center-based video anomaly detection method based on physical motion priors, comprising: Based on the video to be detected, the dense optical flow between adjacent frames is calculated to obtain the motion information of the video to be detected. Foreground-sensing filtering is applied to motion information to suppress background motion noise and obtain foreground motion features; Based on foreground motion features, global macroscopic motion features are extracted after statistical analysis. Based on foreground motion features, perspective adaptive normalization is performed using scene scale maps to obtain corrected motion features; where the scene scale map represents the normal motion energy distribution at different spatial locations in the scene. Based on the corrected motion features, local micro-motion features are extracted; Based on global macroscopic motion characteristics, conditional density estimation is performed on local microscopic motion characteristics to obtain motion anomaly scores; Based on motion anomaly scores, the anomaly detection results of the video to be detected are obtained.

[0008] Specifically, motion information is obtained by calculating dense optical flow between adjacent frames based on the video to be detected. The entire process is processed on a frame-by-frame basis, without the need for a target detector, ensuring a frame-centric processing path from the input end. Then, foreground-aware filtering is applied to the motion information to remove background motion noise, retaining only the foreground motion features. This suppresses the interference of spurious optical flow generated by background dynamics on subsequent motion modeling from the source. Furthermore, foreground-aware filtering itself is implemented based on frame-level processing, again without relying on a target detector. Simultaneously, global macroscopic motion features are extracted using the foreground motion features denoised by foreground-aware filtering. Finally, perspective-adaptive normalization is performed on the foreground motion features using a scene scale map. The scene scale map represents the normal motion energy distribution at different spatial locations in the scene, essentially encoding perspective projection. After geometric priors of the shadow are corrected spatially, motion signals that are numerically suppressed due to perspective effects in the depths of the scene are restored, and motion signals in the vicinity are appropriately compressed. The resulting corrected motion features make the feature responses of the same physical motion at different spatial positions comparable, thereby correcting the motion scale distortion caused by perspective projection. Then, local micro motion features are extracted from the corrected motion features. Finally, conditional density estimation of local micro motion features is performed based on global macro motion features, so that the discrimination criteria of local motion patterns are adaptively adjusted with the global motion state. The dependency relationship between local motion and global motion is explicitly modeled. The resulting motion anomaly score has three guarantees: denoising, scale calibration, and context awareness. The anomaly detection results obtained in this way have higher detection accuracy and robustness in complex motion scenes.

[0009] In one possible implementation, foreground-aware filtering is performed on the motion information to suppress background motion noise and obtain foreground motion features, including: Based on the normal training video frame sequence, the median time value of each pixel position is taken to generate a static background image; The current frame is compared with the static background image and then thresholded to obtain the initial foreground mask; Morphological processing is performed on the initial foreground mask to obtain an optimized foreground mask; Motion information is cropped by optimizing the foreground mask, preserving the motion information of the foreground region, and obtaining the foreground motion features.

[0010] It is understandable that frame-level computation is used to achieve coarse separation of foreground and background motion information. The entire process does not rely on a target detector, providing a clean, denoised input for perspective adaptive normalization and conditional density estimation, and avoiding incorrect motion statistics information encoded in scale maps.

[0011] In one possible implementation, the above-mentioned global macroscopic motion features are extracted after statistical analysis based on foreground motion features, including: Global statistics are performed on the foreground motion features of all pixels in the entire foreground region of the current frame to form a frame-level orientation histogram (as a global macroscopic motion feature), which represents the relative distribution of motion directions in the foreground region.

[0012] In one possible implementation, the scene scale map is generated through the following steps: The foreground motion features of each frame in the normal training video are divided into a preset number of spatial grids. For each grid cell, the optical flow vector of the pixels in the grid cell is statistically converted into a direction histogram. The scale value of each grid cell is obtained using the following formula, and the scale values ​​of all grid cells constitute the scene scale map: ; in, Represents grid cells The scale value, i.e., the average motion energy of the grid cell across all normal training frames; Indicates the total number of training frames; Indicates the first Frame grid unit Directional histogram; This represents the L2 norm.

[0013] In one possible implementation, perspective adaptive normalization includes: Divide the foreground motion features of the current frame into a spatial grid with the same structure as the scene scale map; For each grid cell, the orientation histogram is normalized using the following formula: ; in, Indicates the first Frame grid unit The normalized orientation histogram, also known as the corrected orientation histogram, constitutes the corrected motion characteristics based on the normalized orientation histograms of all grid cells.

[0014] It is understandable that motion signals that are suppressed by perspective effects in the depths of a scene are amplified by dividing by a smaller scale value, while motion signals in the foreground are appropriately compressed by dividing by a larger scale value. As a result, the characteristic responses of the same physical motion at different spatial locations are comparable, reducing the risk of missing abnormal motion in the distance.

[0015] In one possible implementation, local micro-motion features are extracted based on the corrected motion features, including: The corrected orientation histogram of each grid cell is used as a local micro-motion feature.

[0016] In one possible implementation, based on global macroscopic motion features, conditional density estimation is performed on local microscopic motion features to obtain motion anomaly scores, including: A conditional fractional network is constructed, which takes local micro-motion characteristics, global macro-motion characteristics and noise scale as inputs and outputs an estimate of the negative log-conditional density. The conditional score network was trained using normal training videos; During training, multi-scale Gaussian noise is applied to the local micro-motion features; The local micro motion features and global macro motion features of each spatial location in the current frame are input into the trained conditional score network, evaluated and aggregated under multiple preset noise scales, and the negative log-conditional density value of each spatial location is obtained. The motion anomaly score for the current frame is obtained using the following formula: ; in, Indicates the first The motion anomaly score of a frame is such that a higher value indicates a higher motion anomaly score. The more unusual the motion pattern of the frame; Indicates the first Frame grid unit The negative logarithmic conditional density value; This indicates taking the maximum value for all grid cells.

[0017] Understandably, a larger negative logarithmic conditional density value indicates that the local motion pattern is rarer in the current global dynamic context. The discrimination criterion is adaptively adjusted with the global motion state, maximum value aggregation ensures that the frame-level score is sensitive to the most significant local anomalies, and multi-scale evaluation aggregation improves the stability of density estimation.

[0018] In one possible implementation, the conditional score network is parameterized as a conditional energy function; the conditional score network is trained using normal training videos, including minimizing the following loss function: ; in, Represents the training loss function; Represents the trainable parameters of a conditional score network; This represents the expected value of sampling normal training samples and noise scales; The weighting function represents the variation of noise scale; This represents a conditional fractional network whose output is an estimate of the negative log-conditional density; This indicates the local microscopic motion characteristics after noise is added; This represents the local microscopic motion characteristics without added noise; Indicates the overall macroscopic motion characteristics; Indicates the standard deviation of noise; This represents the regularization coefficient.

[0019] Understandably, the first term of the loss function is the fractional matching loss, which constrains the network to approximate the gradient of the negative log-conditional density with respect to the gradient of the noisy micro-features. The second term is the regularization term, which suppresses the drift of the integral constant under different noise scales, making the multi-scale estimates numerically aligned. The gradient matching during the training phase and the direct reading of the function value during the inference phase are connected through energy function parameterization. The regularization term suppresses the drift of the network output, improving the stability and interpretability of the density estimation.

[0020] One possible implementation also includes: Extract the frame-level visual semantic features of the current frame; Density estimation is performed on frame-level visual semantic features to obtain semantic anomaly scores; Based on motion anomaly scores, anomaly detection results are obtained for the video to be detected, including: The semantic anomaly score and motion anomaly score are standardized and then weighted and fused to obtain the anomaly score of the current frame. Based on the anomaly score, obtain the anomaly detection results for the video to be detected.

[0021] Understandably, semantic-level anomalies and motion-level anomalies are processed in separate branches and then merged at the decision level, taking into account both types of anomalies and achieving more comprehensive anomaly detection.

[0022] It is worth noting that the two branches mentioned above are decoupled from each other and can be configured independently. In scenarios where computing resources are limited or only motion anomalies are of concern, the semantic stream can be turned off and detection can be completed solely by relying on the motion stream, resulting in high deployment flexibility.

[0023] Secondly, this application provides a frame-centered video anomaly detection system based on physical motion priors, comprising: The motion information acquisition module is used to calculate the dense optical flow between adjacent frames based on the video to be detected, and to acquire the motion information of the video to be detected. The foreground sensing filtering module is used to perform foreground sensing filtering on motion information, suppress background motion noise, and obtain foreground motion features. The feature extraction module is used to extract global macroscopic motion features based on foreground motion features and after statistical analysis. The perspective adaptive normalization module is used to perform perspective adaptive normalization based on the foreground motion features and the scene scale map to obtain the corrected motion features; wherein, the scene scale map represents the normal motion energy distribution at different spatial locations in the scene; The feature extraction module is also used to extract local micro motion features based on the corrected motion features; The conditional density estimation module is used to estimate the conditional density of local micro motion features based on global macro motion features, and obtain motion anomaly scores. The detection output module is used to obtain the anomaly detection results of the video to be detected based on the motion anomaly score.

[0024] It is understandable that the beneficial effects of the second aspect mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0025] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art: (1) Foreground sensing filtering achieves coarse localization of the foreground region by using the time median background model and morphological processing, filters out background motion noise, avoids incorrect motion statistics information encoded by scale map, and provides a clean foreground motion signal for conditional density estimation. The whole process does not depend on the target detector, overcoming the limitation of the target center method relying on predefined target categories.

[0026] (2) Perspective Adaptive Normalization constructs a scene scale map by statistically analyzing the average motion energy of each spatial location in normal data, performs grid-by-grid calibration on motion features, restores distant motion signals suppressed by perspective effects, makes the feature responses of the same physical motion at different spatial locations comparable, and reduces the risk of missing distant anomalies.

[0027] (3) Conditional density estimation establishes the anomaly detection of local motion patterns on the conditional dependence of the global motion state. By explicitly modeling the rationality of local motion in the context of global motion through conditional probability density, the judgment criteria can be adaptively adjusted according to global dynamics, thereby improving the accuracy of anomaly detection in complex motion scenarios.

[0028] (4) The three interconnected modules of foreground-aware filtering, perspective adaptive normalization and conditional density estimation constitute a multi-level motion modeling framework. The output of the previous level serves as the input of the next level. Starting from the original motion signal, it is refined step by step into clean, calibrated and context-aware motion features, providing a general three-level processing pipeline architecture for motion modeling problems in frame-centered methods.

[0029] (5) A dual-stream architecture is adopted to process visual semantics and physical motion in independent branches and integrate them at the decision layer, taking into account both semantic and motion anomalies to achieve comprehensive anomaly detection. The two branches are decoupled from each other and can be configured independently. In scenarios where computing resources are limited or only motion anomalies are concerned, the semantic stream can be turned off and detection can be completed by relying solely on the motion stream.

[0030] (6) This application improves the detection accuracy and robustness of frame-centered methods for complex motion anomalies without relying on target detectors. Attached Figure Description

[0031] Figure 1 This is a flowchart illustrating the frame-centered video anomaly detection method based on physical motion priors provided in this application embodiment; Figure 2 This is a schematic diagram of the foreground-aware filtering processing flow provided in an embodiment of this application; Figure 3 This is a visual schematic diagram of the scene scale map provided in the embodiments of this application; Figure 4 This is a schematic diagram illustrating the training and inference of the conditional density estimation model provided in the embodiments of this application; Figure 5 This is a schematic diagram of the frame-centered video anomaly detection system based on physical motion prior provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0032] To facilitate a clearer understanding of the various embodiments of this application, some relevant background knowledge will be introduced as follows.

[0033] Frame-centered video anomaly detection refers to a technical approach that models normal patterns and identifies anomalies directly on a frame-by-frame basis without relying on a target detector. Target-centered video anomaly detection refers to a technical approach that first uses a target detector to locate targets in the frame, and then performs appearance and trajectory analysis around each target. This application adopts the frame-centered technical approach.

[0034] Physical motion priors: Prior knowledge obtained from the physical imaging laws of the scene and normal motion statistics. In this application, these are specifically embodied as perspective projection geometry priors (carried by the scene scale map) and normal motion pattern distribution priors (carried by the conditional density estimation model).

[0035] Dense optical flow refers to the motion field describing the displacement vector of each pixel in a video between two adjacent frames; the orientation histogram is a histogram formed by dividing the optical flow direction into several directional intervals and statistically summing the optical flow amplitudes of pixels falling into each directional interval. The value of each interval (bin) reflects the cumulative amount of motion intensity in that direction.

[0036] Surveillance camera imaging follows the laws of perspective projection: the farther a target is from the camera, the smaller its image size in the image, and the smaller the optical flow amplitude corresponding to the same physical displacement in the image. Therefore, the optical flow signal generated by motion deep in the scene is numerically suppressed, which is the motion scale distortion caused by perspective effect.

[0037] Temporal median background modeling is a background modeling method that takes the median value in the time direction of each pixel position in a video frame sequence as the background value of that pixel. It is a non-parametric method that does not require preset distribution assumptions and is robust to occasional foreground targets in the training data.

[0038] Density estimation based on multi-scale denoising and score matching refers to: applying Gaussian noise at multiple scales to the data samples, training the network so that the gradient (i.e., the score) of the network output approximates the log density gradient of the noisy distribution, thereby indirectly obtaining the density estimate of the data distribution; when the network is parameterized as an energy function (i.e., the network output directly represents the unnormalized estimate of the negative log density), this training method is connected with the training of the energy model, and the function value of the network can be directly used for density evaluation.

[0039] This application analyzes existing frame-center methods and finds the following defects.

[0040] Defect 1: Existing frame-centered methods generally input the full-image optical flow, which includes background noise, directly into the subsequent motion model. Background dynamics in the monitored scene (such as swaying leaves, water ripples, etc.) will generate false optical flow, which will be superimposed on the foreground motion signal and seriously interfere with anomaly detection.

[0041] Defect 2: Existing frame-centered methods use a uniform scale across the entire image to process motion features. Due to the perspective effect, motion signals deep within the scene are numerically suppressed. The feature responses of the same physical motion at different spatial locations are not comparable, and abnormal motions located at a distance are easily missed.

[0042] Thirdly, existing frame-centered methods often handle local motion patterns by feature concatenation or independent modeling, resulting in fixed criteria for judging local motion that cannot adaptively adjust to the global motion state. For example, the same local motion pattern should be subject to different validity criteria during peak hours and off-peak hours. Furthermore, target-centered methods rely on predefined target categories, limiting their generalization ability to target types not encountered during training.

[0043] The above defects are summarized after attributing the false detections and false negatives of the existing frame-centered method in typical monitoring scenarios one by one: false detections caused by background dynamics can be attributed to defect one, false negatives of distant moving targets can be attributed to defect two, and false positives caused by changes in the global context can be attributed to defect three.

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0046] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0047] The embodiments of this application are described below with reference to the accompanying drawings.

[0048] Example 1.

[0049] This embodiment provides a frame-center-based video anomaly detection method based on physical motion priors (hereinafter referred to as this detection method), such as Figure 1As shown, the video to be detected sequentially undergoes motion information acquisition (dense optical flow extraction), foreground-aware filtering, global macroscopic motion feature extraction, perspective adaptive normalization, local microscopic motion feature extraction, and conditional density estimation to obtain a motion anomaly score. Simultaneously, the video to be detected undergoes semantic feature extraction and semantic density estimation to obtain a semantic anomaly score. The two anomaly scores are fused at the decision layer (score fusion and anomaly determination) to output the final anomaly detection result. This embodiment targets application scenarios with fixed camera perspectives, covering mainstream video anomaly detection deployment environments such as urban security, park monitoring, and campus security.

[0050] Step 1: Obtain motion information from the video to be detected.

[0051] Motion information is obtained by calculating the dense optical flow between adjacent frames of a video. A pre-trained optical flow estimation network is used to extract the dense optical flow field of each frame of the video to be detected. The value of each pixel in the optical flow field represents the displacement vector of that pixel between two consecutive frames. The optical flow estimation network can use FlowNet2, RAFT, or other neural network structures capable of outputting pixel-level dense optical flow fields. In this embodiment, the FlowNet2 network is used to extract optical flow information.

[0052] Let the first The dense optical flow field of the frame is ,in and These represent the frame's height and width, respectively, with the last dimension corresponding to the optical flow components in the horizontal and vertical directions.

[0053] Step 2: Perform foreground-sensing filtering on the motion information to suppress background motion noise and obtain foreground motion features.

[0054] Existing frame centering methods directly input the optical flow of the entire image, including background noise, into the subsequent motion model. This results in aliasing of spurious optical flow generated by background dynamics (such as swaying leaves or ripples on water) with the foreground motion signal, severely interfering with anomaly detection. This step sets up a foreground-aware filtering module at the very beginning of the motion processing chain, such as... Figure 2 As shown, the static structural prior of the scene is used to achieve coarse separation of foreground and background motion information, providing denoised input for subsequent perspective adaptive normalization and conditional density estimation. Specifically, this includes the following sub-steps.

[0055] (1) Constructing a static background image. Uniformly sample a portion of frames from the normal training video (sampling ratio, for example, 10%~20%), and take the median time value for each pixel location as the background grayscale value for that pixel. All pixels constitute the static background image. Temporal median filtering, as a non-parametric background modeling method, requires no pre-defined distribution assumptions and is naturally robust to sporadic foreground objects in the training data. Optionally, in long-term deployment scenarios, the background image is reconstructed periodically according to a sliding time window: at preset intervals (e.g., 1 hour), the temporal median of each pixel is recalculated using the video frames within the most recent sliding window to update the background image. This allows for tracking of lighting changes and changes in still objects in the scene, preventing large-scale misjudgments of the foreground mask due to outdated background models.

[0056] (2) Foreground mask generation. For the current frame Calculate its relationship with the background image difference image Thresholding is performed on the difference image to obtain the initial foreground mask. ,in This is a preset threshold.

[0057] (3) Post-mask processing. Morphological processing is performed on the initial foreground mask: first, a closing operation (dilation followed by erosion) is performed to fill small holes inside the foreground region; then, a dilation operation is performed to expand the boundary of the foreground region so that the mask can completely cover the edge region of the moving target; finally, connected regions with an area smaller than a preset area threshold are removed to eliminate isolated noise points. After the above processing, an optimized foreground mask is obtained. .

[0058] (4) Motion information cropping. Utilizing an optimized foreground mask. Optical flow field of the current frame Perform pixel-by-pixel cropping: ,in This indicates element-wise multiplication. After cropping, only the foreground region is retained. Optical flow information of the background area ( ) The optical flow is set to zero to obtain the foreground motion characteristics. .

[0059] The key role of this filtering module is that the subsequent perspective adaptive normalization module needs to statistically analyze the normal motion energy of each spatial location to construct a scale map. If the input contains background noise, the scale map will encode incorrect motion statistics, causing normalization to fail. The subsequent conditional density estimation module also needs to be based on clean foreground motion signals to accurately model local-global motion dependencies.

[0060] Step 3: Based on the foreground motion features after foreground-aware filtering, extract the global macroscopic motion features after statistical analysis.

[0061] Global macroscopic motion characteristics : Create the orientation histogram of all pixels in the current frame Element-wise summation yields the frame-level orientation histogram, and the result is denoted as... Its dimensions are Dimensionality (16 dimensions in this embodiment). Global macroscopic motion features are a coarse-grained summary of the motion state of the entire scene, representing the relative distribution of motion directions in the foreground region and reflecting the overall dynamic context of the current frame. Step 4: Perform perspective adaptive normalization on the foreground motion features based on the scene scale map to obtain the corrected motion features.

[0062] This step generates a scene scale map during system initialization, and then uses this scale map to perform spatial adaptive normalization of motion features during the inference phase. Scene Scale Map The generation is completed during the system initialization phase. and (This refers to the number of rows and columns of the spatial grid), and its visualization effect is as follows: Figure 3 As shown, the spatial distribution of scale values ​​in three different monitoring scenarios of the ShanghaiTech dataset is displayed in the form of a heatmap. The specific process is as follows: First, the foreground optical flow of each frame in the normal training video is obtained using the method in step two. , ( (Total number of training frames). Then, the foreground optical flow of each frame is divided into... There are 3 spatial grids, each with a size of 1. Pixels. Next, for each grid cell... The optical flow vectors of all the pixels contained therein are statistically analyzed into a direction histogram. ,in The number of microscopic directional intervals (in this embodiment) The histogram is calculated by summing the optical flow amplitudes of all pixels whose optical flow direction falls within each bin. The L2 norm of the histogram is then calculated. As this grid in the frame The motion energy of each grid cell is calculated. Finally, the average motion energy of each grid cell across all normal training frames is calculated: ; Parameter explanation: : Represents a grid cell The scale value, i.e., the average motion energy of the grid across all normal training frames; : Indicates the total number of training frames; : Indicates the training frame number; : indicates the first Frame grid unit Microscopic orientation histogram; : Indicates the L2 norm, which here represents the grid's position in the frame. The kinetic energy.

[0063] For empty grids that never exhibit foreground motion during training ( Fill with the global average scale value and set a lower threshold. (For example =0.1 pixels / frame), and the final scale value of each grid is taken as... This is to prevent noise from being amplified during subsequent normalization operations due to division by zero or by excessively small values.

[0064] The scale values ​​of all grids constitute the scene scale map. This scale map reflects the average motion intensity of each spatial location in the scene under normal conditions. It essentially encodes the geometric prior of perspective projection, i.e., the scale value of areas deep in the scene (far from the camera) is small, and the scale value of areas closer to the camera is large.

[0065] The specific operation of perspective adaptive normalization is as follows: the foreground optical flow of the current frame is normalized. Divided into the same structure A spatial grid, for each grid cell Extract its orientation histogram Then, perform element-wise division using the scale value at the corresponding position in the scale map: ; Parameter explanation: : indicates the first Frame grid unit The orientation histogram after perspective adaptive normalization, also known as the corrected orientation histogram; : indicates the first Frame grid unit Orientation histogram before normalization; : Represents a grid cell in a scene-scale diagram The corresponding scale value.

[0066] The normalized histograms of all grids constitute the corrected motion characteristics. .

[0067] Through the above normalization, the motion signals in the depths of the scene, which were originally suppressed by the perspective effect, are amplified, while the motion signals in the foreground are appropriately compressed, making the characteristic responses of the same physical motion at different spatial locations comparable.

[0068] Step 5: Extract local micro-motion features from the corrected motion features.

[0069] Local micro motion characteristics : that is, the orientation histogram after correction of each grid. Each histogram is a A dimensional vector represents the fine distribution of motion directions within the grid region. The microscopic features of all grids constitute a set. .

[0070] Step 6: Based on the global macroscopic motion characteristics, perform conditional density estimation on the local microscopic motion characteristics to obtain the motion anomaly score.

[0071] This step models anomaly detection as a conditional density estimation problem, that is, estimating the conditional density given global macroscopic motion characteristics. Under these conditions, local micro-motion characteristics probability density of occurrence The lower the probability density, the rarer the local motion pattern is in the current global dynamic context, and the more likely it is to be an anomaly. The training and inference process of the conditional density estimation model is as follows: Figure 4 As shown: During the training phase, multi-scale Gaussian noise is applied to the local micro-motion features. The noise-added micro-features, the clean (i.e., un-noised) global macro-motion features, and the noise scale are input into the conditional score network. The network is trained using score matching loss and regularization terms. During the inference phase, the micro-features of each grid and the global macro-features are input into the trained network. They are evaluated and aggregated under multiple preset noise scales to obtain the negative logarithmic conditional density values ​​of each grid. The maximum values ​​are then aggregated to obtain the motion anomaly score.

[0072] (1) Model construction. Construct a conditional score network. The network is parameterized as a conditional energy function, which features noisy local micro-motion characteristics. Global macroscopic motion characteristics and noise scale The input is the negative log-conditional density given the conditions. The estimated value; the larger the output value, the rarer the local micro-motion pattern is in the current global macro-motion context.

[0073] (2) Model Training. The conditional score network is trained using normal training videos. During training, multi-scale Gaussian noise is applied to the local micro-motion features: The noise standard deviation In the interval Uniform sampling within (e.g.) =0.01, =1.0). Global macroscopic motion characteristics Keep the environment clean and free of noise. The training objective is to minimize the following loss function: ; Parameter explanation: : Represents the training loss function; : Represents the trainable parameters of the conditional score network; : Indicates the expectation of sampling normal training samples and noise scale; : Represents the weighting function that varies with the noise scale; : Represents a conditional fractional network, whose output is an estimate of the negative log-conditional density; : Represents the local microscopic motion characteristics after noise is added; : Represents local microscopic motion characteristics without noise; : Represents global macroscopic motion characteristics, kept clean during training; : Indicates the standard deviation of noise; : Represents the regularization coefficient.

[0074] The first term is the fractional matching loss, and the second term is the regularization term. The fractional matching loss makes the gradient of the conditional fractional network approximate the gradient of the negative log-conditional density with respect to the noisy micro-features. The regularization term suppresses the drift of the integral constant under different noise scales, making the multi-scale estimates numerically aligned. Since the conditional fractional network is parameterized as a conditional energy function, its function value (not just its gradient) can be directly used as an estimate of the negative log-conditional density. Thus, the gradient matching in the training phase and the direct reading of the function value in the inference phase are interconnected. After training, the network parameters are fixed. We extract the multi-scale response vectors of all normal samples in the training set and fit a Gaussian mixture model as a reference distribution of the normal multi-scale response pattern.

[0075] (3) Reasoning and anomaly scoring. During reasoning, the normalized micro-features of all grids in the current frame are used. and global macroscopic features Input the trained conditional score network, in The negative log-conditional density (NCD) of each grid is evaluated at each preset noise scale, forming a multi-scale estimation set for that grid. This multi-scale estimation set is then input into a fitted Gaussian mixture model to calculate its probability density value under that model, which is taken as the final NCD of the grid, denoted as [missing value]. ; The larger the size, the larger the grid. The rarer a local motion pattern is in the current global dynamic context, the more likely it is to be an anomaly.

[0076] Then, a maximum value aggregation strategy is adopted, that is, the grid with the largest negative log-conditional density value among all grids is taken as the motion anomaly score of the current frame: ; Parameter explanation: : indicates the first The motion anomaly score of a frame; the higher the value, the more abnormal the motion pattern of that frame. : indicates taking the maximum value for all grid cells; : indicates the first Frame grid unit The negative logarithmic conditional density value.

[0077] Step 7: Merge the semantic anomaly score and the motion anomaly score and make an anomaly determination to obtain the final anomaly detection result.

[0078] To detect semantic-level anomalies simultaneously, this detection method employs parallel semantic streams: (1) Extracting visual semantic features. A pre-trained visual Transformer encoder (such as Hiera-L) is used to extract frame-level semantic features: the current frame is scaled to the encoder's input size and then input into the encoder. The global pooled feature vector output from its last layer is taken as the frame-level semantic feature of the current frame, denoted as... Its dimension is denoted as .

[0079] (2) Semantic density estimation. Construct a semantic energy network. It takes frame-level semantic features and noise scale as input and outputs the negative log density estimate of the frame-level semantic features. During training, multi-scale Gaussian noise (the noise scale sampling method is consistent with the motion flow) is applied to the frame-level semantic features of all frames in the normal training video, and the semantic energy network is trained using the same multi-scale denoising fractional matching loss (excluding conditional input) as the motion flow. During inference, The trained semantic energy network is input, evaluated and aggregated under multiple preset noise scales, and the semantic anomaly score is obtained. ; The larger the value, the more abnormal the semantic pattern of the frame, and its score direction is consistent with the motion anomaly score.

[0080] (3) Score Fusion. The outlier scores of the two streams are sequentially subjected to Z-score standardization and Min-Max normalization: first, Z-score standardization is performed using the mean and standard deviation of the outlier scores of each stream on the normal training set; then, Min-Max normalization is performed using the minimum and maximum values ​​of the outlier scores of each stream on the same dataset, mapped to... The mean, standard deviation, minimum, and maximum values ​​mentioned above are pre-estimated and fixed during system initialization. During online detection, the fixed statistics are used to perform transformations frame-by-frame, eliminating the need to wait for the entire sequence to be processed, thus supporting real-time detection. Weighted summation is then applied after normalization. ; Parameter explanation: : indicates the first Anomaly score after frame fusion; : Represents the weighting coefficient of the semantic anomaly score; : Represents the semantic anomaly score after Z-score standardization and Min-Max normalization; : Represents the weighting coefficient of the motion abnormality score; : Represents the motion anomaly score after Z-score standardization and Min-Max normalization.

[0081] in and As a preset weighting coefficient, in this embodiment, we take... = =1.0.

[0082] Finally, based on the anomaly scores of each frame... The video frames are identified as abnormal, and frames with scores exceeding a preset threshold are marked as abnormal frames.

[0083] Example 2.

[0084] This embodiment provides a variant solution based on the aforementioned Embodiment 1. The difference from Embodiment 1 is that this embodiment does not process the visual semantic stream, but relies solely on the motion stream to complete video anomaly detection. This solution is suitable for deployment scenarios with limited computing resources or where only motion anomalies are a concern.

[0085] In Example 2, steps one through six are exactly the same as in Example 1. The difference is that the semantic flow part of step seven in Example 1 is omitted, and the motion anomaly score calculated in step six is ​​used directly. As the anomaly score for the current frame: The higher the score, the more the motion pattern of the frame deviates from the normal pattern, and the more likely it is to be an abnormal frame.

[0086] The following describes the frame-center video anomaly detection system based on physical motion priors provided in this application. The frame-center video anomaly detection system based on physical motion priors described below can be referred to in correspondence with the frame-center video anomaly detection method based on physical motion priors described above.

[0087] like Figure 5 As shown, the system includes: a motion information acquisition module 10, a foreground perception filtering module 20, a perspective adaptive normalization module 30, a feature extraction module 40, a conditional density estimation module 50, and a detection output module 60. Among them: The motion information acquisition module 10 is used to calculate the dense optical flow between adjacent frames based on the video to be detected, and to acquire the motion information of the video to be detected. The foreground sensing filtering module 20 is used to perform foreground sensing filtering on motion information, suppress background motion noise, and obtain foreground motion features. Feature extraction module 40 is used to extract global macroscopic motion features based on foreground motion features and after statistical analysis. The perspective adaptive normalization module 30 is used to perform perspective adaptive normalization based on the foreground motion features and the scene scale map to obtain the corrected motion features; wherein, the scene scale map represents the normal motion energy distribution at different spatial locations in the scene; The feature extraction module 40 is also used to extract local micro motion features based on the corrected motion features; The conditional density estimation module 50 is used to estimate the conditional density of local micro motion features based on global macro motion features, and obtain motion anomaly scores. The detection output module 60 is used to obtain the anomaly detection results of the video to be detected based on the motion anomaly score.

[0088] It is understood that the detailed functional implementation of each of the above units / modules can be found in the description in the aforementioned method embodiments, and will not be repeated here.

[0089] It should be understood that the above system is used to execute the methods in the above embodiments. The corresponding program modules in the system are similar in implementation principle and technical effect to those described in the above methods. The working process of the system can be referred to the corresponding process in the above methods, and will not be repeated here.

[0090] Based on the methods in the above embodiments, this application provides an electronic device, such as... Figure 6As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the methods in the above embodiments.

[0091] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0092] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0093] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0094] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0095] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0096] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0097] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0098] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A frame-centered video anomaly detection method based on physical motion priors, characterized in that, include: Based on the video to be detected, the dense optical flow between adjacent frames is calculated to obtain the motion information of the video to be detected. Foreground-sensing filtering is applied to motion information to suppress background motion noise and obtain foreground motion features; Based on foreground motion features, global macroscopic motion features are extracted after statistical analysis. Based on foreground motion features, perspective adaptive normalization is performed using scene scale maps to obtain corrected motion features; where the scene scale map represents the normal motion energy distribution at different spatial locations in the scene. Based on the corrected motion features, local micro-motion features are extracted; Based on global macroscopic motion characteristics, conditional density estimation is performed on local microscopic motion characteristics to obtain motion anomaly scores; Based on motion anomaly scores, the anomaly detection results of the video to be detected are obtained.

2. The frame-centered video anomaly detection method based on physical motion priors according to claim 1, characterized in that, The process of performing foreground-sensing filtering on motion information to suppress background motion noise and obtain foreground motion features includes: Based on the normal training video frame sequence, the median time value of each pixel position is taken to generate a static background image; The current frame is compared with the static background image and then thresholded to obtain the initial foreground mask; Morphological processing is performed on the initial foreground mask to obtain an optimized foreground mask; Motion information is cropped by optimizing the foreground mask, preserving the motion information of the foreground region, and obtaining the foreground motion features.

3. The frame-centered video anomaly detection method based on physical motion priors according to claim 1, characterized in that, The scene scale map is generated through the following steps: The foreground motion features of each frame in the normal training video are divided into a preset number of spatial grids. For each grid cell, the optical flow vector of the pixels in the grid cell is statistically converted into a direction histogram. The scale value of each grid cell is obtained using the following formula, and the scale values ​​of all grid cells constitute the scene scale map: ； in, Represents grid cells The scale value, i.e., the average motion energy of the grid cell across all normal training frames; Indicates the total number of training frames; Indicates the first Frame grid unit Directional histogram; This represents the L2 norm.

4. The frame-centered video anomaly detection method based on physical motion priors according to claim 3, characterized in that, The perspective adaptive normalization includes: Divide the foreground motion features of the current frame into a spatial grid with the same structure as the scene scale map; For each grid cell, the orientation histogram is normalized using the following formula: ； in, Indicates the first Frame grid unit The normalized orientation histogram, also known as the corrected orientation histogram, constitutes the corrected motion characteristics based on the normalized orientation histograms of all grid cells.

5. The frame-centered video anomaly detection method based on physical motion priors according to claim 4, characterized in that, The extraction of local micro-motion features based on the corrected motion features includes: The corrected orientation histogram of each grid cell is used as a local micro-motion feature.

6. The frame-centered video anomaly detection method based on physical motion priors according to claim 1, characterized in that, The process of estimating the conditional density of local micro-motion features based on global macro-motion characteristics to obtain motion anomaly scores includes: A conditional fractional network is constructed, which takes local micro-motion characteristics, global macro-motion characteristics and noise scale as inputs and outputs an estimate of the negative log-conditional density. The conditional score network was trained using normal training videos; During training, multi-scale Gaussian noise is applied to the local micro-motion features; The local micro motion features and global macro motion features of each spatial location in the current frame are input into the trained conditional score network, evaluated and aggregated under multiple preset noise scales, and the negative log-conditional density value of each spatial location is obtained. The motion anomaly score for the current frame is obtained using the following formula: ； in, Indicates the first The motion anomaly score of a frame, the higher the value, the more significant the motion anomaly score. The more unusual the motion pattern of the frame; Indicates the first Frame grid unit The negative logarithmic conditional density value; This indicates taking the maximum value for all grid cells.

7. The frame-centered video anomaly detection method based on physical motion priors according to claim 6, characterized in that, The conditional score network is parameterized as a conditional energy function; training the conditional score network using normal training videos includes minimizing the following loss function: ； in, Represents the training loss function; Represents the trainable parameters of a conditional score network; This represents the expected value of sampling normal training samples and noise scales; The weighting function represents the variation of noise scale; This represents a conditional fractional network whose output is an estimate of the negative log-conditional density; This indicates the local microscopic motion characteristics after noise is added; This represents the local microscopic motion characteristics without added noise; Indicates the overall macroscopic motion characteristics; Indicates the standard deviation of noise; This represents the regularization coefficient.

8. The frame-centered video anomaly detection method based on physical motion priors according to claim 1, characterized in that, The process of obtaining anomaly detection results for the video under test based on motion anomaly scores includes: The semantic anomaly score and motion anomaly score are standardized and then weighted and fused to obtain the anomaly score of the current frame. Based on the anomaly score, obtain the anomaly detection results for the video to be detected.

9. A frame-centered video anomaly detection system based on physical motion priors, characterized in that, include: The motion information acquisition module is used to calculate the dense optical flow between adjacent frames based on the video to be detected, and to acquire the motion information of the video to be detected. The foreground sensing filtering module is used to perform foreground sensing filtering on motion information, suppress background motion noise, and obtain foreground motion features. The feature extraction module is used to extract global macroscopic motion features based on foreground motion features and after statistical analysis. The perspective adaptive normalization module is used to perform perspective adaptive normalization based on the foreground motion features and the scene scale map to obtain the corrected motion features; wherein, the scene scale map represents the normal motion energy distribution at different spatial locations in the scene; The feature extraction module is also used to extract local micro motion features based on the corrected motion features; The conditional density estimation module is used to estimate the conditional density of local micro motion features based on global macro motion features, and obtain motion anomaly scores. The detection output module is used to obtain the anomaly detection results of the video to be detected based on the motion anomaly score.

Images