A scene classification-based video anomaly detection method, system and device

Abstract

The present application belongs to the field of computer vision, and particularly relates to a video anomaly detection method, system and device based on scene classification. The steps of the present application method are as follows: step S1, performing gray processing on original video frames, using background subtraction method and frame difference method to extract foreground targets, and extracting pixel features through Double-Canny algorithm; step S2, using a pre-trained model to detect the number of people in the video frames and generate a heat map of the video frames; step S3, using K-means clustering algorithm to classify scenes of the video frames, and dividing the video frames into two categories of dense scenes and sparse scenes; and step S4, using an anomaly detection module to respectively perform anomaly detection on the dense scenes and the sparse scenes. The present application can select different network structures for anomaly detection according to scenes with different crowd densities, is better applied to complex real scenes, and achieves better anomaly behavior detection effect.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a video anomaly detection method, system, and device based on scene classification. Background Technology

[0002] In recent years, numerous researchers in the fields of computer vision and artificial intelligence have been dedicated to the automatic understanding and analysis of scenes in images, with crowd analysis in computer vision becoming a research hotspot. To reduce loss of life and property caused by unexpected anomalies, timely alerts can be sent by automatically identifying anomalous events in video streams. Video anomaly detection refers to identifying unexpected motion or appearance patterns. Due to the diversity and unpredictability of anomalous events, video anomaly detection has become an extremely challenging task.

[0003] In real-world applications, various crowd densities exist. Analysis of current common video-based abnormal behavior algorithms reveals that most lack the ability to differentiate between different crowd densities, as the characteristics of people at different densities vary significantly. Even with substantial differences in crowd distribution, most algorithms use the same framework to identify abnormal behavior. When videos involve more complex scenes, such as local clustering or varying densities within the same scene, this not only greatly increases the computational complexity of the system but may even lead to the failure of abnormal behavior detection in intelligent monitoring systems. Furthermore, due to the unpredictability of pedestrian movement, it is difficult to develop adaptive detection methods and to effectively determine the density of crowd distribution in advance.

[0004] To represent crowd distribution, crowd counts are typically combined with crowd entropy, and optical flow is calculated to highlight the movement of the entire crowd, thus solving the anomaly detection problem based on overall crowd movement. However, crowd density is variable, and analyzing crowd density solely from a global perspective has the drawback of insufficient information extraction. Therefore, targeted designs are needed to detect anomalous behavior in unevenly distributed crowds in real-world scenarios. Based on the number of people and the presence of occlusion, detection scenarios are divided into sparse and dense scenarios, and then appropriate methods are selected for anomaly detection in different scenarios. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, the present invention provides a video anomaly detection method, system and device based on scene classification. This detection method is adaptable to application scenarios with different crowd densities and when the crowd is in motion.

[0006] This invention is achieved through the following measures:

[0007] A video anomaly detection method based on scene classification mainly includes the following steps:

[0008] Step S1: Convert the original video frame to grayscale, select a median filter for smoothing, use background subtraction and frame difference to extract the foreground target, and extract pixel features using the Double-Canny algorithm.

[0009] Step S2: Use a pre-trained model to detect the number of people in the video frames and generate a heatmap of the video frames;

[0010] Step S3: Use the K-means clustering algorithm to classify video frames into two categories: dense scenes and sparse scenes.

[0011] Step S4: Use the anomaly detection module to perform anomaly detection on dense and sparse scenes respectively. The anomaly detection module is a dual-branch network structure based on GAN.

[0012] As a further preferred embodiment of the method of the present invention, step S1 specifically includes:

[0013] Step S1.1: Apply the Canny operator to edge detection for both the foreground frame and the original image to obtain the foreground image and the overall edge image;

[0014] Step S1.2: Perform a pixel-by-pixel "sum" operation on the overall edge image and foreground image from step S1.1 to remove irrelevant edges and capture missing information in the foreground to obtain the processed foreground image;

[0015] Step S1.3: Perform an OR operation pixel by pixel on the foreground image in step S1.1 and the processed foreground image in step S1.2 to obtain the final foreground image;

[0016] Step S1.4: Calculate the number of pixels in the binary image of the final foreground image from step S1.3 to obtain the number of foreground pixels N. f Perform Canny operator edge detection on the final foreground image to obtain the final edge pixel image, and calculate the number of edge pixels N. e ; Calculate the ratio of edge pixels to foreground pixels. r indicates the degree of crowd obstruction.

[0017] As a further preferred embodiment of the method of the present invention, the pre-trained model in step S2 uses YOLOv4 to detect the number of people, and generates a heatmap of the image through the pre-trained model, and calculates the proportion p of the highlighted part of the heatmap. i p i It indicates the degree of crowd gathering.

[0018] As a further preferred embodiment of the method of the present invention, step S3 uses the three pixel features obtained in step S1 and the number of people and the highlight ratio in the heat map obtained in step S2 as feature vectors. Where χ i It is the feature vector of the i-th frame, Num i It represents the number of people in the i-th frame. and These are the number of edge pixels and the number of foreground pixels in the i-th frame, respectively, r i yes and The ratio, p i This represents the proportion of the highlighted portion in the heatmap of the i-th frame; the feature vector is used as the input value, and the K-means clustering algorithm automatically clusters the scenes in the video into two categories: sparse and dense.

[0019] As a further preferred embodiment of the method of the present invention, in step S4, the anomaly detection module adopts a prediction model in sparse scenes and uses U-Net as a generator to predict the next frame; in dense scenes, it applies a reconstruction model, wherein the generator is an autoencoder with no skip connections used to reconstruct video frames.

[0020] As a further preferred embodiment of the method of the present invention, intensity loss, gradient loss, and optical flow loss are introduced as constraints into the anomaly detection model; intensity loss L int By comparing the value of each pixel between the predicted frame and the original frame, as shown in Equation (1):

[0021]

[0022] in, I represents the predicted frame, and I represents the original frame;

[0023] Gradient loss L gd It is obtained by comparing the gradient of pixel values ​​of the predicted frame and the original frame, as shown in Equation (2):

[0024]

[0025] in, I represents the spatial index of the predicted frame. i,j Indicates the spatial index of the original frame;

[0026] Optical flow loss L op The calculation is shown in equation (3):

[0027]

[0028] in, I represents the predicted t-th frame. t This represents the original (t+1)th frame. It represents optical flow.

[0029] As a further preferred embodiment of the method of the present invention, the prediction model in sparse scenes uses least squares GAN (LSGAN) to predict the next frame, and uses a generator G p Image spoofing detector D p In the GAN architecture, D is defined. p The losses of the confrontation As shown in equation (4), train G p When D is fixed p The weights, defined by G p The losses of the confrontation As shown in equation (5):

[0030]

[0031]

[0032] in, I represents the predicted frame, and I represents the original frame.

[0033] Combining intensity loss, gradient loss, and optical flow loss, we obtain G. p objective function As shown in equation (6), D p objective function As shown in equation (7):

[0034]

[0035]

[0036] Where, λ int The weighting coefficient λ represents the intensity loss. gd λ represents the weight coefficients of the gradient loss. op The weighting coefficient λ represents the optical flow loss. adv This represents the weighting coefficients for adversarial training loss.

[0037] As a further preferred embodiment of the method of the present invention, in dense scenes, the generator G r Reconstructing the network loss function As shown in equation (8); the discriminator of the reconstructed model is D. r The loss function of its adversarial training As shown in equation (9):

[0038]

[0039]

[0040] By combining intensity loss, gradient loss, and optical flow loss, the objective function of the reconstruction model is obtained. As shown in equation (10):

[0041]

[0042] Where, λ int The weighting coefficient λ represents the intensity loss. gd λ represents the weight coefficients of the gradient loss. op The weighting coefficient λ represents the optical flow loss. adv This represents the weighting coefficients for adversarial training loss.

[0043] Furthermore, this invention proposes a video anomaly detection system based on scene classification, comprising:

[0044] The preprocessing unit performs grayscale processing on the original video frames, selects a median filter for smoothing, uses background subtraction and frame difference methods to extract foreground targets, and extracts pixel features through the Double-Canny algorithm.

[0045] The people detection unit uses a pre-trained model to detect the number of people in video frames and generate a heatmap of the video frames.

[0046] The scene classification unit uses the K-means clustering algorithm to classify video frames into two categories: dense scenes and sparse scenes.

[0047] An anomaly detection unit uses an anomaly detection module to perform anomaly detection on dense and sparse scenes respectively, wherein the anomaly detection module is a GAN-based dual-branch network structure.

[0048] The present invention also proposes an electronic device comprising a memory, a processor, and program instructions stored in the memory that are executable by the processor, the processor executing the program instructions to implement the steps of the method described in the present invention.

[0049] Compared with the prior art, the beneficial effects of the present invention are:

[0050] 1) This invention proposes a scene classification method that combines Double-Canny and YoloV4, and uses the K-means clustering algorithm to adaptively divide the scene into sparse and dense categories. Compared with the current common video-based abnormal behavior detection methods, it can distinguish different crowd densities and improve the accuracy of abnormal behavior detection.

[0051] 2) Considering the significant differences in characteristics between high-density and low-density populations, this invention selects different network structures for anomaly detection in different scenarios, better adapting to situations such as localized focusing and varying density within the same scenario in real-world settings, thereby optimizing the results of abnormal behavior detection. Attached Figure Description

[0052] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0053] Figure 1 This is a flowchart illustrating the overall process of the video anomaly detection method based on scene classification according to the present invention.

[0054] Figure 2 Edge pixel curves and ratio curves for four video examples provided in embodiments of the present invention;

[0055] Figure 3 The results of pedestrian detection and heatmap visualization in sparse and dense scenes provided in the embodiments of the present invention;

[0056] Figure 4 The classification visualization results of the three dataset test sets provided in the embodiments of the present invention;

[0057] Figure 5 A prediction network block diagram provided in an embodiment of the present invention;

[0058] Figure 6 The reconstructed network block diagram provided in the embodiments of the present invention;

[0059] Figure 7 The ROC of this model and the classic method are compared on three datasets provided in the embodiments of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Of course, the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0061] like Figure 1 As shown, this invention provides a video anomaly detection method based on scene classification, which mainly includes the following:

[0062] Step S1: Convert the original video frame to grayscale, select a median filter for smoothing, use background subtraction and frame difference to extract the foreground target, and extract pixel features using the Double-Canny algorithm.

[0063] Step S2: Use a pre-trained model to detect the number of people in the video frames and generate a heatmap of the video frames;

[0064] Step S3: Use the K-means clustering algorithm to classify video frames into two categories: dense scenes and sparse scenes.

[0065] Step S4: Use the anomaly detection module to perform anomaly detection on dense and sparse scenes respectively. The anomaly detection module is a dual-branch network structure based on GAN.

[0066] Step S1 specifically includes:

[0067] Step S1.1: Apply the Canny operator to edge detection for both the foreground frame and the original image to obtain the foreground image and the overall edge image;

[0068] Step S1.2: Perform a pixel-by-pixel "sum" operation on the overall edge image and foreground image from step S1.1 to remove irrelevant edges and capture missing information in the foreground to obtain the processed foreground image;

[0069] Step S1.3: Perform an OR operation pixel by pixel on the foreground image in step S1.1 and the processed foreground image in step S1.2 to obtain the final foreground image;

[0070] Step S1.4: Calculate the number of pixels in the binary image of the final foreground image from step S1.3 to obtain the number of foreground pixels N. f Perform Canny operator edge detection on the final foreground image to obtain the final edge pixel image, and calculate the number of edge pixels N. e ; Calculate the ratio of edge pixels to foreground pixels. r indicates the degree of crowd obstruction.

[0071] In step S2, the pre-trained model uses YOLOv4 to detect the number of people and generates a heatmap of the image, calculating the proportion p of the highlighted parts of the heatmap. i p i It indicates the degree of crowd gathering.

[0072] In step S3, the three pixel features obtained in step S1 and the number of people and the highlight ratio in the heatmap obtained in step S2 are used as feature vectors. Where χ i It is the feature vector of the i-th frame, Num i It represents the number of people in the i-th frame. and These are the number of edge pixels and the number of foreground pixels in the i-th frame, respectively, r i yes and The ratio, p i This represents the proportion of the highlighted portion in the heatmap of the i-th frame; the feature vector is used as the input value, and the K-means clustering algorithm automatically clusters the scenes in the video into two categories: sparse and dense.

[0073] In step S4, the anomaly detection module employs a prediction model in sparse scenes and uses U-Net as a generator to predict the next frame, where the convolutional and deconvolutional layers of U-Net have a kernel size of 3×3 and the max pooling layer kernel size is 2×2; in dense scenes, a reconstruction model is applied, where the generator is an autoencoder with no skip connections used to reconstruct video frames.

[0074] In the anomaly detection model, intensity loss, gradient loss, and optical flow loss are introduced as constraints; intensity loss L int By comparing the value of each pixel between the predicted frame and the original frame, as shown in Equation (1):

[0075]

[0076] in, I represents the predicted frame, and I represents the original frame;

[0077] Gradient loss L gd It is obtained by comparing the gradient of pixel values ​​of the predicted frame and the original frame, as shown in Equation (2):

[0078]

[0079] in, I represents the spatial index of the predicted frame. i,j Indicates the spatial index of the original frame;

[0080] Optical flow loss L op The calculation is shown in equation (3):

[0081]

[0082] in, I represents the predicted t-th frame. t This represents the original (t+1)th frame. It represents optical flow.

[0083] In sparse scenes, the prediction model uses a least-squares GAN (LSGAN) to predict the next frame and uses a generator G... p Image spoofing detector D p In the GAN architecture, D is defined. p The losses of the confrontation As shown in equation (4), train G p When D is fixed p The weights, defined by G p The losses of the confrontation As shown in equation (5):

[0084]

[0085]

[0086] in, I represents the predicted frame, and I represents the original frame.

[0087] Combining intensity loss, gradient loss, and optical flow loss, we obtain G. p objective function As shown in equation (6), D p objective function As shown in equation (7):

[0088]

[0089]

[0090] Where, λ int The weighting coefficient λ represents the intensity loss. gd λ represents the weight coefficients of the gradient loss. op The weighting coefficient λ represents the optical flow loss. adv This represents the weighting coefficients for adversarial training loss.

[0091] In dense scenes, targets that reappear after occlusion are easily lost. Reconstruction-based methods only reconstruct based on the current frame, thus reducing the probability of losing occluded targets. For the reconstruction network, the settings are essentially the same as the prediction network, except for the absence of hop connections. Generator G r Reconstructing the network loss function As shown in equation (8); the discriminator of the reconstructed model is D. r The loss function of its adversarial training As shown in equation (9):

[0092]

[0093]

[0094] By combining intensity loss, gradient loss, and optical flow loss, the objective function of the reconstruction model is obtained. As shown in equation (10):

[0095]

[0096] Where, λ int The weighting coefficient λ represents the intensity loss. gd λ represents the weight coefficients of the gradient loss. op The weighting coefficient λ represents the optical flow loss. adv This represents the weighting coefficients for adversarial training loss.

[0097] The effectiveness of the method proposed in this invention will be verified through experiments below.

[0098] (1) Introduction to the test dataset

[0099] To validate the proposed anomaly detection model in practical applications of video surveillance, this section evaluates its performance on three benchmark datasets: UCSD Ped2, CHUK Avenue, and Shanghai Tech. The UCSD Ped2 dataset is derived from static cameras overlooking sidewalks. Anomalies include non-pedestrian objects (such as vehicles and bicycles) and unusual pedestrian movements (such as running). It contains 16 training videos and 12 test videos, with a resolution of 240×360. The CHUK Avenue dataset consists of 30,652 frames, divided into 16 training videos and 21 test videos. This dataset was collected on a campus avenue and contains 47 anomalies, including throwing objects, loitering, and running, with a resolution of 360×640. The Shanghai Tech dataset contains 330 training videos and 107 test videos, including 130 anomalies such as cars on the sidewalk, theft, and violence, with a resolution of 480×856.

[0100] (2) Parameter settings

[0101] Before training, the size of each frame was adjusted to 256×256. The learning rates for the generator and discriminator in both the prediction and reconstruction models were set to the same value: 1e-4 and 1e-5, respectively. For parameter optimization, we chose the Adam optimizer to train the entire model end-to-end, with a decay rate of 1e-4, a learning rate of 1e-5, a momentum of 0.9, and a batch size of 16. All experiments were conducted on a dedicated GPU server with a 2.1GHz Intel Xeon Silver 4216 CPU, 32GB of RAM, an NVIDIA TITANX GPU, and 24GB of video storage. The anomaly detection architecture was implemented using the PyTorch framework in Windows 10, Python 3.7, and CUDA 10.0.

[0102] (3) Model Framework Process

[0103] In real-world applications, there are multiple crowd densities. Analysis of current common video-based abnormal behavior algorithms shows that most algorithms lack the ability to distinguish between different crowd densities, and the characteristics of crowds at different densities vary greatly.

[0104] The overall process is as follows Figure 1As shown, the Double-Canny algorithm is first used to extract pixel features. Background subtraction and frame differencing are used for initial foreground target extraction. However, due to the presence of slowly moving or briefly stationary pedestrians in the scene, frame differencing alone cannot capture relevant information well, leading to significant errors in pixel extraction. Therefore, in addition to performing the Canny operator on the foreground frame, the operator is also performed on the original image to obtain the overall edge image. A pixel-by-pixel AND operation is performed between the entire edge image and the foreground image to remove irrelevant edges and effectively capture missing information in the foreground. Then, a pixel-by-pixel OR operation is performed between the generated foreground image and the foreground image of the original image to generate the final foreground image, from which the number of foreground pixels can be calculated. The Canny operator is then performed on the final foreground image for edge detection to obtain the final edge detection image, and the number of edge pixels is calculated. To reflect the degree of crowd occlusion, the ratio of the sum to the original image needs to be calculated.

[0105] As an example, several videos were selected from the training set of the UCSD Ped2 dataset, and the curves showing the number of edge pixels and their ratio for each frame in the videos were plotted, such as... Figure 2 As shown.

[0106] YOLOv4 is used to detect the specific number of people in video frames and generate corresponding heatmaps to calculate the proportion of high-heat areas, representing the degree of crowd concentration. For example... Figure 3 The image shows the visualization and heatmap results of pedestrian detection in sparse and dense scenes using YOLOv4. Then, to adaptively distinguish between sparse and dense scenes, the three extracted pixel features, the number of people obtained using YOLOv4, and the proportion of high-heat areas in the heatmap are used as feature vectors. The K-means clustering method is then used to automatically cluster the scenes in the video into sparse and dense categories. The classification visualization results for the three dataset test sets are shown below. Figure 4 As shown.

[0107] Table 1 shows the proportion of sparse and dense scenes in the test set of the three datasets.

[0108]

[0109] Table 1 shows the proportion of sparse and dense scenes in the test set of the three datasets. For sparse scenes, a GAN-based prediction method is used. Given consecutive frames, U-Net is used as a generator to predict the (t+4)th frame. Anomalies are detected by calculating the difference between the predicted frame and the real frame. The prediction network diagram is shown below. Figure 5 As shown in the diagram. For dense scenes, a GAN-based reconstruction method is adopted. The t-th frame is used as input, and an autoencoder is used to output the reconstructed frame. Similarly, anomalies are identified by calculating the difference between the t-th and t-th frames. The reconstruction network diagram is shown in the diagram. Figure 6 As shown.

[0110] (4) Model performance comparison

[0111] This invention assesses model performance by comparing the area under the ROC curve, i.e., the AUC (%) value. The higher the AUC value, the better the model performance.

[0112] Table 2 Comparison of AUC values ​​for different models

[0113]

[0114] As shown in Table 2, the AUC values ​​of the proposed method on the three datasets reached 95.9%, 85.9%, and 73.4%, respectively. Particularly on UCSD Ped2, the proposed method outperformed other methods by at least 0.4% (95.9% vs. 95.5%). This demonstrates the effectiveness of the proposed method in classifying crowd scenes for anomaly detection. Compared to MemAE, the method achieved approximately 1.6% and 1.2% improvements on CHUK Avenue and Shanghai Tech, respectively. This is because, on the one hand, LSGAN outperforms GAN, and on the other hand, scene classification does indeed contribute to enhancing the model's detection capabilities.

[0115] To qualitatively analyze the anomaly detection performance of the model in this invention, ROC curves of three classic algorithms were plotted for comparison, such as... Figure 7 As shown, by changing the threshold parameter, a series of anomaly detection results and their corresponding false positive rate (FPR) and true positive rate (TPR) are obtained. Therefore, the ROC curve can be plotted from a series of coordinate points composed of FPR and TPR. It can be seen that the method of this invention outperforms MemAE and Liu et al. on all three datasets.

[0116] In summary, the proposed scene classification method does indeed improve upon the original model. To verify the effectiveness of the proposed scene classification method, ablation experiments were conducted on three datasets.

[0117] Table 3. Results of classification ablation experiments on the three datasets.

[0118]

[0119] Taking the UCSD Ped2 dataset as an example, Table 3 shows that the proposed reconstruction and prediction methods exhibit significant AUC performance of 94.5% and 95.2% respectively on UCSD Ped2. Compared with the two detection methods without classification, the final detection performance achieves improvements of 1.4% (95.9% vs 94.5%) and 0.7% (95.5% vs 95.2%). The proposed method also enhances model performance on the CHUKAvenue and Shanghai Tech datasets. In summary, the experimental results on the three datasets demonstrate that the proposed method can effectively distinguish between sparse and dense scenes and detect anomalies in different scenes using appropriate methods, thus effectively improving the overall anomaly detection performance.

[0120] This invention also proposes a video anomaly detection system based on scene classification, which includes:

[0121] The preprocessing unit performs grayscale processing on the original video frames, selects a median filter for smoothing, uses background subtraction and frame difference methods to extract foreground targets, and extracts pixel features through the Double-Canny algorithm.

[0122] The people detection unit uses a pre-trained model to detect the number of people in video frames and generate a heatmap of the video frames.

[0123] The scene classification unit uses the K-means clustering algorithm to classify video frames into two categories: dense scenes and sparse scenes.

[0124] An anomaly detection unit uses an anomaly detection module to perform anomaly detection on dense and sparse scenes respectively, wherein the anomaly detection module is a GAN-based dual-branch network structure.

[0125] It should be noted that the description of the system and apparatus in the embodiments of the present invention is similar to that in the above method embodiments, and has similar beneficial effects as the method embodiments, so it will not be repeated.

[0126] Finally, the present invention proposes an electronic device comprising a memory, a processor, and program instructions stored in the memory that are executable by the processor. The processor executes the program instructions to implement the various steps of the video anomaly detection method based on scene classification proposed in this invention.

[0127] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0128] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video anomaly detection method based on scene classification, characterized in that, Includes the following steps: Step S1: Convert the original video frame to grayscale, select a median filter for smoothing, use background subtraction and frame difference to extract the foreground target, and extract pixel features using the Double-Canny algorithm. Step S2: Use a pre-trained model to detect the number of people in the video frames and generate a heatmap of the video frames; Step S3: Use the K-means clustering algorithm to classify video frames into two categories: dense scenes and sparse scenes. Step S4: Use the anomaly detection module to perform anomaly detection on dense and sparse scenes respectively. The anomaly detection module is a dual-branch network structure based on GAN. In step S4, the anomaly detection module uses a prediction model in sparse scenes and uses U-Net as a generator to predict the next frame; in dense scenes, it applies a reconstruction model, and the generator is an autoencoder with no skip connections used to reconstruct video frames.

2. The video anomaly detection method based on scene classification according to claim 1, characterized in that, Step S1 specifically includes: Step S1.1: Apply the Canny operator to edge detection for both the foreground frame and the original image to obtain the foreground image and the overall edge image; Step S1.2: Perform a pixel-by-pixel "AND" operation on the overall edge image and foreground image in step S1.1 to remove irrelevant edges and capture missing information in the foreground to obtain the processed foreground image; Step S1.3: Perform an "OR" operation pixel by pixel on the foreground image in step S1.1 and the processed foreground image in step S1.2 to obtain the final foreground image; Step S1.4: Calculate the number of pixels in the binary image of the final foreground image from step S1.3 to obtain the foreground pixel count. ; Perform Canny operator edge detection on the final foreground image to obtain the final edge pixel image, and calculate the number of edge pixels. ; Calculate the ratio of edge pixels to foreground pixels. , Indicates the degree of crowd obstruction.

3. The video anomaly detection method based on scene classification according to claim 2, characterized in that, In step S2, the pre-trained model uses YOLOv4 to detect the number of people; a heatmap of the image is generated using the pre-trained model, and the proportion of the highlighted parts of the heatmap is calculated. , It indicates the degree of crowd gathering.

4. The video anomaly detection method based on scene classification according to claim 3, characterized in that, In step S3, the three pixel features obtained in step S1 and the number of people and the highlight ratio in the heat map obtained in step S2 are used as feature vectors. ,in It is the feature vector of the i-th frame. It represents the number of people in the i-th frame. and These are the number of edge pixels and the number of foreground pixels in the i-th frame, respectively. yes and The ratio, This represents the proportion of the highlighted portion in the heatmap of the i-th frame; the feature vector is used as the input value, and the K-means clustering algorithm automatically clusters the scenes in the video into two categories: sparse and dense.

5. The video anomaly detection method based on scene classification according to claim 4, characterized in that, The anomaly detection module incorporates intensity loss, gradient loss, and optical flow loss as constraints; intensity loss By comparing the value of each pixel between the predicted frame and the original frame, as shown in Equation (1): ； in, Indicates the predicted frame. Indicates the original frame; Gradient loss It is obtained by comparing the gradient of pixel values ​​of the predicted frame and the original frame, as shown in Equation (2): ； in, Indicates the spatial index of the predicted frame. Indicates the spatial index of the original frame; Optical flow loss The calculation is shown in equation (3): ； in, This represents the predicted t-th frame. This represents the original (t+1)th frame. It represents optical flow.

6. The video anomaly detection method based on scene classification according to claim 5, characterized in that, The prediction model in the sparse scene uses LSGAN to predict the next frame and uses a generator. Image spoofing detector ;definition The losses of the confrontation As shown in equation (4), training At time, fixed The weights are defined. The losses of the confrontation As shown in equation (5): ； ； in, Indicates the predicted frame. Indicates the original frame; Combining intensity loss, gradient loss, and optical flow loss, we obtain objective function As shown in equation (6), objective function As shown in equation (7): ； ； in, The weighting coefficients representing the intensity loss. The weights representing the gradient loss. The weighting coefficients representing optical flow loss This represents the weighting coefficients for adversarial training loss.

7. The video anomaly detection method based on scene classification according to claim 6, characterized in that, In the dense scene, the generator of the reconstructed model The loss function used in the reconstruction network As shown in equation (8); the discriminator of the reconstructed model is The loss function of its adversarial training As shown in equation (9): ； ； By combining intensity loss, gradient loss, and optical flow loss, the objective function of the reconstruction model is obtained. As shown in equation (10): ； in, The weighting coefficients representing the intensity loss. The weights representing the gradient loss. The weighting coefficients representing optical flow loss This represents the weighting coefficients for adversarial training loss.

8. A video anomaly detection system based on scene classification, characterized in that, Include: The preprocessing unit performs grayscale processing on the original video frames, selects a median filter for smoothing, uses background subtraction and frame difference methods to extract foreground targets, and extracts pixel features through the Double-Canny algorithm. The people detection unit uses a pre-trained model to detect the number of people in video frames and generate a heatmap of the video frames. The scene classification unit uses the K-means clustering algorithm to classify video frames into two categories: dense scenes and sparse scenes. An anomaly detection unit uses an anomaly detection module to perform anomaly detection on dense and sparse scenes respectively, wherein the anomaly detection module is a GAN-based dual-branch network structure. In sparse scenes, the anomaly detection module employs a prediction model and uses U-Net as a generator to predict the next frame; in dense scenes, it applies a reconstruction model, where the generator is a non-jumping autoencoder used to reconstruct video frames.

9. An electronic device comprising a memory, a processor, and program instructions stored in the memory and executable by the processor, characterized in that, The processor executes the program instructions to implement the steps of the method according to any one of claims 1-7.

Images