A video anomaly detection system and method based on abnormal region positioning

Abstract

The application discloses a video anomaly detection system and method based on abnormal area positioning. The system comprises a compressed domain video information sampling module, an MV reconstruction module, an I frame future frame prediction module and a motion intensity adaptive abnormal area positioning module. The method comprises the following steps: in the compressed domain video information sampling module, MV sequences and I frame sequences are obtained through sparse sampling; the MV sequences are input into the MV reconstruction module to obtain reconstructed MV sequences, and the mean square error of the reconstructed MV and the residual of the reconstructed MV are calculated; the I frame sequences are input into the I frame future frame prediction module, and the prediction network pays more attention to the abnormal area when generating the predicted I frame under the guidance of the residual of the reconstructed MV, and the mean square error of the predicted I frame is calculated; the mean square error of the reconstructed MV and the mean square error of the predicted I frame are weighted to obtain an error image, which is input into the motion intensity adaptive abnormal area positioning module to realize the positioning of the abnormal area and obtain the abnormal score of the current I frame.

Description

Technical Field

[0001] This invention belongs to the field of electronic information technology, and specifically relates to a video anomaly detection system and method based on anomaly region localization. Background Technology

[0002] Video anomaly detection is an important research area in computer vision, with wide applications, especially in video surveillance. For example, it has high practical value in detecting anomalies such as vehicles entering pedestrian walkways, sudden gatherings and dispersals, and small-scale fights.

[0003] Current video anomaly detection algorithms based on deep neural networks are mostly image-domain based and can be divided into two main categories: reconstruction and prediction. Reconstruction-based methods rely on the fact that models trained on normal data will produce significant reconstruction errors when reconstructing anomalous data. Prediction-based methods utilize the temporal characteristics between consecutive frames to predict future frames, but when anomalies occur, the predicted future frames show significant errors compared to the actual frames. Some algorithms combine reconstruction and prediction. However, existing algorithms, when using motion information such as optical flow and appearance information for joint detection, fail to establish an effective correlation between the two. Ultimately, they focus on large-scale motion regions rather than regions where anomalous changes occur. Furthermore, the size of the anomalous region varies in real-world scenarios. The neglect of the anomalous region size by existing methods results in only a minor improvement in anomaly detection accuracy. Therefore, developing an anomaly detection algorithm that can perceive anomalous regions remains a challenging task.

[0004] In summary, although current anomaly detection algorithms have achieved relatively good results, two problems still need to be addressed: (1) There is no effective correlation between motion information and appearance information. Optical flow information represents motion information, while the purpose of anomaly detection is to detect abnormal motion, not just motion. Existing algorithms directly use optical flow as an auxiliary motion information, which has achieved certain results, but it does not highlight abnormal motion information, and normal motion will also interfere with the detection of abnormal motion. (2) Ignoring the size of the abnormal region leads to inaccurate detection. When calculating the anomaly score of an image frame, the commonly used method is to directly average the squared difference of all pixels, ignoring the fact that the size of the abnormal region is closely related to the detection accuracy. In actual scenarios, the proportion of the abnormal region in the entire monitoring screen is usually very small, and the direct averaging operation will weaken the anomaly score, resulting in an insignificant difference in scores between normal frames and abnormal frames. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the prior art by providing a video anomaly detection system and method based on anomaly region localization.

[0006] The present invention is achieved using the following technical solution:

[0007] A video anomaly detection system based on anomaly region localization includes:

[0008] The compressed domain video information sampling module is used to sample appearance and motion information from the compressed bitstream, which are then used as inputs to the I-frame future frame prediction module and the MV reconstruction module, respectively.

[0009] The MV reconstruction module is used to reconstruct motion information. The mean square error of the reconstructed MV is included as a component of the error image. At the same time, the residual of the reconstructed MV is calculated to guide the prediction network to pay more attention to abnormal regions when predicting future frames.

[0010] The I-frame future frame prediction module is used to generate predicted future frames. The mean square error of the predicted I-frame is one of the components of the error image. The mean square error of the reconstructed MV and the mean square error of the predicted I-frame are weighted to obtain the error image, which is then input into the motion intensity-based adaptive perception anomaly region localization module.

[0011] The motion intensity adaptive perception abnormal region localization module is used to focus on abnormal regions and increase the score difference between normal frames and abnormal frames.

[0012] A further improvement of the present invention is that, in the compressed domain video information sampling module, I-frames representing appearance information and MV prior knowledge representing motion information are directly obtained from the compressed bitstream based on partial decoding, and are respectively used as inputs to the I-frame future frame prediction module and the MV reconstruction module.

[0013] A further improvement of this invention is that the residual between the reconstructed MV sequence output by the MV reconstruction module and the input MV sequence is given a larger weight to the channel that focuses on the abnormal region after passing through the channel attention network, and the I-frame sequence features in the I-frame future frame prediction network are weighted so that more attention is paid to the abnormal region when predicting the future frame.

[0014] A further improvement of this invention is that the mean square errors of the input and output results of the MV reconstruction module and the I-frame future frame prediction module are weighted and summed to obtain the error image E; in the motion intensity-based adaptive perception anomaly region module, the MV corresponding to the future I-frame is subjected to a Laplacian operation to filter out the anomaly regions with high probability and weighted, thereby focusing on the anomaly regions; finally, the anomaly score is calculated on the corresponding region on the error image based on the sliding window mechanism, and the maximum score is taken as the anomaly score for predicting the future frame.

[0015] A video anomaly detection method based on anomaly region localization includes the following steps:

[0016] 1) Obtain the MV sequence and I-frame sequence through sparse sampling in the compressed domain video information sampling module;

[0017] 2) The MV sequence is input into the MV reconstruction module to obtain the reconstructed MV sequence, and the mean square error of the reconstructed MV and the residual of the reconstructed MV are calculated.

[0018] 3) The I-frame sequence is input into the I-frame future frame prediction module. Under the guidance of the residual of the reconstructed MV, the prediction network pays more attention to the abnormal region when generating the predicted I-frame, and the mean square error of the predicted I-frame is calculated.

[0019] 4) The mean square error of the reconstructed MV and the mean square error of the predicted I-frame are weighted to obtain an error image, which is then input into the motion intensity-based adaptive perception anomaly region localization module to achieve accurate localization of the anomaly region and calculate the anomaly score of the current I-frame.

[0020] A further improvement of the present invention is that the specific implementation method of step 1) is as follows:

[0021] 101) The original video is encoded into a compressed bitstream using the HM codec; the compressed video bitstream consists of a series of GoPs, each GoP containing an I-frame and several P-frames. The I-frame represents appearance information, and the MV extracted from the P-frames represents motion information.

[0022] 102) Continuously sampling t+1 GoP I-frames is taken as an I-frame sequence, denoted as I′={I0,I1...,I...} t-1 ,I t In GoP, the MV of the first P-frame best reflects the motion trend of the I-frame. Extracting the corresponding MV sequence, denoted as MV′={MV0,MV1...,MV...}, yields the sequence. t-1 ,MV t}

[0023] A further improvement of the present invention is that the implementation method of step 2) is as follows:

[0024] 201) The first t frames of the MV sequence MV′ are denoted as MV and input into the MV reconstruction module with an AE network structure and no skip connections to obtain the reconstructed MV sequence.

[0025] 202) Calculate the mean square error between the reconstructed MV sequence and the input MV sequence;

[0026] 203) Calculate the residual R between the reconstructed MV sequence and the input MV sequence to obtain the residual R of the reconstructed MV.

[0027] A further improvement of the present invention is that the implementation method of step 3) is as follows:

[0028] 301) The first t frames of the I-frame sequence I′ are denoted as I and input into the I-frame future frame prediction module with AE network structure and skip connection;

[0029] 302) After the residual R passes through the channel attention network, channels that focus on anomalous regions are given greater weights. This weights the features of the I-frame sequence I after passing through the encoder, thereby enabling the prediction network to predict future frames. Pay close attention to abnormal areas;

[0030] 303) Calculate the mean square error of the predicted I-frame and the actual I-frame.

[0031] A further improvement of the present invention is that the implementation method of step 4) is as follows:

[0032] 401) Regarding the MV and in the reconstructed network And the I in the prediction network t and The error image E is obtained by weighting the mean square error in the spatial dimension. The dimension of E is [h, w], where h and w represent the length and width of the I-frame, respectively.

[0033]

[0034] 402) Perform a Laplacian operation on the error image E to extract the edges of the motion region. Specifically, perform a Laplacian operation on the x and y components of the MV array respectively, and then sum them with a weight of 0.5.

[0035] 403) High-pass filtering is performed on the derivative values ​​at all edges to retain only the derivative values ​​greater than the median value, and finally the contour of the motion region with a high probability of anomaly is obtained.

[0036] 404) Assign weights to all pixels within each region; after filtering, the regions contain a lot of noise and linear contours. For the linear contours, take the bounding rectangle and filter out contours with an area smaller than 4x4 to obtain the final motion regions with a high probability of anomalies; furthermore, only the outermost contour is selected to obtain regular regions. The total number of corresponding regions in a frame is denoted as N; for each region, weights are assigned to the pixels within the region based on the second derivative value at the contour; the weights of all pixels in the k-th region are denoted as w. k The value is the average of all derivative values ​​at the region contour; after normalizing the weights of all regions within the frame, the corresponding value is assigned to the pixel value in each region, and the weight of the pixel outside the region is 0; in order to ensure the integrity of the information of the entire error image, the weight of all pixels is multiplied by 0.1; the weight matrix obtained by the above calculation is denoted as W, and the dimension is also [h,w].

[0037] 405) Weight the error image E; use the weight matrix W to weight the error matrix E in the spatial dimension, so that the error value is larger in the motion region with a high probability of anomaly; the weighted error image is denoted as Ev;

[0038] E′(x,y)=E(x,y)+E(x,y)*W(x,y)

[0039] 406) Calculate anomaly scores based on sliding window; Using the sliding window mechanism, calculate the anomaly score of each small region on the error image E′, take the maximum score value as the anomaly score of the frame, and the size of the sliding window is 16x16.

[0040] Score(I t ) = max(∑ m,n∈P E'(m,n))

[0041] Where P represents the area selected by the sliding window, and m, n are the coordinates of the pixels in P;

[0042] 407) Finally, after passing through the anomaly detection network, the anomaly score of all I-frames in a video is obtained; since the motion changes between adjacent frames are very small, the anomaly score of the frame corresponding to the unsampled frame, i.e., all P-frames, is obtained by averaging the scores of two adjacent I-frames, thus obtaining the anomaly score of each video frame in the video.

[0043] Compared with the prior art, the present invention has at least the following beneficial technical effects:

[0044] (1) Effective association between motion information and appearance information has been achieved.

[0045] Video anomaly detection is based on unsupervised machine learning algorithms and can be divided into two main categories: reconstruction and prediction. The videos in the training set represent normal events, and we train an Affective Events (AE) network to reconstruct and predict these events. For events not present in the training set—i.e., anomalous events—our network cannot correctly reconstruct or predict them. In other words, the basic idea behind reconstruction-based methods is that models trained on normal data will produce significant reconstruction errors when reconstructing anomalous data. Prediction-based methods utilize the temporal characteristics between consecutive frames to predict future frames; when an anomalous event occurs, the predicted future frame will have a significant prediction error compared to the true frame. The larger the reconstruction error or prediction error, the greater the likelihood of an anomaly.

[0046] Existing algorithms utilize motion information such as optical flow and appearance information for joint detection, but they fail to establish effective correlations, focusing on large-scale motion regions rather than areas of anomalous change. To address this issue, we input the MV sequence into an MV reconstruction module to obtain a reconstructed MV. Compared to the input MV, the reconstructed MV often shows larger errors at areas where anomalies occur. Therefore, we use the residual between the reconstructed MV and the original MV to reflect the extent and severity of the anomalous region, serving as anomalous motion information. Using this MV residual channel attention network, channels focusing on anomalous regions are given greater weights, weighting the I-frame features and thus making the prediction network pay more attention to anomalous regions when predicting future frames.

[0047] (2) Highlight abnormal areas and calculate abnormal scores within a small range based on a sliding window to increase the gap between normal and abnormal frames.

[0048] An error image is obtained by weighting the mean squared errors of the inputs and outputs of the reconstruction and prediction modules in the spatial dimension. The error value of each pixel in the error image reflects the degree of anomaly. However, anomalous regions usually occupy a small proportion of the entire image, and most algorithms calculate the anomaly score by averaging the error values ​​of all pixels. This weakens the anomaly score, thus missing anomalous events occurring in small regions.

[0049] In grayscale images, the grayscale values ​​change significantly at contours. Therefore, the second derivative (Laplacian operator) is used to extract contours from grayscale images, and the second derivative extracts more refined contours than the first derivative. Similarly, areas where anomalies occur exhibit significant motion changes, meaning the MV value changes greatly. Therefore, the second derivative of MV can be used to extract edges with large motion changes; the larger the derivative value, the greater the motion intensity.

[0050] After taking the second derivative of MV, the filtered regions contain a lot of noise and linear contours. Noise removal and bounding rectangle regularization are then performed to obtain the final abnormal regions. For each abnormal region, a weight is assigned to the error value within the region based on the derivative value of the region's edge in the spatial dimension; the larger the derivative value, the larger the weight. This weight is used to spatially weight the error image, making the error value at the abnormal regions larger, thus highlighting the abnormal regions.

[0051] Finally, a sliding window mechanism is used to reduce interference from background areas. The sliding window selects a region of a specific size within the entire frame and calculates anomaly scores only for the error values ​​within the selected region. When the selected region contains anomalies, the score is higher; conversely, when the proportion of background in the selected region is large, the score is lower. Ultimately, the maximum score across all selected regions in the frame is taken as the frame-level score. This increases the score difference between normal and abnormal frames while also mitigating background interference to some extent. Attached Figure Description

[0052] Figure 1 This is a comparison chart of the traditional cloud model and the cloud-edge collaboration model.

[0053] Figure 2 This is an overall block diagram of the present invention.

[0054] Figure 3 This is a schematic diagram of the compressed domain video information sampling module.

[0055] Figure 4 This is a network structure diagram for the MV reconstruction module and the I-frame future frame prediction module.

[0056] Figure 5 The following are the effect diagrams of each process of the motion intensity-based adaptive perception anomaly region localization module. (a) Original image. (b) Visualization of MV in Elecard HEVC Analyzer software. (c) Second derivative of MV in step ①. (d) Region with high motion intensity in step ②. (e) Regularization and weighting of the selected regions in step ③. (f) Overlaying the selected regions from step ③ onto the original image. (g) Sliding window position corresponding to the maximum anomaly score. Detailed Implementation

[0057] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0058] With the rapid development of technologies such as artificial intelligence and the Internet of Things, intelligent monitoring systems have been widely deployed in various locations such as schools, shopping malls, and public transportation. Video anomaly detection is one of the commonly used technologies in monitoring systems. In the traditional cloud model, a large amount of video data acquired from the monitoring terminal is compressed and transmitted to the cloud server. The compressed video is then fully decoded into image-domain video and input into the detection system for data analysis. However, the monitoring terminal is often located far from the cloud server, and a significant amount of time is spent on data transmission and video decoding, making it impossible to simultaneously achieve both detection speed and accuracy. To achieve efficient anomaly detection, the proposed algorithm is planned for deployment in cloud-edge collaborative application scenarios, such as... Figure 1As shown, the detection system is deployed directly on the edge server to reduce latency and data loading time. Data collected from the monitoring terminal is transmitted to the edge server in the form of compressed video. The system decodes the received compressed video portion, performs anomaly detection, and can directly return the detection results to the monitoring center. Compared with image domain anomaly detection algorithms, this method saves transmission time, and partial decoding requires less decoding time and occupies less storage space compared to full decoding. Depending on the actual application requirements, the edge server can also transmit video data and detection results to a cloud server.

[0059] The overall block diagram of the proposed algorithm is as follows: Figure 2 As shown. This invention provides a video anomaly detection system based on anomaly region localization, comprising:

[0060] The compressed domain video information sampling module samples appearance and motion information from the compressed bitstream, which are then used as inputs to the I-frame future frame prediction module and the MV reconstruction module, respectively. The MV reconstruction module reconstructs motion information, and the mean square error of the reconstructed MV is used as a component of the error image. Simultaneously, the residual of the reconstructed MV is calculated to guide the prediction network to focus more on anomalous regions when predicting future frames. The I-frame future frame prediction module generates predicted future frames, and the mean square error of the predicted I-frame is used as a component of the error image. The mean square error of the reconstructed MV and the mean square error of the predicted I-frame are weighted to obtain the error image, which is then input into the motion intensity-based adaptive perception anomaly region localization module. The motion intensity-based adaptive perception anomaly region localization module focuses on anomalous regions, increasing the score difference between normal and anomalous frames.

[0061] This invention provides a video anomaly detection method based on anomaly region localization, comprising the following steps:

[0062] 1) Obtain the MV sequence and I-frame sequence through sparse sampling in the compressed domain video information sampling module;

[0063] 2) The MV sequence is input into the MV reconstruction module to obtain the reconstructed MV sequence, and the mean square error of the reconstructed MV and the residual of the reconstructed MV are calculated.

[0064] 3) The I-frame sequence is input into the I-frame future frame prediction module. Under the guidance of the residual of the reconstructed MV, the prediction network pays more attention to the abnormal region when generating the predicted I-frame, and the mean square error of the predicted I-frame is calculated.

[0065] 4) The mean square error of the reconstructed MV and the mean square error of the predicted I-frame are weighted to obtain an error image, which is then input into the motion intensity-based adaptive perception anomaly region localization module to achieve accurate localization of the anomaly region and calculate the anomaly score of the current I-frame.

[0066] The compressed video stream consists of a series of Groups of Pictures (GOPs), each GOP containing one I-frame and several P-frames. The I-frame represents appearance information, while the motion signature (MV) extracted from the P-frames represents motion information. In the algorithm, t+1 GOPs are sampled consecutively, and the I-frame and the MV corresponding to the first P-frame in each GOP are extracted. The first t frames of the I-frame and MV sequences are used as inputs to the prediction and reconstruction networks, respectively. The difference between the reconstructed MV sequence and the true MV sequence obtained by the reconstruction network is represented by residuals. These residuals are passed through a channel attention network, which assigns greater weights to channels that focus on anomalous regions, thus making the prediction network pay more attention to anomalous regions when predicting future frames. The errors from the reconstruction and prediction networks are weighted and summed to obtain the error image E. In the motion intensity-based adaptive perception anomalous region module, the MV corresponding to the future I-frame undergoes a Laplacian operation to initially filter out anomalous regions with high probability. Then, the corresponding filtered regions on the error image E are weighted, i.e., the error at the anomalous regions is increased. Finally, the anomaly score is calculated using a sliding window on the corresponding region of the error image E, and the maximum score is taken as the anomaly score for predicting future frames.

[0067] 1. Compressed Domain Information Sampling Module

[0068] This paper uses the HEVC encoding standard for encoding and decoding, and obtains compressed video through the official HM encoder. The compressed video stream is composed of Groups of Pictures (GOPs), which contain one I-frame and several P-frames. I-frames are intra-coded frames, encoded independently without relying on other frames. P-frames are inter-coded frames, eliminating a large amount of redundant information between adjacent frames. A P-frame consists of a motion vector (MV) and a residual; adjacent P-frames have a reference dependency. In HEVC encoding, each prediction unit (PU) corresponds to one MV value. The largest segmentation unit of a PU is 4x4; therefore, in this paper's algorithm, each 4x4 pixel block stores one MV value (e.g., the MV array size for a 256x256 frame is 64x64x2).

[0069] Because the motion changes between adjacent frames are minimal, consecutive frames exhibit significant motion redundancy. Based on the characteristics of compressed bitstreams, I-frames represent appearance information, while MVs represent motion information. A sample is defined as I′ = {I0, I1, ..., I1}, where t+1 consecutive I-frames from a GOP are sampled. t-1 ,I t},like Figure 3 As shown, consecutive I-frame sequences have relatively low redundancy and a large temporal span, better reflecting the trend of motion changes. To capture motion changes more effectively, MV sampling is used as an auxiliary method. The MV of the first P-frame in a GOP best reflects the motion trend of the I-frames, and the corresponding MV sequence is denoted as MV′={MV0,MV1...,MV1}. t-1 ,MVt The first t frames of MV′ and I′ are used as inputs to the reconstruction network and the prediction network, respectively, denoted as MV={MV0,MV1...,MV1}. t-1} and I={I0,I1...,I t-1}

[0070] 2. MV reconstruction module and I-frame future frame prediction module

[0071] Mainstream anomaly detection algorithms can be divided into two main categories: reconstruction-based anomaly detection algorithms and prediction-based anomaly detection algorithms. However, reconstruction-based algorithms lack consideration for contextual relationships, while prediction-based algorithms have a higher probability of false detections. Therefore, this algorithm adopts a network architecture that combines reconstruction and prediction, with the reconstruction network assisting and guiding the prediction network, allowing the two to complement each other's strengths and weaknesses.

[0072] In the algorithm, both the reconstruction network and the prediction network adopt the classic autoencoder (AE) network structure. The difference is that the reconstruction network does not have skip connections, such as... Figure 4 As shown. Because the function of the reconstruction network is to reconstruct the current input sequence, skip connections may allow information from the original domain to be directly passed through these connections for information reconstruction in the decoding domain, rather than emphasizing low-level features through bottlenecks, which can improve the network's generalization ability to some extent. The prediction network, on the other hand, is designed to predict future frames. Both the original and decoding domains can serve as feature information for the predicted frames. Using skip connections can complement high-level and low-level information, increasing the detail of the prediction.

[0073] The first t frames of MV′ and I′ are used as inputs to the reconstruction network and the prediction network, respectively. MV is processed by the reconstruction network to obtain the reconstructed MV sequence. The reconstruction network assumes that a model learned only from normal data cannot accurately reconstruct anomalies. Therefore, ideally, when the input sequence represents normal events, the output reconstructed sequence should be similar. However, when anomalies occur, the two sequences differ significantly, with the regions of larger differences corresponding to the areas where anomalies occur. The residual between the input MV sequence and the reconstructed MV sequence is denoted as:

[0074]

[0075] The '-' symbol represents pixel-level subtraction.

[0076] Therefore, R can be used to reflect the magnitude of the abnormal region and the degree of abnormality. The features encoded by R are then processed by a channel attention network and weighted against the features encoded by I, guiding the prediction network in predicting I. t It pays more attention to abnormal regions. At the same time, it combines the reconstruction task and the prediction task. When an abnormal event is encountered, the error brought by MV reconstruction will further affect the prediction effect of future I-frames, making abnormal events easier to detect.

[0077] In reconstructing the network, MV and The L2 distance between them is used as the loss function for the reconstruction module:

[0078]

[0079] In the prediction network, L2 distance is used as the intensity loss to ensure pixel similarity. At the same time, to prevent the predicted image from being too blurry, gradient loss is added to preserve the original gradient of the appearance image.

[0080]

[0081]

[0082] in To predict future frames in the network output, I is its true value. x, y represent the spatial pixel positions in frame I.

[0083] The overall loss function of the proposed algorithm is:

[0084] L total =λ1L recon +λ2L int +λ3L grad

[0085] Where λ1, λ2 and λ3 are hyperparameters.

[0086] 3. Anomaly region localization module based on motion intensity adaptive sensing

[0087] Existing algorithms often calculate anomaly scores directly based on the entire frame, averaging the reconstruction or prediction error across all pixels. However, in real-world scenarios, anomalous regions typically constitute a small proportion of the entire video, and direct averaging weakens the anomaly score, resulting in insignificant score differences between normal and anomalous frames. Therefore, identifying anomalous regions is crucial for improving detection accuracy. To address this issue, we first weight regions with a higher probability of anomalies in the error image, increasing the error at these regions. Then, we calculate the anomaly score on the weighted error image using a sliding window. This algorithm highlights anomalous regions while avoiding weakening the anomaly score, achieving excellent performance.

[0088] In the algorithm, the anomaly score is calculated on the joint error of reconstruction error and prediction error. This applies to the MV and... And the I in the prediction network t and Calculate the mean squared error (MSE) in the spatial dimension separately, and then weight the two results to obtain the error image E:

[0089]

[0090] Where α1 and α2 are hyperparameters, and x and y represent the spatial pixel positions in the error image. The dimension of E is [h, w], where h and w represent the length and width of the I-frame, respectively.

[0091] In grayscale images, the grayscale values ​​change significantly at contours; therefore, the second derivative (Laplacian operator) is used to extract contours. The second derivative extracts more refined contours than the first derivative. Similarly, the second derivative of MV can be used to extract edges with large motion variations; the larger the derivative value, the greater the motion intensity. Based on this characteristic, a module for adaptively perceiving abnormal regions based on motion intensity was designed to obtain anomaly scores. The specific steps are as follows. To more clearly illustrate the purpose of each step, the corresponding results of each step are visualized in two different scenarios. The original images of the two scenarios are as follows: Figure 5 As shown in (a), the MV visualization is as follows: Figure 5 As shown in (b).

[0092] ①Regarding MV t The Laplacian operation is used to extract the edges of the motion region; the MV value changes abruptly at the edges of the motion region, and the contour of the motion region can be extracted by taking the second derivative. Figure 5 As shown in (c). The specific operation involves performing a Laplace operation on the x and y components of the MV array respectively, and then summing them with a weight of 0.5.

[0093] ② High-pass filtering is applied to the derivative values ​​at all edges to filter out regions with high motion intensity. The magnitude of the derivative value reflects the strength of the motion; abnormal regions exhibit more intense motion than normal regions. Therefore, high-pass filtering is applied to the derivative values, retaining only those greater than the median derivative. This ultimately yields the contours of motion regions with a high probability of anomalies, such as... Figure 5 As shown in (d);

[0094] ③ Assign weights to all pixels within each region. The regions filtered in ② contain a lot of noise and linear contours, and the irregularity of these contours makes weight setting difficult. By taking the bounding rectangle of the linear contours and filtering out contours with an area smaller than 4x4, we obtain the final motion regions with a high probability of anomalies. Furthermore, only the outermost contour is selected. For example... Figure 5 As shown in (e), a number of regular regions will eventually be obtained, and the total number of corresponding regions in a frame image is denoted as N. The effect of superimposing these regions onto the original image is as follows. Figure 5 As shown in (f), each region assigns weights to its pixels based on the second derivative value at the contour. The weights of all pixels in the k-th region are denoted as w. kThe value is the average of all derivative values ​​at the region contour. After normalizing the weights of all regions within the frame, the corresponding value is assigned to the pixel value within each region, and the weight of pixels outside the region is 0. To ensure the integrity of the information of the entire error image, the weight of all pixels is multiplied by 0.1. The weight matrix obtained after the above calculation is denoted as W, and its dimension is [h, w].

[0095] ④ Weight the error image E. Use a weight matrix W to weight the error matrix E in the spatial dimension, making the error values ​​larger in motion regions with a high probability of anomalies. The weighted error image is denoted as E′.

[0096] E′(x,y)=E(x,y)+E(x,y)*W(x,y)

[0097] ⑤ Calculate anomaly scores based on a sliding window. Anomaly regions typically occupy a small proportion of the overall image. Most algorithms calculate anomaly scores by averaging the squared differences of all pixels, which weakens the anomaly score and may miss anomalies occurring within small regions. Therefore, a sliding window mechanism is used to calculate the anomaly score for each small region on the error image E′, and the maximum score is taken as the anomaly score for that frame. The position of the sliding window with the maximum score is as follows: Figure 5 As shown in (g).

[0098] Score(I t )=max(Σ m,n∈P E'(m,n))

[0099] Where P represents the area selected by the sliding window, and m and n are the coordinates of the pixels in P.

[0100] Finally, after passing through the anomaly detection network, anomaly scores for all I-frames in the video are obtained. Since the motion changes between adjacent frames are very small, the anomaly scores for unsampled frames (i.e., frames corresponding to all P-frames) are obtained by averaging the scores of two adjacent I-frames.

[0101] The inventive points protected by this invention are:

[0102] 1. The residual between the reconstructed MV and the original MV is used to reflect the magnitude of the abnormal area and the degree of abnormality. This residual is used to guide the appearance information to pay more attention to the abnormal area during prediction, so as to achieve effective correlation between motion information and appearance information.

[0103] 2. Achieve adaptive motion intensity perception and anomaly region localization. Utilizing the characteristics of motion information represented by motion intensity (MV), areas with a high probability of anomalies are initially screened. Based on this, a sliding window mechanism is introduced to further achieve precise localization of the anomaly regions.

[0104] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A video anomaly detection system based on anomaly region localization, characterized in that, include: The compressed domain video information sampling module is used to sample appearance and motion information from the compressed bitstream, which are then used as inputs to the I-frame future frame prediction module and the MV reconstruction module, respectively. The MV reconstruction module is used to reconstruct motion information. The mean square error of the reconstructed MV is included as a component of the error image. At the same time, the residual of the reconstructed MV is calculated to guide the prediction network to pay more attention to abnormal regions when predicting future frames. The I-frame future frame prediction module is used to generate predicted future frames. The mean square error of the predicted I-frame is one of the components of the error image. The mean square error of the reconstructed MV and the mean square error of the predicted I-frame are weighted to obtain the error image, which is then input into the motion intensity-based adaptive perception anomaly region localization module. The motion intensity adaptive perception abnormal region localization module is used to focus on abnormal regions and increase the score difference between normal frames and abnormal frames. The error image is obtained by weighted summing the mean square errors of the input and output results of the MV reconstruction module and the I-frame future frame prediction module. In the motion intensity-based adaptive perception anomaly region localization module, the MV corresponding to the future I-frame is subjected to a Laplacian operation to filter out highly probable anomaly regions and weighted accordingly, thereby focusing on the anomaly regions. Finally, based on the sliding window mechanism, anomaly scores are calculated for the corresponding regions on the error image, and the maximum score is taken as the anomaly score for predicting the future frame, including: For reconstructing the network and and prediction networks and The error image is obtained by weighting the mean square error in the spatial dimension. , The dimension is ,in and These represent the length and width of the I-frame, respectively. For error images To extract the edges of the motion region, a Laplace operation is performed. Specifically, the Laplace operation is performed on the x and y components of the MV array, and then the components are summed with a weight of 0.

5. High-pass filtering is applied to the derivative values ​​at all edges, retaining only the derivative values ​​greater than the median value, and finally the contour of the motion region with a high probability of anomaly is obtained. Weights are assigned to all pixels within each region. After filtering, the regions contain a lot of noise and linear contours. A bounding rectangle is taken for the linear contours, and regions with an area smaller than 4x4 are filtered out to obtain the final motion regions with a high probability of anomalies. Furthermore, only the outermost contour is selected, resulting in regular regions. The total number of regions in a frame is denoted as N. Weights are assigned to the pixels within each region based on the second derivative value at the contour. The weights of all pixels in the k-th region are denoted as... The value is the average of all derivative values ​​at the region contour; after normalizing the weights of all regions within the frame, the pixel values ​​within each region are assigned corresponding values, and the weights of pixels outside the region are 0; to ensure the integrity of the entire error image information, the weights of all pixels are multiplied by 0.1; the weight matrix obtained after the above calculation is denoted as... Dimensions are also ; For error images Perform weighting; use a weight matrix Error matrix in spatial dimension Weighting is applied to make the error values ​​larger in motion regions with a high probability of anomalies; The weighted error image is denoted as ; Anomaly scores are calculated based on a sliding window mechanism; the sliding window mechanism is used in the error image. The anomaly score for each small region is calculated, and the maximum score is taken as the anomaly score for that frame. The sliding window size is 16x16. in This represents the area selected by the sliding window. for The coordinates of the middle pixel; Finally, after passing through the anomaly detection network, the anomaly score of all I-frames in the video is obtained. Since the motion changes between adjacent frames are very small, the anomaly score of the frame corresponding to the unsampled frame, i.e., all P-frames, is obtained by averaging the scores of two adjacent I-frames, thus obtaining the anomaly score of each video frame in the video.

2. The video anomaly detection system based on anomaly region localization according to claim 1, characterized in that, In the compressed domain video information sampling module, I-frames representing appearance information and MV prior knowledge representing motion information are directly obtained from the compressed bitstream based on partial decoding, and are used as inputs to the I-frame future frame prediction module and the MV reconstruction module, respectively.

3. The video anomaly detection system based on anomaly region localization according to claim 1, characterized in that, The residual between the reconstructed MV sequence output by the MV reconstruction module and the input MV sequence is passed through the channel attention network and given greater weight to the channels that focus on anomalous regions. This weights the I-frame sequence features in the I-frame future frame prediction network, making more attention paid to anomalous regions when predicting future frames.

4. A video anomaly detection method based on anomaly region localization, characterized in that, Includes the following steps: 1) Obtain the MV sequence and I-frame sequence through sparse sampling in the compressed domain video information sampling module; 2) The MV sequence is input into the MV reconstruction module to obtain the reconstructed MV sequence, and the mean square error of the reconstructed MV and the residual of the reconstructed MV are calculated; 3) The I-frame sequence is input into the I-frame future frame prediction module. Guided by the residual of the reconstructed MV, the prediction network pays more attention to abnormal regions when generating the predicted I-frame, and the mean square error of the predicted I-frame is calculated. 4) The mean square error of the reconstructed MV and the mean square error of the predicted I-frame are weighted to obtain an error image, which is then input into the motion intensity-based adaptive perception anomaly region localization module to achieve accurate localization of anomaly regions and calculate the anomaly score of the current I-frame, including: 401) For the reconstruction of the network and and prediction networks and The error image is obtained by weighting the mean square error in the spatial dimension. , The dimension is ,in and These represent the length and width of the I-frame, respectively. 402) For error images To extract the edges of the motion region, a Laplace operation is performed. Specifically, the Laplace operation is performed on the x and y components of the MV array, and then the components are summed with a weight of 0.

5. 403) Perform high-pass filtering on the derivative values ​​at all edges, and only retain the derivative values ​​that are greater than the median value of the derivative, and finally obtain the contour of the motion region with a high probability of anomaly. 404) Assign weights to all pixels within each region; the filtered regions contain a lot of noise and linear contours. For the linear contours, take the bounding rectangle and filter out contours with an area smaller than 4x4 to obtain the final motion regions with a high probability of anomalies; furthermore, only the outermost contour is selected to obtain regular regions. The total number of corresponding regions in a frame is denoted as N; for each region, weights are assigned to the pixels within the region based on the second derivative value at the contour; the weights of all pixels in the k-th region are denoted as... The value is the average of all derivative values ​​at the region contour; after normalizing the weights of all regions within the frame, the pixel values ​​within each region are assigned corresponding values, and the weights of pixels outside the region are 0; to ensure the integrity of the entire error image information, the weights of all pixels are multiplied by 0.1; the weight matrix obtained after the above calculation is denoted as... Dimensions are also ; 405) Error Image Perform weighting; use a weight matrix Error matrix in spatial dimension Weighting is applied to make the error values ​​larger in motion regions with a high probability of anomalies; The weighted error image is denoted as ; 406) Calculate anomaly scores based on a sliding window; employing a sliding window mechanism in the error image... The anomaly score for each small region is calculated, and the maximum score is taken as the anomaly score for that frame. The sliding window size is 16x16. in This represents the area selected by the sliding window. for The coordinates of the middle pixel; 407) Finally, after passing through the anomaly detection network, the anomaly score of all I-frames in a video is obtained; since the motion changes between adjacent frames are very small, the anomaly score of the frame corresponding to the unsampled frame, i.e., all P-frames, is obtained by averaging the scores of two adjacent I-frames, thus obtaining the anomaly score of each video frame in the video.

5. The video anomaly detection method based on anomaly region localization according to claim 4, characterized in that, The specific implementation method for step 1) is as follows: 101) The original video is encoded into a compressed bitstream using the HM codec; the compressed video bitstream consists of a series of GoPs, each GoP containing an I-frame and several P-frames. The I-frame represents appearance information, and the MV extracted from the P-frames represents motion information. 102) Continuously sampling t+1 GoP I-frames is taken as an I-frame sequence, denoted as In GoP, the motion signature (MV) of the first P-frame best reflects the motion trend of the I-frame. Extracting the corresponding MV sequence is denoted as... .

6. The video anomaly detection method based on anomaly region localization according to claim 5, characterized in that, The implementation method for step 2) is as follows: 201) MV sequence The first t frames are denoted as The input is fed into the MV reconstruction module, which has an AE network structure and no skip connections, to obtain the reconstructed MV sequence. ; 202) Calculate the mean square error between the reconstructed MV sequence and the input MV sequence; 203) Calculate the residual between the reconstructed MV sequence and the input MV sequence to obtain the residual of the reconstructed MV. , .

7. A video anomaly detection method based on anomaly region localization according to claim 6, characterized in that, The implementation method for step 3) is as follows: 301) I-frame sequence The first t frames are denoted as The input is fed into the I-frame future frame prediction module, which has an AE network structure and skip connections; 302) Residual After passing through the channel attention network, channels that focus on abnormal regions are given greater weights, thus affecting the I-frame sequence after the encoder. The features are weighted to improve the prediction network's ability to predict future frames. Pay close attention to abnormal areas; 303) Calculate the mean square error of the predicted I-frame and the actual I-frame.

Images