A video anomaly detection method based on inter-block spatio-temporal relationship prediction
By employing a self-supervised task based on inter-block spatiotemporal relationship prediction, and utilizing a target extraction and fusion module and a distance constraint module, the problems of model overfitting and missed detection in video anomaly detection are solved, achieving more accurate anomaly detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING JIAOTONG UNIV
- Filing Date
- 2024-03-08
- Publication Date
- 2026-07-21
AI Technical Summary
Existing deep learning-based video anomaly detection methods are prone to overfitting and missed detections, especially when the sparse complexity of anomalous events leads to an overwhelming number of anomalous targets, resulting in poor detection performance.
A self-supervised task based on inter-block spatiotemporal relationship prediction is adopted. Spatiotemporal target blocks are extracted through the target extraction and fusion module, and the location encoding prediction is performed using a two-stream ViT model. The generalization ability of the model is enhanced by combining a distance constraint module, and deep spatiotemporal features are learned.
It improves the accuracy and robustness of video anomaly detection, reduces the false negative rate, and enhances the model's ability to detect abnormal events.
Smart Images
Figure CN118196680B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a video anomaly detection method based on inter-block spatiotemporal relationship prediction. Background Technology
[0002] Video anomaly detection aims to automatically detect and identify abnormal behaviors or events by analyzing video data. It has wide applications in the security field. By improving the accuracy and robustness of detection, it can enhance the efficiency of monitoring systems, reduce false alarm rates, and thus reduce labor costs, which has a positive economic impact on improving the overall performance of monitoring systems. Since abnormal events are rare, and the number of positive samples (anomalies) is far less than the number of negative samples, for video anomaly detection tasks, the first step is to model normal events, and then classify events that are significantly different from the learned normal behavior patterns as anomalies. Although traditional methods have achieved some success in some scenarios, they are often limited by their reliance on handcrafted features (the design process can become cumbersome due to the complexity of the problem and may not be applicable to all tasks) and poor adaptability to scene changes (the model performs poorly when facing different scenes or environmental changes). Deep learning-based video anomaly detection methods can learn more complex and higher-level feature representations from raw data through end-to-end learning, but early methods, such as those based on future frame prediction, ... Figure 8 As shown, and the detection method based on the reconstructed frame, such as Figure 9 As shown, due to the powerful learning ability of convolutional neural networks (CNN), they often face the problem of overfitting. On the other hand, since anomalies often appear in small areas of the video, the complexity of the anomaly events can easily overwhelm the anomaly targets, leading to missed detections by the model.
[0003] Therefore, in order to prevent the model from overfitting, some self-supervised video anomaly detection methods have been proposed, such as the Jigsaw method.
[0004] The model architecture of the Jigsaw method is as follows: Figure 10 As shown, the model first uses a pre-trained YOLO algorithm to extract targets from the video. Then, based on a self-supervised spatiotemporal jigsaw puzzle task, it decouples the spatial and temporal dimensions, segmenting and shuffling the extracted target blocks spatially and temporally. The 3D convolutional network outputs the correct jigsaw puzzle result. During the detection phase, the model identifies targets with poor puzzle completion as anomalous targets, and the frames containing these targets are designated as anomalous frames. This model captures highly differentiated appearance and motion features based on this challenging self-supervised task, thereby enhancing the model's generalization ability. Summary of the Invention
[0005] The embodiments of the present invention provide a video anomaly detection method based on inter-block spatiotemporal relationship prediction, which is used to solve the technical problems existing in the prior art.
[0006] To achieve the above objectives, the present invention adopts the following technical solution.
[0007] A video anomaly detection method based on inter-block spatiotemporal relationship prediction includes:
[0008] S1 inputs the obtained video sequence into the target extraction and fusion module of the detection model for processing to obtain a set of spatiotemporal target boxes;
[0009] S2 is based on a spatiotemporal target set. It divides the target into spatial and temporal blocks by using the block order prediction module of the detection model.
[0010] S3 encodes the positions of spatial blocks and temporal embeddings by randomly sorting them through the linear mapping layer of the detection model;
[0011] S4 uses the position-encoded sequence after step S3 as the labels s and t for the sequential prediction self-supervised task, and trains the detection model based on the labels s and t and in conjunction with the distance constraint module.
[0012] S5 inputs the spatial and temporal blocks, after location encoding, into the two deformers of the trained detection model for processing to obtain the sequential prediction matrix M. s and M T Based on the sequential prediction matrix M s and M T By calculating pattern scores, anomaly scores for video sequences are obtained;
[0013] S6 detects anomalies in video sequences by using anomaly scores.
[0014] Preferably, step S1 includes:
[0015] The YOLO v3 module is used to extract the set of visual bounding boxes for video frames in a video sequence.
[0016] The action target bounding box set is obtained by calculating the inter-frame gradient difference between adjacent video frames in the video sequence.
[0017] Perform a union operation on the set of appearance target boxes and the set of action target boxes to obtain a set of spatiotemporal target boxes.
[0018] Preferably, in step S4, the operation of the distance constraint module includes:
[0019] The target block is divided into equal-sized spatial blocks and temporal blocks based on spatial and temporal dimensions.
[0020] Establish the Canberra distance matrix D for each pair of spatial blocks. Canberra Establish a cosine distance matrix D for each pair of time blocks. Cosine ;
[0021] For a pair of spatial blocks, take the edge vector p in the h direction of vector block i. i Take the edge vector p of spatial block j in the opposite direction to h. j The difference is used to obtain the distance c(p) in the opposite direction. i ,p j ); where c(p) i ,p j ) = p i (h,1,c)-p j (-h,1,c);
[0022] The correlation distance between spatial block pairs is calculated separately in the four directions: up, down, left, and right. Perform this calculation process on all time blocks, and combine the results to obtain the spatial block relation matrix D. Canberra ;
[0023] For a pair of time blocks, directly use that time block as the computation vector p. i p j Through formula The correlation distance between a pair of time blocks is obtained. This calculation process is performed on all time blocks, and the results are combined to obtain the relationship matrix D between time blocks. Cosine ;
[0024] The spatial block relation matrix D Canberra Inter-time block relationship matrix D Cosine As a label for the distance constraint module, it enables the model to also handle the spatial block relation matrix D during operation. Canberra Inter-time block relationship matrix D Cosine Make predictions and obtain
[0025] pass And combine L2 loss to construct the loss function of the distance constraint module The loss function of this distance constraint module is used for training the detection model.
[0026] Preferably, in step S5, the spatial block and temporal block after position encoding are respectively input into the two deformers of the trained detection model for processing to obtain the sequential prediction matrix M. s and M T The process includes:
[0027] Predicting matrix by block order The order prediction matrix M is calculated separately.s and M T ;
[0028] In step S5, based on the sequential prediction matrix M s M T The process of obtaining anomaly scores for video sequences by calculating pattern scores includes:
[0029] The sequential prediction matrix M s and M T The minimum value on the diagonal is taken as the target-level regularity score, and the formula is used to... Calculate the minimum regularity score of the target in the obtained video frame, and take the minimum regularity score of the target in the video frame as the frame-level regularity score;
[0030] Through The target's pattern score R = ω is obtained by weighted summation of the two frame-level pattern scores. s *R s +ω t *R t Equation R = ω s *R s +ω t *R t In the middle, ω s ω t As a preset constant, it is set to 0.5;
[0031] Based on the target pattern score, the anomaly score S of the video sequence is calculated using the formula S = 1 - R.
[0032] Step S6 includes:
[0033] The presence of anomalies in a video sequence is determined by comparing the anomaly score S with a preset threshold.
[0034] As can be seen from the technical solutions provided by the embodiments of the present invention described above,
[0035] This invention provides a video anomaly detection method based on inter-block spatiotemporal relationship prediction, comprising: inputting the obtained video sequence into the target extraction and fusion module of a detection model for processing to obtain a set of spatiotemporal target boxes; based on the spatiotemporal target set, dividing the video into blocks according to the spatial and temporal dimensions through the block order prediction module of the detection model to obtain spatial blocks and temporal blocks; using the linear mapping layer of the detection model to randomly sort and encode the positions of the spatial blocks and temporal embeddings; using the position-encoded sequences as labels s and t for a self-supervised order prediction task, and training the detection model in conjunction with a distance constraint module; and inputting the position-encoded spatial blocks and temporal blocks into the two-way deformers of the trained detection model for processing to obtain an order prediction matrix M. s and M TAnomaly scores are obtained for video sequences by calculating pattern scores; anomaly detection is performed on the video sequences based on these anomaly scores.
[0036] This invention designs a self-supervised task based on the prediction of spatiotemporal relationships between patches in ViT. Due to the model's missed detection problem, a target extraction and fusion module is used to extract spatiotemporal target blocks, which are then input in blocks. A two-stream ViT model is used to predict out-of-order positional codes to learn the deep spatiotemporal features of the targets. This invention also designs a distance constraint module. This module calculates the Canberra distance and cosine distance for spatial and temporal blocks respectively, representing the correlation between blocks as an inter-block relationship matrix. The model predicts this correlation to enhance its generalization ability, thereby constraining the model to learn accurate spatiotemporal features and achieving more precise video anomaly detection.
[0037] The method provided by this invention also has the following advantages:
[0038] The Jigsaw method directly extracts the target set through YOLO. However, since the pre-trained YOLO may not be able to completely extract all potentially abnormal targets in the video frame, there is a certain risk of missing targets. This invention uses a target extraction and fusion module to extract a target set containing appearance and action information. The targets detected by gradient images are fused with the targets detected by YOLO to complete the target set and greatly reduce the risk of missing targets.
[0039] This invention is the first to propose a two-stream ViT model structure for a self-supervised task of predicting block spatiotemporal relationships. It cleverly utilizes the characteristics of ViT to transform the jigsaw puzzle task into a positional encoding prediction task, making it applicable to spatiotemporal two-stream ViT models. Furthermore, for the 3D convolutional neural network used in Jigsaw, ViT enhances the model's global and positional awareness capabilities, enabling it to learn more global features and improve anomaly detection accuracy. Additionally, this invention includes a distance constraint module to constrain the model to learn accurate spatiotemporal features, achieving more precise video anomaly detection.
[0040] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description
[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0042] Figure 1 The flowchart of a video anomaly detection method based on inter-block spatiotemporal relationship prediction provided by the present invention is shown below.
[0043] Figure 2 The detection model structure diagram of a video anomaly detection method based on inter-block spatiotemporal relationship prediction provided by the present invention is shown below.
[0044] Figure 3 A flowchart illustrating the workflow of the target extraction and fusion module involved in a video anomaly detection method based on inter-block spatiotemporal relationship prediction provided by this invention.
[0045] Figure 4 The processing effect diagram of the target extraction and fusion module involved in the video anomaly detection method based on inter-block spatiotemporal relationship prediction provided by the present invention is shown in the figure.
[0046] Figure 5 This is a schematic diagram of the anomaly scoring results in the UCSDPed2 dataset for a video anomaly detection method based on inter-block spatiotemporal relationship prediction provided by the present invention.
[0047] Figure 6 This is a schematic diagram of the anomaly scoring results in the Avenue dataset for a video anomaly detection method based on inter-block spatiotemporal relationship prediction provided by the present invention.
[0048] Figure 7 This is a schematic diagram of the anomaly scoring results of a video anomaly detection method based on inter-block spatiotemporal relationship prediction provided by the present invention in the ShanghaiTech dataset.
[0049] Figure 8 A comparison diagram of existing prediction-based video anomaly detection model structures;
[0050] Figure 9 A comparison diagram of existing reconstruction-based video anomaly model structures;
[0051] Figure 10 This is a comparison diagram of existing supervised video anomaly detection model structures. Detailed Implementation
[0052] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0053] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.
[0054] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.
[0055] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.
[0056] This invention provides a method for video anomaly detection, which addresses the following technical problems existing in the prior art.
[0057] Current deep learning-based video anomaly detection methods, such as those based on future frame prediction and those based on reconstructed frames, often face the problem of model overfitting due to the powerful learning capabilities of convolutional neural networks (CNNs). Furthermore, since anomalies often appear in small areas of the video, the sparse complexity of the anomaly events can easily overwhelm the anomaly targets, leading to missed detections by the model.
[0058] This invention focuses on addressing the sparsity and complexity of anomalous events and the overfitting problem of anomaly detection models, proposing a self-supervised video anomaly detection method. To address the sparsity and complexity of anomalous events, this invention designs a target extraction and fusion module to extract potentially anomalous targets, reducing the risk of missed detections. To address the overfitting problem, we propose a self-supervised task based on ViT (Vibrational Transformation) block-to-block spatiotemporal relationship prediction, employing a two-stream network structure. By predicting the correct sequence of target blocks, it learns the deep spatiotemporal features of the video. This self-supervised task, while challenging, enhances the model's generalization ability and prevents overfitting.
[0059] Furthermore, to enable the model to learn the correct inter-block spatiotemporal relationships, this invention also incorporates an inter-block distance constraint module. This module calculates the Canberra distance and cosine distance between spatial and temporal blocks respectively to represent their correlation, obtaining an inter-block relationship matrix that constrains the network to make correct block arrangement predictions. Block order prediction and inter-block relationship prediction are performed simultaneously in the temporal and spatial dimensions, learning the spatiotemporal features of events while reducing the risk of model overfitting.
[0060] See Figure 1 This invention provides a video anomaly detection method based on inter-block spatiotemporal relationship prediction, comprising the following steps:
[0061] S1 inputs the obtained video sequence into the target extraction and fusion module of the detection model for processing to obtain a set of spatiotemporal target boxes;
[0062] S2 is based on a spatiotemporal target set. It divides the target into spatial and temporal blocks by using the block order prediction module of the detection model.
[0063] S3 encodes the positions of spatial blocks and temporal embeddings by randomly sorting them through the linear mapping layer of the detection model;
[0064] S4 uses the position-encoded sequence after step S3 as the label s,t for the sequential prediction self-supervised task, and trains the detection model based on the label s,t and in conjunction with the distance constraint module.
[0065] S5 inputs the spatial and temporal blocks, after location encoding, into the two deformers of the trained detection model for processing to obtain the sequential prediction matrix M. s M T Based on the sequential prediction matrix M s M T By calculating the pattern score, the anomaly score of the video sequence is obtained;
[0066] S6 detects and identifies abnormal frames (behaviors or events) in a video sequence by using the anomaly score of the video sequence.
[0067] This invention is the first to design a self-supervised task based on the prediction of spatiotemporal relationships between pacths in ViT. Due to the missed detection problem of the detection model, this invention uses a target extraction and fusion module to extract spatiotemporal target blocks, inputs them in blocks, and uses a two-stream ViT model to predict disordered position codes to learn the deep spatiotemporal features of the target.
[0068] For the detection model, this invention also designs a distance constraint module. This module calculates the Canberra distance and cosine distance of spatial blocks and temporal blocks respectively, and represents the correlation between blocks as an inter-block relationship matrix. The module predicts this correlation through the model to enhance the model's generalization ability, thereby constraining the model to learn accurate spatiotemporal features and achieving more accurate video anomaly detection.
[0069] In the preferred embodiment provided by the present invention, the specific execution process of each step is as follows.
[0070] This invention proposes a video anomaly detection method (PRP) based on inter-block spatiotemporal relationship prediction, the specific model structure of which is as follows: Figure 2 As shown, the model consists of a target extraction and fusion module and a block order prediction module. Figure 2 The model consists of three parts: a temporal block partitioning module, a spatial block partitioning module, a linear mapping layer, and a sequence labeling module, as well as a distance constraint module (the encoder and linear layer in the diagram). During the training phase, the target set obtained from the target extraction and fusion module is trained jointly by the block sequence prediction module and the distance constraint module. During the inference phase, the block sequence prediction matrix obtained from the block sequence prediction module is used to calculate anomaly scores to identify anomalies. This method aims to address the problem of missed detections of anomalous events and enhance the model's generalization ability while preventing overfitting.
[0071] First, the present invention inputs the video sequence into the target extraction and fusion module designed in this invention, as shown in the flowchart below. Figure 3 As shown, this invention first extracts the set of appearance object bounding boxes from the frames using YOLO v3 (submodule) pre-trained on the COCO dataset. Simultaneously, it calculates the gradient difference between adjacent frames and extracts the set of motion object bounding boxes from the gradient difference image. Finally, it combines the two sets of object bounding boxes to obtain the spatiotemporal object bounding box set. The extraction effect is as follows: Figure 4 As shown, this set includes both appearance and motion targets, supplementing the input set of general target-level video anomaly detection methods to enable the model to learn more comprehensive target features. Subsequently, this invention crops n frames before and after the corresponding target bounding box position, scaling them to the same size target data block, which serves as the model's input.
[0072] Next, the spatiotemporal target block is divided into spatial and temporal patches, respectively. These patches are then processed through a linear mapping layer to embed randomly ordered positional codes, which are used as the sequence labels s and t for the block sequence prediction self-supervised task. The encoded temporal and spatial patches are then input into two separate Transformers, allowing the model to predict the output block sequence prediction matrix M. s M TThe block order prediction matrix is represented as follows:
[0073] Where p A,2 This represents the probability that the position code of patch A is 2.
[0074] The rest follow the same pattern.
[0075] Next, this invention calculates the cross-entropy loss using the prediction results and the order labels: This is how training is conducted.
[0076] Simultaneously, this invention designs a distance constraint module to constrain the learning of the model. This invention first calculates the inter-patch relationship (IPR) matrix of the spatiotemporal blocks, the specific calculation process of which is as follows:
[0077]
[0078] D obtained through calculation Canberra D Cosine As a label for the constraint module of this invention, the model simultaneously predicts the inter-block relationship matrix of this invention during output, thus obtaining... We take L2 loss as the loss function for the distance constraint module:
[0079] In the event of anomalies during reasoning, this invention will use the inter-block relationship prediction matrix M s M T The minimum value on the diagonal is taken as the target-level pattern score: Therefore, if the model mispredicts even one location label, the regularity score for that target will be very low. This invention then takes the minimum regularity score of the target within a frame as the frame-level regularity score. This means that the presence of even one anomalous object in a video frame will directly affect the frame's pattern score. The final pattern score is obtained by weighted summation of the pattern scores from both channels (spatial block order prediction and temporal block order prediction): R = ω s *R s +ω t *R t , where ω s ω t In this invention, all values are set to 0.5. Therefore, if any one prediction task fails, it will affect the overall regularity score. The final anomaly score S is S = 1 - R. This invention can set a threshold; when S is greater than the threshold, the frame is considered anomaly, thus detecting abnormal events. This anomaly score calculation method can amplify the impact of abnormal events on the anomaly score, enhancing the model's accuracy in detecting abnormal events.
[0080] The method of this invention achieves AUROCs of 98.7%, 92.5%, and 80.4% on three currently popular video anomaly detection datasets: UCSD Ped2, Avenue, and ShanghaiTech. It surpasses most mainstream video anomaly detection methods on all three datasets and achieves the best AUROC performance on the Avenue and ShanghaiTech datasets. Figure 5 , 6 Figures 7 and 8 represent the anomaly scores for each frame of test instances in the three datasets UCSD Ped2, Avenue, and ShanghaiTech, after being trained using the detection model of this invention. The shaded areas represent the frame intervals of anomalies, and the curves represent the anomaly scores. It can be seen that when anomalies (such as objects being thrown, running, riding bicycles, or vehicles appearing on the sidewalk) occur, the anomaly scores show a sharp upward trend.
[0081] The video anomaly detection method (PRP) based on inter-block spatiotemporal relationship prediction proposed in this invention can be trained and tested on a single RTX 3090 GPU.
[0082] In summary, this invention provides a video anomaly detection method based on inter-block spatiotemporal relationship prediction, comprising: inputting the obtained video sequence into the target extraction and fusion module of a detection model for processing to obtain a set of spatiotemporal target boxes; based on the spatiotemporal target set, dividing the video into blocks from the spatial and temporal dimensions respectively through the block order prediction module of the detection model to obtain spatial blocks and temporal blocks; using the linear mapping layer of the detection model to randomly sort the spatial blocks and temporal embeddings into positional encoding; using the position-encoded sequence as labels s and t for a self-supervised task of order prediction, and training the detection model in conjunction with a distance constraint module; and inputting the position-encoded spatial blocks and temporal blocks into the two-way deformers of the trained detection model for processing to obtain an order prediction matrix M. s and M T Anomaly scores are obtained for video sequences by calculating pattern scores; anomaly detection is performed on the video sequences based on these anomaly scores.
[0083] This invention designs a self-supervised task based on the prediction of spatiotemporal relationships between patches in ViT. Due to the model's missed detection problem, a target extraction and fusion module is used to extract spatiotemporal target blocks, which are then input in blocks. A two-stream ViT model is used to predict out-of-order positional codes to learn the deep spatiotemporal features of the targets. This invention also designs a distance constraint module. This module calculates the Canberra distance and cosine distance for spatial and temporal blocks respectively, representing the correlation between blocks as an inter-block relationship matrix. The model predicts this correlation to enhance its generalization ability, thereby constraining the model to learn accurate spatiotemporal features and achieving more precise video anomaly detection.
[0084] The method provided by this invention also has the following advantages:
[0085] The Jigsaw method directly extracts the target set through YOLO. However, since the pre-trained YOLO may not be able to completely extract all potentially abnormal targets in the video frame, there is a certain risk of missing targets. This invention uses a target extraction and fusion module to extract a target set containing appearance and action information. The targets detected by gradient images are fused with the targets detected by YOLO to complete the target set and greatly reduce the risk of missing targets.
[0086] This invention is the first to propose a two-stream ViT model structure for a self-supervised task of predicting block spatiotemporal relationships. It cleverly utilizes the characteristics of ViT to transform the jigsaw puzzle task into a positional encoding prediction task, making it applicable to spatiotemporal two-stream ViT models. Furthermore, for the 3D convolutional neural network used in Jigsaw, ViT enhances the model's global and positional awareness capabilities, enabling it to learn more global features and improve anomaly detection accuracy. Additionally, this invention includes a distance constraint module to constrain the model to learn accurate spatiotemporal features, achieving more precise video anomaly detection.
[0087] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.
[0088] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.
[0089] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0090] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A video anomaly detection method based on inter-block spatiotemporal relationship prediction, characterized in that, include: S1 inputs the obtained video sequence into the target extraction and fusion module of the detection model for processing to obtain a set of spatiotemporal target boxes; S2 is based on a spatiotemporal target set. It divides the target into spatial and temporal blocks by using the block order prediction module of the detection model. S3 encodes the position of spatial blocks and temporal embeddings by detecting the linear mapping layer of the model; S4 uses the position-encoded sequence after step S3 as the label for the sequential prediction self-supervised task. and Based on this tag and The detection model is trained using a distance constraint module; the working process of the distance constraint module includes: The target block is divided into equal-sized spatial blocks and temporal blocks based on spatial and temporal dimensions. Establish the Canberra distance matrix for each pair of spatial blocks. Establish a cosine distance matrix for each pair of time blocks. ; For a pair of spatial blocks, take the vector block. of Edge vectors in the direction Take space block and Edge vectors in opposite directions The difference is used to obtain the distance in the opposite direction. ;in, ; The correlation distance between spatial block pairs is calculated separately in the four directions: up, down, left, and right. This calculation process is performed on all time blocks, and the results are combined to obtain the spatial block relation matrix. ; For a pair of time blocks, directly use that time block as the computation vector. , Through formula The correlation distance between a pair of time blocks is obtained. This calculation process is performed on all time blocks, and the results are combined to obtain the relationship matrix between time blocks. ; The spatial block relationship matrix Relationship matrix between time blocks As a label for the distance constraint module, it enables the model to also handle the relationship matrix between spatial blocks during operation. Relationship matrix between time blocks Make predictions and obtain , ; pass , And combine L2 loss to construct the loss function of the distance constraint module The loss function of this distance constraint module is used for training the detection model. S5 inputs the spatial and temporal blocks, after location encoding, into the two deformers of the trained detection model for processing to obtain the sequential prediction matrix. and Based on the sequential prediction matrix and By calculating pattern scores, anomaly scores for video sequences are obtained; S6 detects anomalies in video sequences by using anomaly scores.
2. The method according to claim 1, characterized in that, Step S1 includes: The YOLO v3 module is used to extract the set of visual bounding boxes for video frames in a video sequence. The action target bounding box set is obtained by calculating the inter-frame gradient difference between adjacent video frames in the video sequence. Perform a union operation on the set of appearance target boxes and the set of action target boxes to obtain a set of spatiotemporal target boxes.
3. The method according to claim 1, characterized in that, In step S5, the spatial and temporal blocks encoded with location are respectively input into the two deformers of the trained detection model for processing to obtain the sequential prediction matrix. and The process includes: Predicting matrix by block order Calculate the order prediction matrix separately and In the formula, Indicates the first The probability that the position code of the block patch is 2; In step S5, the step based on the sequential prediction matrix... , The process of obtaining anomaly scores for video sequences by calculating pattern scores includes: Order prediction matrix and The minimum value on the diagonal is taken as the target-level regularity score, and the formula is used to... Calculate the minimum regularity score of the target in the obtained video frame, and take the minimum regularity score of the target in the video frame as the frame-level regularity score; Through The target's pattern score is obtained by weighted summation of the two frame-level pattern scores. ;Mode middle, , As a preset constant, it is set to 0.5; Based on the target pattern score, through the formula Calculate the anomaly score of the obtained video sequence S ; Step S6 includes: By abnormal scores S The video sequence is compared with a preset threshold to determine if there are any anomalies.