A video anomaly detection method based on a dual cross-attention autoencoder

Abstract

This invention discloses a video anomaly detection method based on a dual-cross-attention autoencoder, comprising: preprocessing the original video set by splitting it into multiple frame-level sequences according to different video samples; using frame difference features and optical flow features to represent the motion features of the target, and using HOG features to describe the appearance features, extracting the motion features and appearance features of the video frames; inputting the motion features and appearance features extracted from the training samples into an autoencoder network incorporating a dual-cross-attention module for training, establishing a video anomaly detection model; and obtaining the reconstruction error of the test samples through the video anomaly detection model to obtain the video anomaly detection result. This invention introduces a dual-cross-attention module, which can effectively improve the accuracy of video anomaly event detection by associating local features with global features.

Description

Technical Field

[0001] This invention belongs to the field of video anomaly detection technology, specifically relating to a video anomaly detection method based on a dual cross-attention autoencoder. Background Technology

[0002] With the expanding application of surveillance cameras and the increasing volume of video footage, existing video anomaly detection methods are no longer sufficient to meet the demands. Researchers in the field of computer vision are exploring innovations and improvements to this technology. Video anomaly detection faces many challenges because the probability of abnormal events is low, and the determination of anomalies depends heavily on the scene. To address these challenges, researchers typically propose corresponding video anomaly detection methods and build models for specific anomalies. For example, to detect whether pedestrians are abnormal, researchers often use methods such as human trajectory or motion recognition. However, in non-open scenes, occlusion of pedestrians or vehicles can affect the accuracy of anomaly detection because existing detection methods can only extract unobstructed feature information. Furthermore, in real-world scenarios, most events are normal, with only a very small number being abnormal. This can lead to problems such as the scarcity of abnormal events and the easy omission of information-intensive features, thus affecting the accuracy of anomaly detection.

[0003] Therefore, a new technological solution is needed to address these issues. Summary of the Invention

[0004] Purpose of the invention: In order to overcome the problems of the scarcity of abnormal events in the processed videos and the easy omission of information-dense features in the existing technology, a video anomaly detection method based on dual cross-attention autoencoder is provided. It introduces a dual cross-attention module, which can effectively improve the accuracy of video anomaly detection by associating local features with global features.

[0005] Technical Solution: To achieve the above objectives, this invention provides a video anomaly detection method based on a dual cross-attention autoencoder, comprising the following steps:

[0006] S1: Preprocess the original video set by splitting it into multiple frame-level sequences based on different video samples;

[0007] S2: Use frame difference features and optical flow features to represent the motion features of the target, use HOG features to describe the appearance features, and extract the motion features and appearance features of video frames.

[0008] S3: Input the motion features and appearance features extracted from the training samples into the autoencoder network with a dual cross-attention module for training to establish a video anomaly detection model;

[0009] S4: Obtain the reconstruction error of the test sample through the video anomaly detection model to get the video anomaly detection result.

[0010] Furthermore, the method for extracting motion features in step S2 is as follows: using the frame difference method, the pixel values ​​of corresponding frames in the obtained image sequence are subtracted to obtain a difference image, which is then binarized to obtain the position of the moving target; adjacent video frames are input into Flownet2 to obtain the optical flow features of the video frames, and the frame difference features are combined with the optical flow features to obtain the motion features.

[0011] Furthermore, the method for extracting appearance features in step S2 is as follows: using the HOG feature extraction method, the appearance and shape of the video frame to be detected are described by the gradient or edge direction density distribution, and the appearance features are the appearance and shape of the local target.

[0012] Furthermore, in step S3, the autoencoder network includes an encoder, a decoder, and a dual cross-attention module. The encoder is composed of two convolutional layers and one pooling layer, and the decoder is composed of one upsampling layer and two convolutional layers.

[0013] The autoencoder network operates as follows: the obtained features are uniformly compressed into a 128×128×3 image and input into the encoder. The encoder consists of two 3×3 convolutional layers and a 2×2 pooling layer, repeated over and over. Each downsampling iteration halves the size of the previous feature map and doubles the number of channels, resulting in a doubled depth feature map. This process is repeated four times to obtain depth features at different scales. The 3×3 convolutional layers in the encoder can add local contextual information to the input feature map through weighted stacking, resulting in multiple lower-dimensional feature vectors and ultimately, deep features. These features, compared to the input feature map, further integrate a global perspective from local features.

[0014] To facilitate the construction of new feature maps using the decoder, a theoretical receptive field is added to the encoder, making the trained model more accurate. The formula for calculating the receptive field is:

[0015] RF k-1 =(RF) k -1)×s k +f k (3)

[0016] In the formula, RF k For the receptive field of the k-th layer, f k The kernel size of the k-th layer is given by kernel_size = 3. k Let stride be the convolutional stride of the k-th layer, with stride = 1. Therefore, the receptive field RF of the k-th layer... k It is determined by the receptive field of the (k-1)th layer and the convolution stride sk and kernel size f k Determined jointly. Convolution stride s k The kernel size f determines the stride by which the receptive field of the previous layer moves within the current layer. k This determines the spatial extent of the receptive field of the current layer. By calculating the receptive field size and adding a theoretical receptive field to the network, we can help the network better understand the contextual information of the input image, thereby improving the model's accuracy and expressive power.

[0017] Secondly, the obtained deep features are input into the dual cross-attention module, so that the feature map obtains richer and denser contextual information, resulting in feature maps of different sizes with close correlation between global and local features.

[0018] The decoder consists of a 2×2 upsampling layer and two 3×3 convolutional layers. This structure reconstructs the depth features and then uses a skip connection method to connect the reconstructed features with the information-dense feature map before outputting an image of the same size as the input feature map, thereby generating a high-quality reconstructed frame.

[0019] Furthermore, the dual cross-attention module in step S3 consists of two cross-attention modules, and its operation method is as follows: after the feature map is input into the cross-attention module, the context information in the horizontal and vertical directions will be collected to generate an attention map (upper part) and an adaptation feature map (lower part) respectively, and the two are aggregated by weighted sum to obtain a new feature map; the new feature map is input into the next cross-attention module, and each pixel in the feature map will collect information from all other pixels to enhance the global association of pixels; after two cross-attention module operations, each pixel is finally associated with the information of all pixels in the feature map, where all cross-attention modules share parameters to reduce additional parameters.

[0020] Furthermore, the operation flow of the dual-cross attention module in step S3 is as follows:

[0021] A1: After inputting the feature map x into the cross-attention module, it first enters two different 1×1 convolutional layers for dimensionality reduction, generating two feature maps h1 and h2 respectively;

[0022] A2: After obtaining feature maps h1 and h2, attention maps are obtained through Affinity and Softmax operations, as shown in formulas (4) and (5):

[0023]

[0024]

[0025] In the formula, d i,u h1u with h 2iu The degree of correlation, h 1u The vector is represented as a vector at each position u in the spatial dimension of the feature map h1. Similarly, the set of feature vectors v is obtained from h2. u i represents the value in v u The i-th element in the array, where p is the number of categories, and z is the number of categories. a 'a' is the output, and 'a' is the input.

[0026] A3: Input the feature map into another 1×1 convolutional layer for dimensionality reduction to generate a feature map h3. Then, combine it with the attention map through aggregation to obtain a feature map h with distant contextual information, as shown in the following formula:

[0027]

[0028] In the formula, A iu Represents the scalar value at channel i and position u, i∈|y u |,y u Let x represent the set of eigenvectors on feature map h3. u A vector representing each position u in the spatial dimension of the feature map x;

[0029] A4: Finally, the feature map is connected with the feature map that has obtained distant context information to obtain a feature map h that is associated with long-distance context information and where each pixel exchanges and synchronizes information in a cross-shaped manner.

[0030] Furthermore, in step S3, the video anomaly detection model is optimized using the cross-entropy loss function and the category consistency loss function;

[0031] The expression for the cross-entropy loss function is:

[0032]

[0033] In the formula, N is the number of classification categories, i is the input sample, and p i This represents the probability that sample i is predicted to be of the positive class;

[0034] Category consistency loss is caused by tensile loss L var Thrust loss L dis and regularization loss L reg The three components are composed of the following functional expressions:

[0035]

[0036]

[0037]

[0038] In the formula, C is the set of classes in the input image, and N... c h represents the number of valid elements in C. i Let μ be the feature vector at spatial location i. c For the cluster centers of category c∈C, For fragment distance function, and The formula is shown below:

[0039]

[0040]

[0041] In the formula, and Represents the boundary value, set

[0042] The final category consistency loss function is shown below:

[0043] L cc =ρL var +τL dis +γL reg (13)

[0044] In the formula, ρ=τ=1, γ=0.001;

[0045] The loss value of a single feature map can be obtained from formulas (7) and (13):

[0046] L = L ee +L ec (14)

[0047] Ultimately, the reconstruction error between the input features and the reconstructed features can be obtained. The smaller the reconstruction error during training, the closer the reconstructed features are to the input features, and the higher the accuracy of the model obtained after training.

[0048] Further, step S4 specifically includes:

[0049] The optical flow and HOG features of frame t are respectively fed into the trained DCAE to calculate the reconstruction error. When the reconstruction error is large, the test sample contains abnormal events; otherwise, it is normal. The HOG feature reconstruction error L is then obtained. hog and frame stream reconstruction error L ff Then, the total reconstruction error is calculated using formula (15):

[0050] Loss=αL hog +βL ff ,α+β=1 (15)

[0051] Formula (16) can be used to determine whether an abnormal event has occurred in a video frame:

[0052]

[0053] Where θ is the reconstruction error threshold. If Loss > θ, an abnormal event has occurred in the frame. If Loss ≤ θ, it means that all events are normal.

[0054] This invention proposes a feature fusion extraction algorithm combining frame difference and optical flow methods—the frame flow method—and uses Histogram of Oriented Gradients (HOG) to extract texture information from videos to obtain appearance features. The frame flow method using neural networks can effectively extract relatively complete motion features, avoiding information loss caused by slow-moving targets. HOG can clearly display the texture information contained in the video, and this feature extraction algorithm is relatively simple. Combining multiple feature extraction algorithms can obtain complete video features, thereby improving the accuracy of video anomaly detection models. In real-world scenarios, most events are normal, with only a very small number of abnormal events. Therefore, this invention uses a skip convolutional autoencoder and trains the model using only normal events. By reconstructing normal events, the reconstructed image is made similar to the original video frame, so that the model cannot reconstruct abnormal frames during testing, i.e., the abnormal image is dissimilar to the reconstructed image. Since existing autoencoders can only increase the correlation of feature information between adjacent pixels in a video frame, a dual cross-attention module is introduced to better associate global contextual feature information with local features and reduce time and space complexity.

[0055] The specific process of this invention includes: first, preprocessing the video set to extract appearance and motion features from the video frames; then, designing a dual-cross attention module integrated into the autoencoder to enable the feature maps to better associate with global features within the autoencoder; second, feeding the extracted features into their respective autoencoders to learn normal behavior, so that video frames containing normal events can be reconstructed by the model, while video frames containing abnormal events cannot be reconstructed, because the model only learns normal patterns and has no awareness of abnormal patterns; finally, using the detection model to obtain the reconstruction error of each video frame to determine abnormal events. Since video frames containing abnormal events cannot be reconstructed, and the output obtained by the model from abnormal video frames is inconsistent with the input abnormal video frames, the detection result can be obtained by calculating the reconstruction error between the input and output frames.

[0056] Beneficial Effects: Compared with existing technologies, this invention improves the correlation between pixels in the feature map and global information by introducing a dual-cross-attention module into the autoencoder, thereby enhancing the training effect and detection rate of the autoencoder network. The dual-cross-attention autoencoder exhibits higher detection accuracy and lower isomorphic error rate. While maintaining accuracy, the dual-cross-attention autoencoder removes invalid and useless information from the dataset before training and testing, and uses continuous sparse attention maps instead of ordinary single dense attention maps. This significantly reduces computational resources used, thus ensuring good running speed of the entire method. Attached Figure Description

[0057] Figure 1 This is a flowchart of the process of the method of the present invention;

[0058] Figure 2 This is a flowchart of the feature fusion process in this invention;

[0059] Figure 3 This is a structural diagram of the dual-cross attention autoencoder of the present invention;

[0060] Figure 4 This is a diagram of the cross-attention module;

[0061] Figure 5 A flowchart for cross-attention;

[0062] Figure 6 ROC curve for CUHK Avenue;

[0063] Figure 7 This is the ROC curve of UCSD Ped2. Detailed Implementation

[0064] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0065] This invention provides a video anomaly detection method based on a dual cross-attention autoencoder, such as... Figure 1 As shown, it includes the following steps:

[0066] I. Preprocessing the original video set:

[0067] The original video set is cleaned and split, and then divided into multiple frame-level sequences {frame1, frame2, ..., frame...} based on different video samples. n}

[0068] II. Extracting motion and appearance features from video frames:

[0069] The motion features of the target are represented by frame difference features and optical flow features, and the appearance features are described by HOG features. In order to extract video features more completely, the appearance features are extracted by HOG.

[0070] Motion features are extracted from a limited number of video volumes based on frame difference features and optical flow features. Specifically, the frame difference method is used to subtract the corresponding pixel values ​​of adjacent frames in the obtained image sequence to obtain a difference image, which is then binarized to obtain the position of the moving target. Adjacent video frames are input into Flownet2 to obtain the optical flow features of the video frames. The frame difference features are combined with the optical flow features to obtain the motion features.

[0071] Frame difference features and optical flow features can accurately describe motion anomalies, such as crowd panic, running, and other sudden changes;

[0072] By fusing frame difference features and optical flow features, a new fused feature, called frame flow feature, is obtained. Figure 2 Frame stream features can effectively reflect abnormal object motion, reducing the detection error rate. The formulas for optical flow features and frame difference features are as follows:

[0073]

[0074] Ifd(x,y)=|frame fn (x,y)-frame fn-1 (x,y)| (2)

[0075] Where Countp is the total number of pixels in the video frame. and These correspond to the horizontal and vertical components of optical flow, respectively. Similarly, f n Indicates the number of frames;

[0076] Appearance features are the appearance and shape of local targets. The HOG feature extraction method is used to describe the appearance and shape of the video frame to be detected well by gradient or edge direction density distribution.

[0077] 3. Input the motion features and appearance features extracted from the training samples into an autoencoder network with a dual cross-attention module for training to establish a video anomaly detection model:

[0078] like Figure 3 As shown, the present invention designs an autoencoder network DCAE, which mainly includes an encoder, a decoder and a dual cross-attention module;

[0079] The encoder consists of two 3×3 convolutional layers and one 2×2 pooling layer, repeated over and over. Its operating principle is as follows: the obtained features are uniformly compressed into a 128×128×3 image and input into the encoder. With each downsampling, the size of the feature map becomes half that of the previous feature map, and the number of channels doubles, resulting in a doubled depth feature map. This process is repeated four times to obtain depth features at different scales. (See details...) Figure 3 Features 1-4 are processed by the cross-attention module C to obtain features 1-4 represented by the dashed box on the right. The 3×3 convolutional layer in the encoder can increase the local context information of the input feature map in a weighted superposition manner, obtain multiple feature vectors of lower dimension, and finally obtain deep features. Compared with the input feature map, this feature further integrates the global view of local features, and at the same time, it is convenient to use the decoder to construct a new feature map, and increase the theoretical receptive field, making the training model more accurate. The formula for calculating the receptive field is shown in (3):

[0080] RF k-1 =(RF) k -1)×s k +f k (3)

[0081] In the formula, RF k For the receptive field of the k-th layer, f k The kernel size of the k-th layer is given by kernel_size = 3. k Let stride be the convolutional stride of the k-th layer, with stride = 1. Therefore, the receptive field RF of the k-th layer... k It is determined by the receptive field of the (k-1)th layer and the convolution stride s k and kernel size f k Determined jointly. Convolution stride s k The kernel size f determines the stride by which the receptive field of the previous layer moves within the current layer. k This determines the spatial extent of the receptive field of the current layer. By calculating the receptive field size and adding a theoretical receptive field to the network, we can help the network better understand the contextual information of the input image, thereby improving the model's accuracy and expressive power.

[0082] After obtaining deep features through the encoder, the deep features are input into the dual cross-attention module, which allows the feature map to obtain richer and denser contextual information, resulting in feature maps of different sizes with close correlation between global and local features.

[0083] Reference Figure 4The dual cross-attention module consists of two cross-attention modules. Based on the non-local attention module, it uses two attention weighting operations and replaces a single dense attention map with multiple sparse attention maps to reduce the required computational resources. Its operation method is as follows: After the feature map is input into the cross-attention module, the context information in the horizontal and vertical directions is collected to generate an attention map (upper part) and an adaptation feature map (lower part) respectively. The two are then aggregated using a weighted sum to obtain a new feature map. The new feature map is input into the next cross-attention module. Each pixel in this feature map collects information from all other pixels to enhance the global association of pixels. After the two cross-attention module operations, each pixel is finally associated with the information of all pixels in the feature map. All cross-attention modules share parameters to reduce additional parameters.

[0084] Reference Figure 5 The operation flow of the dual cross-attention module is as follows:

[0085] A1: After inputting the feature map x into the cross-attention module, it first enters two different 1×1 convolutional layers for dimensionality reduction, generating two feature maps h1 and h2 respectively;

[0086] A2: After obtaining feature maps h1 and h2, attention maps are obtained through Affinity and Softmax operations, as shown in formulas (4) and (5):

[0087]

[0088]

[0089] In the formula, d i,u h 1u with h 2iu The degree of correlation, h 1u The vector is represented as a vector at each position u in the spatial dimension of the feature map h1. Similarly, the set of feature vectors v is obtained from h2. u i represents the value in v u The i-th element in the array, where p is the number of categories, and z is the number of categories. a 'a' is the output, and 'a' is the input.

[0090] A3: Input the feature map into another 1×1 convolutional layer for dimensionality reduction to generate a feature map h3. Then, combine it with the attention map through aggregation to obtain a feature map h with distant contextual information, as shown in the following formula:

[0091]

[0092] In the formula, A iu Represents the scalar value at channel i and position u, i∈|yu |,y u Let x represent the set of eigenvectors on feature map h3. u A vector representing each position u in the spatial dimension of the feature map x;

[0093] A4: Finally, the feature map is connected with the feature map that has obtained distant context information to obtain a feature map h that is associated with long-distance context information and where each pixel exchanges and synchronizes information in a cross-shaped manner.

[0094] The decoder consists of a 2×2 upsampling layer and two 3×3 convolutional layers. This structure reconstructs the depth features, then uses skip connections to concatenate the reconstructed features with information-dense feature maps before outputting an image of the same size as the input feature map, thus generating a high-quality reconstructed frame. In this embodiment, the information-dense feature maps include... Figure 3 Features 1-4 of the dashed box on the right.

[0095] The overall operating principle of the DCAE (Digital Encoder Network) is explained as follows:

[0096] Based on a dual cross-attention mechanism and an autoencoder, this invention designs an autoencoder network DCAE to learn features from training samples. The trained model can detect the presence of abnormal events within a video set, and if present, identify the corresponding frame in the video. The difference between DCAE and traditional autoencoders lies in the fact that traditional autoencoders use fully connected layers, which leads to the loss of spatial information in two-dimensional images. DCAE, however, uses a convolutional structure to transform input patches. The convolutional layers effectively preserve the required spatial information from the input feature map. Furthermore, skip connections are used in convolution and deconvolution, and a dual cross-attention module is introduced, allowing global features to be well correlated with local features, thereby improving the accuracy of the video anomaly detection model. Unlike other literature that uses traditional fully connected autoencoders, this invention inputs the frame stream features and HOG features of the training samples into the corresponding DCAE for reconstruction. This allows the model to learn what constitutes a normal event during the process, enabling it to better reconstruct the frame stream features and HOG features of the test samples and calculate the reconstruction error during testing. This leads to a better determination of whether an abnormal event has occurred.

[0097] IV. Optimizing the video anomaly detection model using a loss function:

[0098] To optimize the dual-cross-attention autoencoder model in step three, in addition to the cross-entropy loss function, a category consistency loss function was further introduced for optimization.

[0099] The expression for the cross-entropy loss function is:

[0100]

[0101] In the formula, N is the number of classification categories, i is the input sample, and p i This represents the probability that sample i is predicted to be of the positive class;

[0102] Category consistency loss is caused by tensile loss L var Thrust loss L dis and regularization loss L reg The three components are composed of the following functional expressions:

[0103]

[0104]

[0105]

[0106] In the formula, C is the set of classes in the input image, and N... c h represents the number of valid elements in C. i Let μ be the feature vector at spatial location i. c For the cluster centers of category c∈C, For fragment distance function, and The formula is shown below:

[0107]

[0108]

[0109] In the formula, and Represents the boundary value, set

[0110] The final category consistency loss function is shown below:

[0111] L cc =ρL var +τL dis +γL reg (13)

[0112] In the formula, ρ=τ=1, γ=0.001;

[0113] The loss value of a single feature map can be obtained from formulas (7) and (13):

[0114] L = L ce +L cc (14)

[0115] Ultimately, the reconstruction error between the input features and the reconstructed features can be obtained. The smaller the reconstruction error during training, the closer the reconstructed features are to the input features, and the higher the accuracy of the model obtained after training.

[0116] V. Obtain the reconstruction error of the test samples through the video anomaly detection model to obtain the video anomaly detection results:

[0117] During the testing phase, the optical flow and HOG features of frame t are respectively fed into the trained DCAE to calculate the reconstruction error. When the reconstruction error is large, the test sample contains abnormal events; otherwise, it is normal. The HOG feature reconstruction error L is then obtained. hog and frame stream reconstruction error L ff Then, the total reconstruction error is calculated using formula (15):

[0118] Loss=αL hog +βL ff ,α+β=1 (15)

[0119] Formula (16) can be used to determine whether an abnormal event has occurred in a video frame:

[0120]

[0121] Where θ is the reconstruction error threshold. If Loss > θ, an abnormal event has occurred in the frame. If Loss ≤ θ, it means that all events are normal.

[0122] Based on the above, in order to verify the effectiveness and practical effect of the present invention, this embodiment conducted experimental verification, as follows:

[0123] Simulation experiments were conducted using the PyTorch framework and an NVIDIA GeForce GTX 3080ti GPU in PyCharm. Two publicly available datasets were used (as shown in Table 1): CUHK Avenue and UCSD Ped2. The UCSD dataset depicts a sidewalk scene, while Ped2 shows a scene where pedestrians move parallel to the camera. Anomalous events in these scenes include bicycles and wheelchairs on the sidewalk, pedestrians running, skateboards, and cars. The CUHK Avenue dataset depicts a campus avenue scene, containing anomalous events such as pedestrians running, walking in the wrong direction, trucks, bicycles, and suspicious items. In all three datasets, the training set contains only normal events, while the test set contains both normal and anomalous events.

[0124] The receiver operating characteristic curve (ROC) measures the accuracy of anomaly detection by the DCAE network. It is composed of the true positive rate (TPR) and the false positive rate (FPR), and their formulas are shown below:

[0125] TPR=TP / (TP+FN) (17)

[0126] FPR=FP / (TN+FP) (18)

[0127] In the formula, TP represents a true positive (correct affirmation), TN represents a true negative (correct negation), FP represents a false positive (incorrect affirmation), and FN represents a false negative (incorrect negation).

[0128] Table 1 Experimental Dataset

[0129]

[0130] Therefore, two evaluation criteria were used for the experimental results: (1) Area under the curve (AUC); (2) Equal error rate (EER). Both are similar performance evaluation indicators, and when EER approaches 0, AUC approaches 100%.

[0131] Experimental results

[0132] In this embodiment, the self-encoder DCAE provided by the present invention is compared with several advanced video anomaly detection methods. As shown in Tables 2 and 3, DCAE has a higher anomaly detection accuracy and a lower false positive rate, which is significantly better than the other comparison methods.

[0133] The experimental results of the video anomaly detection method using the dual-cross self-attention autoencoder on the CUHKAvenue dataset are shown in Table 2. Figure 6 As shown in Table 2, the AUC and EER evaluation results of DCAE and other methods can be observed under this dataset. The corresponding ROC curves are shown in Table 2. Figure 6 The Chinese side indicated that...

[0134] As shown in Table 2, the method proposed in this invention has improved the accuracy and error rate of detecting the CUHKAvenue dataset compared to other video anomaly detection methods.

[0135] Table 2 Comparison results with other methods under CUHK Avenue

[0136]

[0137] Figure 7 The ROC curves for the UCSD Ped2 dataset are shown in Table 3, and the corresponding AUC and EER are evaluated. It can be seen that DCAE is superior to other video anomaly detection algorithms.

[0138] Table 3 Comparison with other methods for UCSD Ped2

[0139]

[0140] The experimental results show that abnormal events in the video have a certain probability of being judged as normal, thus affecting the detection accuracy. This is because some events considered abnormal in a video clip are identified as normal in a single video frame. DCAE is based on an improved autoencoder model, which has excessive generalization ability. Since the input data is a single video frame, abnormal events that require a video clip to determine are reconstructed as normal events in a single frame. The error between the reconstructed frame and the true frame is below a threshold, leading to misjudgment as normal events.

[0141] ablation experiment

[0142] To verify the role of the dual cross-attention module in the entire autoencoder network and to test whether the module is beneficial to improving the detection accuracy of video anomalies, the experimental results of the video anomaly detection model with the dual cross-attention module were compared with those of the model without the module, as shown in Table 4.

[0143] Table 4. Impact of the dual-cross attention module on detection.

[0144]

[0145] As shown in Table 4, compared with the video anomaly detection model without the dual cross-attention module, the model with the dual cross-attention module can detect video anomalies more accurately.

Claims

1. A video anomaly detection method based on a dual-cross-attention autoencoder, characterized in that, Includes the following steps: S1: Preprocess the original video set by splitting it into multiple frame-level sequences based on different video samples; S2: Use frame difference features and optical flow features to represent the motion features of the target, use HOG features to describe the appearance features, and extract the motion features and appearance features of video frames. S3: Input the motion features and appearance features extracted from the training samples into the autoencoder network with a dual cross-attention module for training to establish a video anomaly detection model; S4: Obtain the reconstruction error of the test sample through the video anomaly detection model to get the video anomaly detection result; The method for extracting motion features in step S2 is as follows: using the frame difference method, the pixel values ​​of corresponding frames in the obtained image sequence are subtracted to obtain the difference image, and then binarized to obtain the position of the moving target; adjacent video frames are input into Flownet2 to obtain the optical flow features of the video frames, and the frame difference features are combined with the optical flow features to obtain the motion features; The method for extracting appearance features in step S2 is as follows: using the HOG feature extraction method, the appearance and shape of the video frame to be detected are described by the gradient or edge direction density distribution. The appearance features are the appearance and shape of the local target. In step S3, the dual cross-attention module consists of two cross-attention modules. Its operation method is as follows: after the feature map is input into the cross-attention module, the context information in the horizontal and vertical directions will be collected to generate attention maps and adaptation feature maps respectively, and the two will be aggregated by weighted sum to obtain a new feature map. The new feature map is fed into the next cross-attention module, where each pixel in the feature map gathers information from all other pixels to enhance the global association between pixels; After two cross-attention module operations, each pixel is ultimately associated with information from all pixels in the feature map, with all cross-attention modules sharing parameters to reduce additional parameters.

2. The video anomaly detection method based on a dual-cross-attention autoencoder according to claim 1, characterized in that, In step S3, the autoencoder network includes an encoder, a decoder, and a dual cross-attention module. The encoder is composed of two convolutional layers and one pooling layer, and the decoder is composed of one upsampling layer and two convolutional layers.

3. The video anomaly detection method based on a dual cross-attention autoencoder according to claim 2, characterized in that, The encoder in step S3 incorporates a theoretical receptive field, calculated using the following formula: (3)； In the formula, For the receptive field of the k-th layer, The kernel size for the k-th layer is kernel_size=3. Let stride be the convolution stride of the k-th layer, with stride=1.

4. The video anomaly detection method based on a dual cross-attention autoencoder according to claim 3, characterized in that, The operation flow of the dual-cross attention module in step S3 is as follows: A1: After inputting the feature map x into the dual-cross attention module, it first enters two different... Dimensionality reduction is performed in the convolutional layer to generate two feature maps. , ; A2: After obtaining feature maps h1 and h2, attention maps are obtained through Affinity and Softmax operations, as shown in formulas (4) and (5): (4)； (5)； In the formula, express and The degree of correlation, Represented as in the feature map The vector of each position u in the spatial dimension, In the feature map The set of feature vectors at position u in the spatial dimension, where i represents the position u. The i-th element in the array; p is the number of categories. 'a' is the output, and 'a' is the input. A3: Feature Map and Enter another Dimensionality reduction to generate feature maps in convolutional layers This is combined with the attention map through an aggregation operation to obtain a feature map with distant contextual information. The formula is as follows: (6)； In the formula, This represents the scalar value at channel i and position u. , Representation of feature map The set of eigenvectors on, A vector representing each position u in the spatial dimension of the feature map x; A4: Finally, the feature map is connected with the feature map that has obtained distant context information to obtain a feature map h that is associated with long-distance context information and where each pixel exchanges and synchronizes information in a cross-shaped manner.

5. A video anomaly detection method based on a dual cross-attention autoencoder according to claim 1, characterized in that, In step S3, the cross-entropy loss function is used. The video anomaly detection model is optimized using the category consistency loss function; Category consistency loss is caused by tensile loss Thrust loss and regularization loss constitute; The final category consistency loss function is shown below: (13)； In the formula, , ; The loss value of a single feature map is obtained from formula (13): (14)； Finally, the reconstruction error between the input features and the reconstructed features can be obtained.

6. The video anomaly detection method based on a dual cross-attention autoencoder according to claim 1, characterized in that, Step S4 specifically involves: The optical flow and HOG features of frame t are respectively fed into the trained video anomaly detection model to calculate the reconstruction error. The HOG feature reconstruction error is then obtained. and frame stream reconstruction error Then, the total reconstruction error is calculated using formula (15): (15)； Formula (16) can be used to determine whether an abnormal event has occurred in a video frame: (16)； in, To reconstruct the error threshold, if An abnormal event occurred in this frame. If , it means that all of them are normal events.

Images