Video anomaly detection method and system based on pseudo anomaly reconstruction, medium and equipment

Abstract

The invention provides a video anomaly detection method and system based on pseudo anomaly reconstruction, a medium and equipment, relates to the technical field of video anomaly detection, and aims to solve the problems that in existing video anomaly detection, abnormal data is scarce in a training process, normal data and abnormal data cannot be accurately distinguished, a single reconstruction error is only adopted for measurement in a reconstruction process, and the detection efficiency is low. Reconstruction errors have limitations, so that abnormal missing detection is caused, and the model abnormal detection accuracy is poor. The method comprises the steps of obtaining a video sequence to be detected, performing preprocessing, inputting the video sequence to a trained reconstruction model for anomaly detection, generating a pseudo abnormal video frame sequence based on a normal video frame sequence, constructing the reconstruction model, performing training, obtaining a final reconstruction image, and calculating a reconstruction error based on an original image and the final reconstruction image. According to the method, the problems of scarcity of abnormal data and limitation of reconstruction errors are solved, normal data and abnormal data can be distinguished more accurately, and the model anomaly detection accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video anomaly detection, and in particular relates to a video anomaly detection method, system, medium and device based on pseudo-anomaly reconstruction. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Video anomaly detection (VAD) is a very challenging task. In recent years, video surveillance has played a huge role in protecting public safety, and video anomaly detection technology that reads video clips and identifies abnormal behaviors has also attracted great attention. In the real world, the collection cost of abnormal samples is very high, which leads to less abnormal data and imbalanced training data sets. Therefore, anomaly detection is regarded as a single classification problem (OCC) and the model is trained only based on normal samples.

[0004] Reconstruction-based video anomaly detection methods are widely used to solve single classification problems. The reconstruction model is trained on normal data to minimize its reconstruction error. The trained reconstruction model is difficult to reconstruct abnormal data during testing, resulting in a large reconstruction error. The anomaly is identified by calculating the reconstruction error. However, these methods rely on the high reconstruction error obtained after training on abnormal data, but when the model is "generalized" too well and can reconstruct abnormal samples well, it may cause anomalies to be missed. To solve this problem, many methods suppress abnormality-related information during the reconstruction process and increase the reconstruction error of abnormal data, such as the widespread application of memory models. This module stores the features of normal data during training, forcing the model to apply these memorized features when reconstructing the input, effectively ensuring that the reconstruction error of abnormal data is large. However, this method depends on the size of the memory module. Too small a memory module will limit the reconstruction of normal data, but the application of multiple memory modules will lead to information filtering.

[0005] In order to enable the reconstruction model to better distinguish normal data from abnormal data, the idea of ​​using pseudo-anomalies to assist training has been proposed and gradually recognized. However, how to effectively generate pseudo-anomalies to enhance the learning ability of the model remains a challenge. In addition, during the reconstruction process, a single reconstruction error calculation method may not fully reflect the model performance. Therefore, a more comprehensive measurement standard is needed to conduct multi-dimensional evaluation to guide subsequent optimization and improvement.

[0006] From the above, we can see that the existing video anomaly detection technology has certain shortcomings: in the training of video anomaly detection models, abnormal data is scarce, the model cannot better learn and detect abnormal data, cannot accurately distinguish normal data from abnormal data, and the model anomaly detection accuracy is low; although the reconstruction method is adopted, the reconstruction process only uses a single reconstruction error calculation method to measure, and the single reconstruction error calculation method has limitations and cannot fully reflect the model performance, resulting in missed anomalies and poor model anomaly detection accuracy. Summary of the invention

[0007] In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides a video anomaly detection method, system, medium and device based on pseudo-anomaly reconstruction. A data enhancement method for pseudo-anomaly data-assisted training is adopted to solve the problem of scarcity of abnormal data, provide more abnormal data for the model to learn, and enhance the model's ability to distinguish normal data from abnormal data; and a self-supervised masked convolutional quantized autoencoder is proposed as a reconstruction model, quantization and self-supervised masked convolution are introduced in the autoencoder, and additional steps are introduced outside the traditional reconstruction process to solve the limitations of reconstruction errors.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] In a first aspect, a video anomaly detection method based on pseudo anomaly reconstruction is disclosed, comprising:

[0010] Obtain the video frame sequence to be detected and perform preprocessing;

[0011] The preprocessed video frame sequence to be detected is input into the trained reconstruction model, and whether the video sequence to be detected is abnormal is determined according to the reconstruction error of the reconstruction model;

[0012] Wherein, the reconstruction model, during training, includes:

[0013] Obtain a normal video frame sequence, and the pseudo-anomaly generator processes the normal video frame sequence by skipping frames, including:

[0014] A frame index is randomly selected, and multiple consecutive video frames are selected to construct a normal input sequence. Parameters are introduced to control the number of frame skipping and generate the corresponding pseudo-abnormal video frame sequence.

[0015] As a further technical solution, the reconstruction model, when training, further includes:

[0016] The normal and pseudo-abnormal video frame sequences are input into the encoder for processing to obtain the original feature vector of the mixed data. The quantization layer is used to discretize the information of the original feature vector to obtain the quantized feature vector. The decoder is used to decode the quantized feature vector.

[0017] As a further technical solution, the normal and pseudo-abnormal video frame sequences are input into an encoder for processing, and the encoder generates a continuous feature vector, that is, the encoder outputs the original feature vector.

[0018] As a further technical solution, after the encoder outputs the original feature vector, it also includes:

[0019] Input the original feature vector into the quantization layer to obtain the quantized feature vector;

[0020] Based on the quantized feature vector and the original feature vector, a quantized training square error is constructed;

[0021] Optimize the quantization training square error to obtain the optimized quantization square error;

[0022] Construct quantization loss based on quantized training square error and optimized quantized square error;

[0023] Based on the quantization loss, the quantized feature vector is optimized;

[0024] And the optimized quantized feature vector is input to the decoder.

[0025] As a further technical solution, the decoder includes a first part decoder, a self-supervised convolution mask layer and a second part decoder. The optimized quantized feature vector is input into the first part decoder, an intermediate layer output vector is generated in the intermediate layer of the decoder, and the intermediate layer output vector is input into the self-supervised convolution mask layer for self-supervised mask reconstruction to obtain a mask reconstruction vector, and the mask loss of the intermediate layer output vector and the mask reconstruction vector is calculated, and the mask reconstruction vector is input into the second part decoder to obtain the final decoder output vector.

[0026] As a further technical solution, the intermediate layer output vector is subjected to self-supervised mask reconstruction in the self-supervised convolution mask layer of the decoder to obtain a mask reconstruction vector. The specific process is as follows:

[0027] Get a set of 3D sub-kernels and mask convolution kernels, set the mask area and the size of the convolution kernel;

[0028] Apply masked convolution to the output vector of the intermediate layer, and convolve the sub-kernel at each position to obtain a feature map;

[0029] Extract features from multiple channels to obtain output tensors, perform spatial average pooling on the output tensors, and reshape the feature matrix;

[0030] Process the reshaped feature matrix, obtain global dependencies and update feature representation;

[0031] Based on the updated feature representation, a mask reconstruction vector is obtained.

[0032] As a further technical solution, preprocessing includes: using 3D-CNN to extract the video sequence to be detected into video frames, adjusting the size of the video frames and performing intensity normalization operations.

[0033] In a second aspect, a video anomaly detection system based on pseudo anomaly reconstruction is disclosed, comprising:

[0034] A data acquisition module is used to acquire the video frame sequence to be detected and perform preprocessing;

[0035] The abnormal training detection module is used to input the preprocessed detection video frame sequence into the trained reconstruction model, and judge whether the video sequence to be detected is abnormal according to the reconstruction error of the reconstruction model;

[0036] When training, the reconstruction model includes: obtaining a normal video frame sequence, and the pseudo-abnormal generator processes the normal video frame sequence by skipping frames, including: randomly selecting a frame index, and selecting multiple consecutive video frames to construct a normal input sequence, introducing parameters to control the number of skipped frames, and generating a corresponding pseudo-abnormal video frame sequence.

[0037] The third aspect of the present invention is to provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0038] A fourth aspect of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which executes the steps of the above method when executed by a processor.

[0039] One or more of the above technical solutions have the following beneficial effects:

[0040] In this embodiment, the present invention constructs pseudo-abnormal samples during the training process, overcomes the difficulty of scarce abnormal data, and helps the model to better learn and detect abnormal data; a quantization layer is added to the autoencoder (AE) to quantize the encoder output and retain key discriminant features; at the same time, on the basis of the traditional reconstruction method, a self-supervised convolutional mask layer is added to realize mask reconstruction, and additional supplements are made on the basis of the original reconstruction, which solves the limitation problem of reconstruction error, enhances the model's contextual understanding of video frames, and improves the model's anomaly detection accuracy.

[0041] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0043] Figure 1 This is a main flow chart of the video anomaly detection method based on pseudo anomaly reconstruction of the first embodiment of the present invention;

[0044] Figure 2 This is a training diagram of the overall reconstruction model based on pseudo abnormal data in the first embodiment of the present invention;

[0045] Figure 3 is a schematic diagram of the structure of the reconstruction model of the first embodiment;

[0046] Figure 4 This is a schematic diagram of mask reconstruction in the first embodiment of the present invention. DETAILED DESCRIPTION

[0047] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0048] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0049] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0050] The overall idea proposed in the present invention is as follows: first, pseudo-abnormal data is added to solve the problem of scarcity of abnormal data, and Gondo abnormal data is provided for the model to learn, a reconstruction model is constructed, a quantization layer and a self-supervised mask convolution layer are added to solve the limitation of reconstruction error, and the reconstruction model is trained. The trained reconstruction model is used to perform anomaly detection on the data to be detected.

[0051] Embodiment 1

[0052] This embodiment discloses a video anomaly detection method based on pseudo-anomaly reconstruction.

[0053] In order to more clearly illustrate the present embodiment, a video anomaly detection implementation process based on pseudo anomaly reconstruction can be specifically described as follows:

[0054] A video anomaly detection method based on pseudo anomaly reconstruction, comprising:

[0055] S1, obtaining a video frame sequence to be detected and performing preprocessing;

[0056] S2, inputting the preprocessed video frame sequence to be detected into the trained reconstruction model, and judging whether the video frame sequence to be detected is abnormal according to the reconstruction error of the reconstruction model;

[0057] Wherein, the reconstruction model, during training, includes:

[0058] A normal video frame sequence is obtained, and the pseudo-abnormal generator processes the normal video frame sequence by skipping frames, including: randomly selecting a frame index, and selecting multiple consecutive video frames to construct a normal input sequence, introducing parameters to control the number of skipped frames, and generating a corresponding pseudo-abnormal video frame sequence.

[0059] like Figure 1 As shown, in step S1, a video frame sequence to be detected is obtained and preprocessed.

[0060] From the given abnormal and normal video sequences, 3D-CNN is used to extract the video sequences to be detected into video frames, and the sizes of all input video frames are adjusted to 256×256 pixels. After adjusting the size of the video frames, intensity normalization and other operations are performed.

[0061] Specifically, the pixel values ​​are converted from the range of [0, 255] to [-1, 1]. The normalization process involves converting the frame to a floating point type and dividing the pixel value by 127.5 and then subtracting 1. Normalization ensures data consistency and improves model training results.

[0062] The preprocessed video frames are combined into a continuous video frame sequence.

[0063] like Figure 1 , Figure 2 As shown, in step S2, the preprocessed continuous video frames are input into the trained reconstruction model, and whether the video sequence to be detected is abnormal is determined based on the reconstruction error of the reconstruction model.

[0064] like Figure 1 , Figure 2 As shown, S21, the reconstruction model includes:

[0065] Obtain a normal video frame sequence, and the pseudo-anomaly generator processes the normal video frame sequence by skipping frames, including:

[0066] A frame index is randomly selected, and multiple consecutive video frames are selected to construct a normal input sequence. Parameters are introduced to control the number of frame skipping and generate the corresponding pseudo-abnormal video frame sequence.

[0067] Specifically, Figure 1 , Figure 2 As shown, S211, based on the normal video frame sequence, generate a pseudo abnormal video frame sequence.

[0068] The pseudo-anomaly generator uses frame skipping to process normal video frame sequences, simulating data outside the distribution of normal video frame sequences to assist in the construction and training of the reconstruction model.

[0069] Randomly select a frame index n and select T consecutive video frames to construct a normal input sequence X N =(I n ,I n+1 ,…,I n+T-2 ,I n+T-1 ). The parameter s is introduced to control the number of frame skipping and generate the corresponding pseudo abnormal video frame sequence X. A .

[0070] X A =(I n ,I n+s ,…,I n+(T-2)s ,I n+(T-1)s ) (1)

[0071] Among them, X N represents a normal video frame sequence, T represents the length of the normal video frame sequence, s is the step length used to skip frames, n is a random frame index, and I n Represents the nth frame image in the training data, I n+(T-1)s Indicates the last frame in the selected video frame, X A Represents a sequence of pseudo-anomalous video frames.

[0072] Frame skipping simulates motion changes that do not exist in normal video sequences and expands the training data for anomaly detection. During the training process, is the i-th normal video frame sequence, For the generated i-th pseudo-anomaly video frame sequence, the pseudo-anomaly is constructed according to the probability of p.

[0073]

[0074] The probability parameter p is used to control the ratio of pseudo-abnormal video frame sequences to normal video frame sequences introduced during the training process. Specifically, p defines the probability of selecting a pseudo-abnormal sample in each training step, where 1-p means that the input sequence X is a normal video frame sequence X. N The probability of p is that the input sequence is a pseudo-anomaly video frame sequence X generated by the pseudo-anomaly generator. A probability.

[0075] By adjusting the value of p, the degree of introduction of pseudo-abnormal video frame sequences can be flexibly controlled. The steps are as follows:

[0076] First, generate a random number in the range [0,1).

[0077] Then, the random number is compared with the probability p. If the random number is less than p, the current image is replaced with a pseudo abnormal video frame sequence and its state is recorded; otherwise, a normal video frame sequence is used and recorded as a normal state.

[0078] After the above steps, pseudo abnormal data is obtained. In model training, the problem of scarcity of abnormal data is solved. Model training provides more abnormal data for learning and enhances the model's ability to distinguish normal data from abnormal data.

[0079] S212: Mark the normal video frame sequence and the pseudo-abnormal video frame sequence respectively.

[0080] Normal video frame sequence, and with a certain probability load pseudo abnormal video frame sequence for replacement. For normal video frame sequence, it is marked as JPS (jump pseudostat) = False. If pseudo abnormal video frame sequence replacement is performed, it is marked as JPS = True, that is, normal video frame sequence is marked with JPS as False, and pseudo abnormal video frame sequence is marked with JPS as True.

[0081] like Figure 1 , Figure 2 , Figure 3 As shown, during the training of the S22 reconstruction model, the following are also included:

[0082] The normal and pseudo-abnormal video frame sequences are input into the encoder for processing to obtain the original feature vector of the mixed data image. The quantization layer is used to discretize the information of the original feature vector to obtain the quantized feature vector. The decoder is used to decode the quantized feature vector.

[0083] The reconstruction model is used to reconstruct the input video frame of the mixed normal and pseudo abnormal data. A quantization layer is added between the encoder and the decoder, where the encoder is used to map the input video frame to the latent representation in the latent space, the quantization layer is used to discretize the information, and the decoder is used to map the latent representation back to the space of the reconstructed video frame.

[0084] Specifically, S221, the normal video frame sequence and the pseudo abnormal video frame sequence are mixed and input into the encoder to obtain the original feature vector of the mixed data image, and the original feature vector is input into the quantization layer.

[0085] The normal video frame sequence and the pseudo-abnormal video frame sequence are mixed and input into the encoder, and the encoder generates a continuous feature vector z e ∈R D , that is, the original feature vector output by the encoder, and the continuous feature vector z generated by the encoder e Input to the quantization layer.

[0086] S222. In the quantization layer, the original feature vector of the mixed data image is forcibly quantized to obtain a quantized feature vector.

[0087] In the quantization layer, there is a discrete codebook C = {c1, c2, ..., c K}, where each c K ∈R D , represents a vector in the codebook. The goal of the quantization process is to find the original feature vector z in the codebook C that is consistent with the encoder output e The closest vector is used as the quantized output, then the quantized feature vector z q The formula is:

[0088]

[0089] Among them, z q is the quantized feature vector, z e is the original feature vector, C is the discrete codebook, c K ∈R D , represents a vector in the codebook.

[0090] S223. Construct a quantized training square error based on the quantized feature vector and the original feature vector.

[0091] In order to force the quantized eigenvector z q With the original eigenvector z e As close as possible, this is achieved by calculating the square error between the quantized feature vector and the original feature vector, using the formula:

[0092]

[0093] Among them, L com is the square error of the quantized line, N represents the number of samples involved in the calculation, Indicates from z q The tensor separated from z, that is, z is not updated when calculating the loss q By minimizing the squared error, the quantizer can learn to effectively map the input data into a discrete embedding space.

[0094] S224. Based on the quantized eigenvector and the original eigenvector, optimize the quantization square error and retain the valid original eigenvector information.

[0095] In order to ensure that the quantized feature vector z q Can effectively represent the original feature vector z e , optimize the quantized feature vector, and its calculation is the same as L com Consistent, the quantitative optimization formula is:

[0096]

[0097] Among them, L em Optimize the quantized square error, N represents the number of samples involved in the calculation, Indicates from z e The tensor separated from z, that is, z is not updated when calculating the loss e The gradient of , by minimizing the optimized average variance, can help the quantizer learn how to choose the appropriate discrete embedding to preserve the information of the original feature vector as much as possible.

[0098] S225. Based on the quantization training average error and the optimized quantization square error, a quantization loss is constructed to optimize the quantized feature vector.

[0099] Based on the quantized training square error and optimized quantized square error in steps 223 and 224, a quantized loss is constructed to ensure that the quantized representation can effectively represent the original data. The quantized loss formula is:

[0100] L vq =(α×L com +β×L em )(6)

[0101] Among them, L vq represents the quantization loss, α and β represent the quantization loss weights, assuming α = 0.25, β = 1; L em represents the optimized quantized square error, L com Represents the quantized training squared error.

[0102] When L em and L com When it is minimum, the quantization loss is minimum, which means that the quantized feature vector obtained by the quantization layer is more accurate.

[0103] After the above steps, the original feature information of the encoder is discretized, the encoder output is quantized and the key discriminative features are retained.

[0104] like Figure 1 , Figure 2 , Figure 3 As shown, S23, the decoder includes a first partial decoder, a self-supervised convolutional mask layer and a second partial decoder. During training, the optimized quantized feature vector is input into the first partial decoder, an intermediate layer output vector is generated in the intermediate layer of the decoder, and the intermediate layer output vector is input into the self-supervised convolutional mask layer for self-supervised mask reconstruction to obtain a mask reconstruction vector, and the mask loss of the intermediate layer output vector and the mask reconstruction vector is calculated, and the mask reconstruction vector is input into the second partial decoder to obtain the final decoder output vector.

[0105] S231. Input the optimized quantized feature vector into the first part decoder, generate an intermediate layer output vector in the intermediate layer of the decoder, and input the intermediate layer output vector into the self-supervised convolutional mask layer for self-supervised mask reconstruction to obtain the mask reconstruction vector, and calculate the mask loss of the intermediate layer output vector and the mask reconstruction vector.

[0106] In the decoder, a self-supervised convolutional mask layer is added to mask some information during the decoding process and reconstruct the masked information using contextual information. At the same time, the mask reconstruction loss is calculated to further learn and discover the global structure of the feature hierarchy extracted by the convolutional layer.

[0107] The quantized feature vector z q Input to the decoder to get the intermediate layer output vector X t .

[0108] The intermediate layer output vector X t Self-supervised mask reconstruction is performed in the self-supervised convolutional mask layer of the decoder to obtain the mask reconstruction vector. The specific process is:

[0109] like Figure 3 , Figure 4 As shown, S2311, obtain a set of 3D sub-kernels and mask convolution kernels, and set the size of the mask area and the convolution kernel.

[0110] Specifically: Select a set of 3D sub-kernels K learned i ∈R k′×k′×k′×ch ,i∈{1,2,…,8}, where k′ represents the spatial dimension of each sub-kernel and ch represents the number of channels of the input data. The position of the mask region M is located at the center of the mask convolution kernel and is aligned with each 3D sub-kernel K i The distance between them is equal, which is controlled by adjusting the expansion rate d. In order to reduce the amount of hyperparameters in the model, the size of M is set to 1×1×1×ch, and each channel has only one pixel.

[0111] The size of the entire mask convolution kernel is k×k×k, where k=2k′+2d+1. When performing mask convolution calculations, each sub-kernel K i will perform convolution operation with the corresponding local area in the input data X. The size of this local area is related to K i At the same time, the center of the sub-core will be aligned with the central pixel of M, which ensures that each sub-core can accurately observe the local information corresponding to its specified position.

[0112] like Figure 3 , Figure 4 As shown, S2312, mask convolution is applied to the intermediate layer output vector, and the sub-kernel is convolved at each position to obtain a feature map.

[0113] The output vector X of the decoder intermediate layer t , convolution operations are performed at positions (u,v,w) through masked convolution kernels. These convolution operations convolve the input with each sub-kernel independently, and only consider the input values ​​that overlap with the sub-kernel. Each sub-kernel Ki has a different spatial offset relative to the current position (u,v,w), and the convolution results are summed to obtain a single output value. Therefore, the output value represents the reconstructed value of the tensor M at the same position (u,v,w). Y (u,v,w) It is the output value at position (u,v,w). The specific formula is:

[0114]

[0115] Among them, Y (u,v,w) is the output value at position (u,v,w), (u,v,w) represents the reconstructed position, X t *K i Represents the sub-core K i With input data X t This process ensures that local features can be extracted and reconstructed from the input tensor through the combination of multiple sub-kernels.

[0116] like Figure 3 , Figure 4 As shown, S2313 extracts features from multiple channels to obtain an output tensor, performs spatial average pooling on the output tensor, and reshapes the feature matrix.

[0117] Apply ch masked convolution kernels and output tensor X from the intermediate layer t Extract features from different channels and get the output tensor Z. The formula is:

[0118]

[0119] Among them, X t *K i Represents the sub-core K i With input data X t The convolution operation.

[0120] Perform spatial average pooling on S and reshape the obtained values ​​into a matrix A with a shape of ch×m, where m is the spatial dimension after average pooling. The feature matrix A formula is:

[0121] A=Rs(Avg(S))(9)

[0122] Among them, S represents the output tensor extracted from different channels, Rs is reshaping, Avg is spatial average pooling, and each row of the feature matrix A represents the masked convolution result of the corresponding channel, which captures the spatial feature representation of each channel.

[0123] like Figure 3 , Figure 4 As shown, S2314, the reshaped feature matrix is ​​processed to obtain global dependencies and update feature representation.

[0124] After being processed by multiple transformer blocks, each block consists of the following steps:

[0125] First, the matrix A is input into the linear projection layer and added to the position embedding to get T, and then T is normalized to get the initial value R. Then, R is sent to the multi-head attention layer H for calculation to get the intermediate result O, the formula is:

[0126] O=H(norm(R))+R (10)

[0127] Among them, R is the initial value after linear projection and position embedding addition, norm(R) represents the normalization operation of R, and H(·) represents the calculation of the multi-head attention layer.

[0128] O is normalized again and input into the multi-layer perceptron G to obtain the updated R, the formula is:

[0129] R=G(norm(P))+P (11)

[0130] Here, G(·) represents the calculation of the multilayer perceptron.

[0131] The new R is fed back into H again, and the whole process is repeated L times, each time capturing higher-level feature relationships and contextual information. In each transformer block, H captures the global dependencies in the input sequence T by calculating the weighted sum of the query Q, key K, and value V. The output of H is further processed by MLP to provide higher-dimensional representation capabilities to better understand and process the complex structure in the input sequence.

[0132] like Figure 3 , Figure 4 As shown, S2315, based on the updated feature representation, a mask reconstruction vector is obtained, and the mean square error between the input vector and the reconstructed output vector is calculated.

[0133] After being processed by L transformer blocks, the output R is average pooled into a vector The final attention weights are generated by the sigmoid function. These weights are applied to the original output tensor S and the reconstructed output tensor is obtained by element-by-element multiplication.

[0134]

[0135] Among them, S represents the output tensor extracted from different channels, The vector after average pooling, σ(·) represents the sigmoid function.

[0136] After step S2311 to step S2315, the central area is masked on the input feature map and the surrounding visible parts are used for reconstruction. Combined with the channel self-attention mechanism of the transformer, weights are generated through the sigmoid layer to dynamically adjust the influence of different channels in the reconstruction. Self-supervised learning is used to optimize the reconstruction effect without labels and learn a more robust feature representation.

[0137] The target of self-supervised training is set as the intermediate vector X of the input t The output vector reconstructed with the masked convolution block The mean square error between , that is, the mask loss formula is:

[0138]

[0139] Among them, L m represents the mask loss, X t represents the intermediate vector, represents the mask reconstruction vector.

[0140] After the above steps, part of the information in the decoding process is masked, and the masked information is reconstructed using context information. At the same time, a mask reconstruction loss is added to the reconstruction model to further learn and discover the global structure of the feature hierarchy extracted by the convolutional layer.

[0141] S232: Input the mask reconstruction vector into the second partial decoder to obtain a reconstruction vector.

[0142] Reconstruct the mask into vector Input into the second part of the decoder to get the reconstructed vector z d .

[0143] like Figure 1 , Figure 2 , Figure 3 As shown, S24, calculate the standard reconstruction error of the mixed data to minimize the reconstruction error of the normal data and maximize the reconstruction error of the pseudo-abnormal data.

[0144] S241. Calculate the standard reconstruction error of the mixed data.

[0145] The mixed data is passed through the encoder, quantization layer, and decoder with self-supervised mask convolution module in sequence to obtain the final reconstructed vector z dIn this process, the quantization loss and mask loss are calculated, as shown in formula (6) (13), and the standard reconstruction error is calculated at the same time, as shown in formula:

[0146] L rec =MSE(z,z d )(14)

[0147] Among them, L rec represents the standard reconstruction error, z represents the initial vector of the mixed data, and z d Represents the reconstruction vector.

[0148] S242: Minimize the standard reconstruction error of the normal video frame sequence, and maximize the standard error of the pseudo-abnormal video frame sequence.

[0149] By minimizing the standard reconstruction error of normal video frame sequences, the model can effectively learn the characteristics of normal data and establish a benchmark; while maximizing the standard reconstruction error of pseudo-abnormal video frame sequences, the model can identify and learn the characteristics of abnormal data. The specific steps are as follows:

[0150] S2421. Create an empty list L mod , used to store the standard reconstruction error of each sample.

[0151] If sample a is marked as a pseudo-anomaly (JPS[a] is True), its negative standard reconstruction error is calculated and added to L mod middle:

[0152] L mod [a]=-L rec [a](15)

[0153] If sample b is marked as normal (JPS[b] is False), its standard reconstruction error is calculated and added to L mod middle:

[0154] L mod [b] = L rec [b] (16)

[0155] S2422. Stack the losses and calculate the average loss to obtain the final standard reconstruction error.

[0156] L mod All loss values ​​in the list are stacked into a tensor, and the average of the stacked tensor is calculated to get the final standard reconstruction error L rec .

[0157] Among them, the stacked loss uses torch.stack(Lmod) to convert all values ​​in the Lmod list into a tensor stacked_loss_mse.

[0158] Calculate the average loss and use torch.mean(stacked_loss_mse) to calculate the average of the stacked tensors to get the final loss value.

[0159] The standard reconstruction error is combined with the previously calculated quantization loss and mask loss to obtain the final reconstruction error. The final reconstruction error is L all , the formula is

[0160] L all =L rec +λ1L vq +λ2L m (17)

[0161] Among them, L all represents the final reconstruction error, L rec represents the standard reconstruction error, L vq represents the quantization loss, L m represents the mask loss, λ1 and λ2 are weights, and λ1=0.2 and λ2=0.01.

[0162] In this way, the final reconstruction error will comprehensively consider the standard reconstruction error, quantization loss and mask loss, thereby more comprehensively reflecting the performance and learning effect of the model.

[0163] like Figure 1 , Figure 2 As shown, in step S25, anomaly detection is performed on the video sequence to be detected according to the final reconstruction error.

[0164] The video sequence to be detected is input into the trained reconstruction model to generate the reconstructed output, and then the reconstruction error is calculated, and the peak signal-to-noise ratio (PSNR) is calculated based on it. The formula is:

[0165]

[0166] in, is the reconstructed image of the tth frame, I t is the original t-th frame image, R is the total number of pixels in the image, which is used as a normalization factor when calculating PSNR to ensure that the result is independent of the image size. I^t is the reconstructed frame The maximum possible value, usually refers to the maximum brightness value of all pixels in the image, which is used in the formula for calculating PSNR to ensure that the output value correctly reflects the ratio of signal to noise.

[0167] Next, the PSNR value is used to calculate the anomaly score, and the score is added to the total anomaly score list, and the video index is updated at the same time.

[0168] Finally, by converting the total anomaly score into a NumPy array, the AUC method is used to evaluate the detection accuracy of the model, thereby effectively identifying anomalies in the video.

[0169] Embodiment 2

[0170] The purpose of this embodiment is to provide a video anomaly detection system based on pseudo-anomaly reconstruction, comprising:

[0171] A data acquisition module is used to acquire the video frame sequence to be detected and perform preprocessing;

[0172] The abnormal training detection module is used to input the preprocessed video frame sequence to be detected into the trained reconstruction model, and judge whether the video sequence to be detected is abnormal according to the reconstruction error of the reconstruction model;

[0173] Among them, when training, the reconstruction model includes: obtaining a normal video frame sequence, and the pseudo-abnormal generator processes the normal video frame sequence by skipping frames, including: randomly selecting a frame index, and selecting multiple consecutive video frames to construct a normal input sequence, introducing parameters to control the number of frame skipping, and generating a corresponding pseudo-abnormal video frame sequence.

[0174] Based on providing a video anomaly detection system based on pseudo-anomaly reconstruction, the method steps in embodiment 1 are implemented.

[0175] Embodiment 3

[0176] The purpose of this embodiment is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the program.

[0177] Embodiment 4

[0178] The purpose of this embodiment is to provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the above method are performed.

[0179] The steps involved in the apparatus of the above embodiment correspond to the method embodiment 1, and the specific implementation method can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0180] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0181] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A video anomaly detection method based on pseudo anomaly reconstruction, characterized in that: include: Obtain the video frame sequence to be detected and perform preprocessing; The preprocessed video frame sequence to be detected is input into the trained reconstruction model, and whether the video sequence to be detected is abnormal is determined according to the reconstruction error of the reconstruction model; Wherein, the reconstruction model, during training, includes: Obtain a normal video frame sequence, and the pseudo-anomaly generator processes the normal video frame sequence by skipping frames, including: A frame index is randomly selected, and multiple consecutive video frames are selected to construct a normal input sequence. Parameters are introduced to control the number of frame skipping and generate the corresponding pseudo-abnormal video frame sequence.

2. A video anomaly detection method based on pseudo anomaly reconstruction as claimed in claim 1, characterized in that: The reconstruction model, during training, also includes: The normal and pseudo-abnormal video frame sequences are input into the encoder for processing to obtain the original feature vector of the mixed data. The quantization layer is used to discretize the information of the original feature vector to obtain the quantized feature vector. The decoder is used to decode the quantized feature vector.

3. The video anomaly detection method based on pseudo anomaly reconstruction as claimed in claim 1, characterized in that: The normal and pseudo-abnormal video frame sequences are input to the encoder for processing, and the encoder generates a continuous feature vector, that is, the encoder outputs the original feature vector.

4. A video anomaly detection method based on pseudo anomaly reconstruction as claimed in claim 3, characterized in that: The encoder outputs the original feature vector and also includes: Input the original feature vector into the quantization layer to obtain the quantized feature vector; Based on the quantized feature vector and the original feature vector, a quantized training square error is constructed; Optimize the quantization training square error to obtain the optimized quantization square error; Construct quantization loss based on quantized training square error and optimized quantized square error; Based on the quantization loss, the quantized feature vector is optimized; And the optimized quantized feature vector is input to the decoder.

5. The video anomaly detection method based on pseudo anomaly reconstruction as claimed in claim 4, characterized in that: The decoder includes a first part decoder, a self-supervised convolution mask layer and a second part decoder. The optimized quantized feature vector is input into the first part decoder, an intermediate layer output vector is generated in the intermediate layer of the decoder, and the intermediate layer output vector is input into the self-supervised convolution mask layer for self-supervised mask reconstruction to obtain a mask reconstruction vector, and the mask loss of the intermediate layer output vector and the mask reconstruction vector is calculated, and the mask reconstruction vector is input into the second part decoder to obtain the final decoder output vector.

6. A video anomaly detection method based on pseudo anomaly reconstruction as claimed in claim 5, characterized in that: The output vector of the intermediate layer is self-supervised masked and reconstructed in the self-supervised convolutional mask layer of the decoder to obtain the masked reconstruction vector. The specific process is as follows: Get a set of 3D sub-kernels and mask convolution kernels, set the mask area and the size of the convolution kernel; Apply masked convolution to the output vector of the intermediate layer, and convolve the sub-kernel at each position to obtain a feature map; Extract features from multiple channels, obtain output tensors, perform spatial average pooling on the output tensors, and reshape the feature matrix; Process the reshaped feature matrix, obtain global dependencies and update feature representation; Based on the updated feature representation, a mask reconstruction vector is obtained.

7. The video anomaly detection method based on pseudo anomaly reconstruction as claimed in claim 1, characterized in that: The preprocessing includes: using 3D-CNN to extract the video sequence to be detected into video frames, adjusting the size of the video frames and performing intensity normalization operations.

8. A video anomaly detection system based on pseudo anomaly reconstruction, characterized in that: include: A data acquisition module is used to acquire the video frame sequence to be detected and perform preprocessing; The abnormal training detection module is used to input the preprocessed video frame sequence to be detected into the trained reconstruction model, and judge whether the video sequence to be detected is abnormal according to the reconstruction error of the reconstruction model; Among them, when training, the reconstruction model includes: obtaining a normal video frame sequence, and the pseudo-abnormal generator processes the normal video frame sequence by skipping frames, including: randomly selecting a frame index, and selecting multiple consecutive video frames to construct a normal input sequence, introducing parameters to control the number of skipped frames, and generating a corresponding pseudo-abnormal video frame sequence.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are performed.

Images