A video anomaly detection method based on dual generator and channel attention mechanism

Abstract

The application discloses a video anomaly detection method based on a double generator and a channel attention mechanism, and comprises the following steps: splitting a video set to obtain a plurality of frame-level sequences, and dividing the frame-level sequences into normal training video frames and test video frames; training a noise generator by using the normal training video frames and a discriminator, generating pseudo abnormal frames by using the trained noise generator; training a reconstruction generator by using the pseudo abnormal frames and the normal training video frames to obtain a trained reconstruction generator; inputting the test video frames into the trained reconstruction generator to obtain reconstructed frames, calculating reconstruction errors of the reconstructed frames and real frames, and classifying the video frames according to the reconstruction errors. The application proposes a video anomaly event detection method taking a generative adversarial network based on a double generator and a channel attention mechanism as a core, introduces a second-order channel attention module into the generator by using a noise generator and a reconstruction generator, and improves the detection accuracy of the video anomaly event detection.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and relates to video anomaly detection technology, specifically to a video anomaly detection method based on a dual generator and channel attention mechanism. Background Technology

[0002] In this era of rapid technological advancement, to ensure the smooth operation of society and the safety of the people, a large number of surveillance cameras are used in various public places such as roads, schools, and shopping malls, resulting in countless surveillance videos. Manually screening and inspecting these videos requires a significant amount of time and effort, thus video anomaly detection has seen substantial development. Video anomalies can be considered abnormal appearances or movements, or normal appearances or movements occurring in abnormal locations or at abnormal times. Whether an event in a video is abnormal needs to be determined based on the context. An event deemed abnormal in one context may be normal in another; for example, a truck or other vehicle might be abnormal on a school campus but normal in a traffic scene.

[0003] Existing video anomaly detection methods can only train the detection model using normal video frames. This method cannot learn abnormal patterns in advance and has no ability to perceive anomalies. During detection, there will be ambiguity in the classification of abnormal frames, so the final anomaly detection accuracy is not ideal. Summary of the Invention

[0004] Purpose of the invention: In order to overcome the shortcomings of the existing technology, a video anomaly detection method based on dual generator and channel attention mechanism is provided to solve the problem that the training data only includes normal events, which makes it difficult for the model to learn abnormal patterns in advance. It can improve the ability to perceive abnormal phenomena, learn the abnormal distribution in advance, and ultimately improve the anomaly detection accuracy.

[0005] Technical Solution: To achieve the above objectives, this invention provides a video anomaly detection method based on a dual-generator and channel attention mechanism, comprising the following steps:

[0006] S1: Split the video set to obtain multiple frame-level sequences and divide them into normal training video frames and test video frames;

[0007] S2: Train the noise generator using normal training video frames and the discriminator, and generate pseudo-abnormal frames using the trained noise generator.

[0008] S3: Train the reconstruction generator using pseudo-abnormal frames and normal training video frames to obtain the trained reconstruction generator;

[0009] S4: Input the test video frame into the trained reconstruction generator to obtain the reconstructed frame, calculate the reconstruction error between the reconstructed frame and the real frame, and classify the video frame as anomaly based on the reconstruction error.

[0010] Furthermore, in step S2, the noise generator includes a noise module, a generator, and a second-order channel attention module. The noise generator is trained twice: first, the generator without a noise module is trained, and second, the noise generator with a noise module is trained. The training method is as follows: the generator is trained by adversarially interacting with the discriminator, so that the generated frames cannot be distinguished as abnormal by the discriminator. After obtaining the trained generator, the parameters of the generator are fixed, and a noise module is added to the generator. The generator is then subjected to adversarial interaction with the discriminator again, and the parameters of the noise module are updated to obtain the noise generator.

[0011] Furthermore, in step S2, the generator employs an autoencoder network, consisting of an encoder and a decoder. The training method for the generator is as follows:

[0012] When a normal training video frame is input into the generator, it first enters the encoder part, which consists of pooling layers, convolutional layers, and repeated activation functions. After multiple halvings in size and doublings in channels, the latent features are obtained.

[0013] Secondly, the latent features are input into the second-order channel attention module to improve the interdependence of features, obtain richer and denser information correlation, and obtain features that are closely related to global features and local features.

[0014] Finally, the obtained features are input into the decoder to reconstruct the latent features, and the output is an image of the same size as the input frame, thereby generating a high-quality reconstructed frame.

[0015] The reconstructed frame and the real frame are input into the discriminator together, so that the discriminator and the generator compete against each other, and the reconstructed frame generated by the generator is judged by the discriminator as a real frame.

[0016] Furthermore, in step S2, the noise module consists of a three-layer fully connected autoencoder, including noise module A and noise module B. The training method for the noise generator through the noise module is as follows:

[0017] Random noise is input into the noise module, and three convolutions, batch normalization, and activation function operations are performed to obtain noise suitable for generating pseudo video frames.

[0018] After obtaining the trained generator, a noise module is added to the generator. Skip connections are used after the encoder’s first convolutional pooling and before the decoder’s last upsampling, and noise module A is added. After obtaining the latent features, noise module B is added. The latent features with noise are reconstructed to obtain pseudo-abnormal frames.

[0019] The pseudo-abnormal frames and real frames are input into the discriminator together, so that the discriminator and the generator compete against each other to maximize the distance between the two frames. The size of noise module A and noise module B is the same as the corresponding feature size.

[0020] Furthermore, the training method for the reconstruction generator in step S3 is as follows: a maximum constraint function and a minimum constraint function are added to the reconstruction pseudo-abnormal frames and the reconstruction normal training video frames, respectively, so that the reconstruction pseudo-abnormal frames, the reconstruction real frames and the real frames compete with each other, thereby widening the distance between the reconstruction pseudo-abnormal frames and the real frames and narrowing the distance between the reconstruction real frames and the real frames.

[0021] Furthermore, in step S3, the reconstruction generator is composed of an autoencoder and a second-order channel attention module. The autoencoder consists of an encoder and a decoder, and uses U-net as its network structure. The specific training process is as follows: pseudo-anomaly frames and real frames are input into the reconstruction generator. First, they enter the encoder module and are processed through multiple convolutional pooling to obtain latent features of different sizes and number of channels in each layer. Then, the latent features of each layer are input into the second-order channel attention module to learn interdependent and closely related information, resulting in features with high information correlation. The features of each layer are then concatenated with the reconstructed features obtained by upsampling the decoder after skip connections, finally obtaining a reconstructed frame with the same size as the input frame. After obtaining the reconstructed pseudo-anomaly frames and reconstructed real frames, a maximum and minimum constraint is applied between them and the real frames to create an adversarial relationship with the real frames, maximizing the distance between the reconstructed pseudo-anomaly frames and the real frames, and minimizing the distance between the reconstructed real frames and the real frames.

[0022] Furthermore, the specific process of obtaining features with high information relevance through the second-order channel attention module in step S3 is as follows:

[0023] After inputting the feature map into the second-order channel attention module, a global covariance pooling operation is first performed, transforming the input feature into a C×H×W matrix X, and calculating its corresponding covariance matrix ∑, as shown in the following formula:

[0024]

[0025]

[0026] Where I and O are the s×s identity matrix (with 1s on the diagonal) and the all-1 matrix, respectively;

[0027] Using ∑ for covariance normalization, since this matrix is ​​a symmetric positive definite matrix, its eigenvalue decomposition formula is as follows:

[0028] ∑=UAU T

[0029] U is an orthogonal matrix, and Λ refers to the diagonal matrix whose eigenvalues ​​are not increasing.

[0030] The covariance normalization is converted into a power of the eigenvalues, as shown in the following formula:

[0031]

[0032] α represents a positive real number, Λ α This represents a diagonal matrix. When α = 1, it does not need to be standardized. When α < 1, it will non-linearly shrink eigenvalues ​​greater than 1.0. Usually, α = 0.5 is set.

[0033] Next, the channel attention vector is processed through two convolutional layers. The first convolutional layer is used for channel dimensionality reduction, and the second convolutional layer is used for channel restoration, resulting in the final second-order channel attention vector ω, as shown in the following formula:

[0034] ω=f(W U δ(W D z))

[0035] W U and W D δ represents the weights of the convolutional layer, the feature channels are C / r and C, f(·) represents the sigmoid function, and δ represents the ReLU function.

[0036] set up Channel statistics z∈R C×1 It can be contracted The formula for the c-th dimension of z is shown below:

[0037]

[0038] Among them, H GCP (·) denotes the global covariance pooling function; compared with first-order pooling, global covariance pooling can obtain higher-order feature information and more discriminative feature information;

[0039] Finally, the second-order channel attention vector is multiplied by the input features by channel to obtain new features related to internal information.

[0040] Furthermore, in step S2, the noise generator is optimized using appearance constraints, which are divided into gradient constraints L. gc With strength constraint L sc The formula is as follows:

[0041]

[0042]

[0043] Where a and b represent the horizontal and vertical coordinates of the video frame pixels;

[0044] By gradient constraint L gc and strength constraint L sc The apparent constraint L can be obtained. app The formula is as follows:

[0045] L app =mL gc +nL sc

[0046] Where m∶n=1∶1;

[0047] The specific optimization process is as follows:

[0048] During the training phase of the noise generator, the generator with a noise-free module is trained first. In order to narrow the gap between the reconstructed frame and the real frame, the discriminator is fixed, and the generator is constrained at the appearance level. The objective function is as follows:

[0049]

[0050] Next, the corresponding discriminator is trained while the generator is fixed, enabling the discriminator to classify reconstructed frames as anomalous frames and real frames as normal frames, thus accurately classifying reconstructed and real frames. After label smoothing, the label outlier value 0 representing "normal" and the label outlier value 1 representing "abnormal" are replaced with 0.05 and 0.95 respectively. The loss function is:

[0051]

[0052] Where i and j are the indices of the frames, D(·)∈[0,1], and L(·,·) represents the absolute value of the difference between the two.

[0053] After obtaining the trained generator and discriminator, a noise module is added to the generator, the discriminator is fixed, and a constraint function is added at the appearance level to increase the distance between the generated frame and the real frame. The parameters of the noise module are then updated, ultimately enabling the noise generator to generate pseudo-abnormal frames. The objective function is shown below:

[0054]

[0055] Where θ n These are the parameters for the noise module.

[0056] Furthermore, in step S3, the reconstruction generator is optimized using maximum constraint functions and minimum constraint functions, specifically as follows:

[0057] Regarding pseudo-abnormal frames, maximum constraints are used to widen the distance between the reconstructed pseudo-abnormal frames and the real frames. Intensity constraints are used to increase the inter-pixel distance between the reconstructed pseudo-abnormal frames and the real frames. Gradient constraints are used to increase the difference between the distance between adjacent pixels in the reconstructed pseudo-abnormal frames and the distance between adjacent pixels in the real frames. At the same time, optical flow constraints are added to constrain it at the motion level.

[0058] The motion constraint is the optical flow constraint, and its formula is shown below:

[0059]

[0060] Where, x t This represents the t-th frame;

[0061] The maximum constraint formula is shown below:

[0062]

[0063] λ gc , λ sc , λ opt These are the weights for gradient, intensity, and optical flow, respectively.

[0064] In terms of real frames, the minimum constraint is used to shorten the distance between reconstructed real frames. At the appearance level, the inter-frame distance and intra-frame distance between reconstructed real frames are shortened. At the motion level, the optical flow similarity between reconstructed real frames and the previous real frame is improved.

[0065] The minimum constraint formula is shown below:

[0066]

[0067] Furthermore, in step S4, the reconstruction generator is trained using pseudo-abnormal frames and normal training frames, enabling the generator to identify abnormal distributions in advance and improve its ability to perceive anomalies. During the testing phase, test frames are input into the trained reconstruction generator to generate reconstructed test frames. The reconstructed frames are compared with the real frames, the reconstruction error is calculated and normalized to obtain a reconstruction score. When the reconstruction score is greater than a set threshold, the frame is determined to contain an abnormal event and is an abnormal frame; conversely, the frame is determined not to contain an abnormal event and is a normal frame.

[0068] The formula for calculating the reconstruction error is:

[0069] S(x″) = λL app +(1-λ)L opt

[0070] Where λ is the weighting parameter;

[0071] After obtaining the reconstruction error, it is normalized to obtain the reconstruction score:

[0072]

[0073] If the reconstruction score of frame t is less than the threshold, the frame is determined to be a normal frame; otherwise, the video frame contains an abnormal event.

[0074] The implementation of the method in this invention mainly includes a noise generator, a reconstruction generator, a discriminator, and a second-order channel attention module. The noise generator is mainly used to generate pseudo-abnormal frames, and the reconstruction generator aims to comprehensively learn the distribution of training video frames. In addition, a second-order channel attention module is introduced to enhance the model's learning ability.

[0075] This invention classifies anomalous events in videos by reconstructing video frames and calculating the reconstruction error. In anomaly detection methods for video reconstruction, autoencoders are commonly used deep learning networks, which are effective at extracting and reconstructing the latent features of video frames. However, autoencoders can produce blurred edges during frame reconstruction. To avoid this problem, this invention uses a generative adversarial network (GAN) as the main model structure, while incorporating the autoencoder as a generator to ensure stable training of the network model and improve the clarity of the reconstructed frames. To enhance the contextual relevance of the extracted feature information, a second-order channel attention module is added to the generator. This module learns the internal dependencies of features through the distribution of second-order features, enabling the detection model to focus on more relevant information and ultimately improving detection capabilities.

[0076] To reduce the probability of the model perfectly reconstructing anomalous frames due to its strong reconstruction capabilities, this invention incorporates a generator into the generative adversarial network (GAN). This generator transforms normal training frames into pseudo-anomalous frames, supplementing the anomalous frames lacking during the original training. The pseudo-anomalous frames, along with the original training frames, jointly train the existing generator in the GAN, enabling it to learn anomalous distribution patterns in advance and improving its ability to perceive unknown anomalous events.

[0077] In the training phase, the method of this invention trains a noise generator and a reconstruction generator separately. The former generates pseudo-abnormal frames using normal training frames, while the latter is trained using both normal training frames and the generated pseudo-abnormal frames to improve its ability to reconstruct normal frames. In the testing phase, test frames are input into the trained reconstruction generator, and the generated reconstruction scores are used to classify whether video frames are abnormal.

[0078] Based on the above, the method of the present invention can be divided into three parts: training the noise generator, training the reconstruction generator, and judging abnormal frames. The specific analysis is as follows:

[0079] Training the noise generator: The noise generator uses a noise module to generate pseudo-abnormal frames and narrows the gap between pseudo-abnormal frames and normal frames through adversarial gameplay with the discriminator. To make the contextual information in the video frames more closely related, a second-order channel attention module is added to the generator to obtain more useful information from the video frames, thereby improving the discriminative learning ability.

[0080] Training the Reconstruction Generator: The reconstruction generator is trained using pseudo-abnormal frames and normal frames. A minimax constraint is added to challenge it against real frames, narrowing the gap between reconstructed normal frames and real frames, and widening the gap between reconstructed pseudo-abnormal frames and real frames. This allows the reconstruction generator to better learn the learning distribution of normal video frames and improve its ability to perceive abnormal frames. To strengthen the connection between contextual information in video frames, a second-order channel attention module is added to the generator to obtain more useful information from the video frames, thereby improving its discriminative learning ability.

[0081] Abnormal frame identification: The test frame is reconstructed using the trained reconstruction generator. The distance between the reconstructed frame and the real frame is calculated and normalized to obtain the reconstruction score. The video frame is then used to determine whether it is an abnormal frame.

[0082] This invention proposes a video anomaly detection method based on a generative adversarial network (GAN) with dual generators and a channel attention mechanism. The dual-generator GAN constructs two generators with different functions: a reconstruction generator and a noise generator. The noise generator produces pseudo-anomaly frames using normal training frames. The reconstruction generator is trained jointly by the generated pseudo-anomaly frames and normal training frames, improving its ability to detect anomalies. Simultaneously, a second-order channel attention module is introduced into the generator to increase the interdependence of information within features and improve the correlation of information in the feature maps. Ultimately, this improves the detection accuracy of video anomaly events.

[0083] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0084] 1. This invention improves upon previous methods that only used normal frames to train models by designing a noise generator to generate pseudo-abnormal frames for training. A video anomaly detection method with dual generators and a channel attention mechanism is proposed. After the noise generator generates pseudo-abnormal frames, the reconstruction generator is trained using both pseudo-abnormal frames and normal frames. This allows the reconstruction generator to learn the anomaly distribution in advance, improving its ability to perceive anomalies.

[0085] 2. In the dual-generator generative adversarial network, this invention adds a second-order channel attention module to the generator so that the model can better learn the intra-frame interdependencies, improve the correlation of information in the feature map, make better use of important information, and ultimately improve the anomaly detection accuracy.

[0086] 3. This invention employs a maximum-minimum constraint function, enabling the dual-generator generative adversarial network to converge better. This allows the trained model to ensure that normal frames are close to reconstructed frames and abnormal frames are far from their reconstructed frames during testing, which is beneficial for improving the final anomaly detection accuracy. Attached Figure Description

[0087] Figure 1 The diagram shown is an overall framework diagram of the video anomaly detection method of the present invention.

[0088] Figure 2 The diagram shown is a noise generator structure diagram of the dual-generator generative adversarial network of the present invention.

[0089] Figure 3 The diagram shown is a structural diagram of the noise module of the noise generator of the present invention;

[0090] Figure 4 The diagram shown is a reconstruction generator structure diagram of the dual-generator generative adversarial network of the present invention.

[0091] Figure 5 The diagram shown is a structural diagram of the second-order channel attention module of the present invention. Detailed Implementation

[0092] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0093] like Figure 1 As shown, this invention provides a video anomaly detection method based on a dual generator and channel attention mechanism, comprising the following steps:

[0094] S1: Split the video set to obtain multiple frame-level sequences and divide them into normal training video frames and test video frames;

[0095] S2: Train the noise generator using normal training video frames and the discriminator, and generate pseudo-abnormal frames using the trained noise generator.

[0096] S3: Train the reconstruction generator using pseudo-abnormal frames and normal training video frames to obtain the trained reconstruction generator;

[0097] S4: Input the test video frame into the trained reconstruction generator to obtain the reconstructed frame, calculate the reconstruction error between the reconstructed frame and the real frame, and classify the video frame as anomaly based on the reconstruction error.

[0098] I. The following provides a detailed explanation of steps S2 to S4:

[0099] In step S2:

[0100] like Figure 2 As shown, the noise generator includes a noise module, a generator, and a second-order channel attention module. The noise generator is trained twice. The first training is performed on a generator without a noise module, and the second training is performed on a noise generator with a noise module. The training method is as follows: the generator is trained by adversarially interacting with the discriminator, so that the generated frames cannot be distinguished as abnormal by the discriminator. After obtaining the trained generator, the parameters of the generator are fixed, and a noise module is added to the generator. The generator is then subjected to adversarial training with the discriminator again, and the parameters of the noise module are updated to obtain the noise generator.

[0101] The generator uses an autoencoder network, consisting of an encoder and a decoder. The training method for the generator is as follows:

[0102] When a normal training video frame is input into the generator, it first enters the encoder part, which consists of pooling layers, convolutional layers, and repeated activation functions. After multiple halvings in size and doublings in channels, the latent features are obtained.

[0103] Secondly, the latent features are input into the second-order channel attention module to improve the interdependence of features, obtain richer and denser information correlation, and obtain features that are closely related to global features and local features.

[0104] Finally, the obtained features are input into the decoder to reconstruct the latent features, and the output is an image of the same size as the input frame, thereby generating a high-quality reconstructed frame.

[0105] The reconstructed frame and the real frame are input into the discriminator together, so that the discriminator and the generator compete against each other, and the reconstructed frame generated by the generator is judged by the discriminator as a real frame.

[0106] like Figure 3 As shown, the noise module consists of a three-layer fully connected autoencoder, including noise module A and noise module B. The training method for the noise generator through the noise module is as follows:

[0107] Random noise is input into the noise module, and three convolutions, batch normalization, and activation function operations are performed to obtain noise suitable for generating pseudo video frames.

[0108] After obtaining the trained generator, a noise module is added to the generator. Skip connections are used after the encoder’s first convolutional pooling and before the decoder’s last upsampling, and noise module A is added. After obtaining the latent features, noise module B is added. The latent features with noise are reconstructed to obtain pseudo-abnormal frames.

[0109] The pseudo-abnormal frames and real frames are input into the discriminator together, so that the discriminator and the generator compete against each other to maximize the distance between the two frames. The size of noise module A and noise module B is the same as the corresponding feature size.

[0110] In step S3:

[0111] The training method for the reconstruction generator is as follows: maximum constraint function and minimum constraint function are added to the reconstruction pseudo-abnormal frames and reconstruction normal training video frames, respectively, so that the reconstruction pseudo-abnormal frames, reconstruction real frames and real frames compete with each other, widening the distance between the reconstruction pseudo-abnormal frames and real frames, and narrowing the distance between the reconstruction real frames and real frames.

[0112] like Figure 4 As shown, the reconstruction generator consists of an autoencoder and a second-order channel attention module. The autoencoder comprises an encoder and a decoder, using U-net as its network structure. The specific training process is as follows: pseudo-anomaly frames and real frames are input into the reconstruction generator. First, they enter the encoder module, where multiple convolutional pooling processes are performed to obtain latent features of different sizes and channel numbers at each layer. Next, the latent features of each layer are input into the second-order channel attention module to learn interdependent and closely related information, resulting in features with high information correlation. The features of each layer are then concatenated with the reconstructed features obtained by upsampling the decoder after skip connections, ultimately yielding a reconstructed frame with the same size as the input frame. After obtaining the reconstructed pseudo-anomaly frames and reconstructed real frames, a maximum-minimum constraint is applied between them and the real frames to create an adversarial relationship with the real frames, maximizing the distance between the reconstructed pseudo-anomaly frames and the real frames, and minimizing the distance between the reconstructed real frames and the real frames.

[0113] like Figure 5 As shown, the second-order channel attention module explores attention based on second-order feature statistics, building upon the first-order channel attention module. It employs global covariance pooling and uses Newton's iteration method to solve for covariance normalization, thereby reducing the required computational resources. The specific process of obtaining highly correlated features through the second-order channel attention module is as follows:

[0114] After inputting the feature map into the second-order channel attention module, a global covariance pooling operation is first performed, transforming the input feature into a C×H×W matrix X, and calculating its corresponding covariance matrix ∑, as shown in the following formula:

[0115]

[0116]

[0117] Where I and O are the s×s identity matrix (with 1s on the diagonal) and the all-1 matrix, respectively;

[0118] Using ∑ for covariance normalization, since this matrix is ​​a symmetric positive definite matrix, its eigenvalue decomposition formula is as follows:

[0119] ∑=UAU T

[0120] U is an orthogonal matrix, and Λ refers to the diagonal matrix whose eigenvalues ​​are not increasing.

[0121] The covariance normalization is converted into a power of the eigenvalues, as shown in the following formula:

[0122]

[0123] α represents a positive real number, Λ α This represents a diagonal matrix. When α = 1, it does not need to be standardized. When α < 1, it will non-linearly shrink eigenvalues ​​greater than 1.0. Usually, α = 0.5 is set.

[0124] Next, the channel attention vector is processed through two convolutional layers. The first convolutional layer is used for channel dimensionality reduction, and the second convolutional layer is used for channel restoration, resulting in the final second-order channel attention vector ω, as shown in the following formula:

[0125] ω=f(W U δ(W D z))

[0126] W U and W D δ represents the weights of the convolutional layer, the feature channels are C / r and C, f(·) represents the sigmoid function, and δ represents the ReLU function.

[0127] set up Channel statistics z∈R C×1 It can be contracted The formula for the c-th dimension of z is shown below:

[0128]

[0129] Among them, H GCP (·) denotes the global covariance pooling function; compared with first-order pooling, global covariance pooling can obtain higher-order feature information and more discriminative feature information;

[0130] Finally, the second-order channel attention vector is multiplied by the input features by channel to obtain new features related to internal information.

[0131] In step S4:

[0132] The reconstruction generator is trained using pseudo-abnormal frames and normal training frames, enabling it to identify anomalous distributions in advance and improving its ability to detect anomalies. During the testing phase, test frames are input into the trained reconstruction generator to generate reconstructed test frames. The reconstructed frames are compared with the real frames, the reconstruction error is calculated and normalized to obtain a reconstruction score. When the reconstruction score is greater than a set threshold, the frame is determined to contain an anomalous event and is considered an anomalous frame; conversely, the frame is determined not to contain an anomalous event and is considered a normal frame.

[0133] The formula for calculating the reconstruction error is:

[0134] S(x″) = λL opp +(1-λ)L opt

[0135] Where λ is the weighting parameter;

[0136] After obtaining the reconstruction error, it is normalized to obtain the reconstruction score:

[0137]

[0138] If the reconstruction score of frame t is less than the threshold, the frame is determined to be a normal frame; otherwise, the video frame contains an abnormal event.

[0139] II. Optimization of Dual-Generator Generative Adversarial Networks:

[0140] The dual-generator generative adversarial network (GAN) obtained in the first part above also needs to be optimized simultaneously. To ensure that the dual-generator GAN converges better during training to achieve the desired effect, constraint functions are used to constrain the dual-generator GAN. To ensure that the pseudo-anomaly frames generated by the noise generator are far from the real frames, the reconstructed pseudo-anomaly frames generated by the reconstruction generator are far from the real frames, and the reconstructed real frames are close to the real frames, constraints are applied at the appearance and motion levels.

[0141] In step S2, the noise generator is optimized using apparent constraints, which are divided into gradient constraints L. gc With strength constraint L sc The formula is as follows:

[0142]

[0143]

[0144] Where a and b represent the horizontal and vertical coordinates of the video frame pixels;

[0145] By gradient constraint L gc and strength constraint L sc The apparent constraint L can be obtained. appThe formula is as follows:

[0146] L app =mL gc +nL sc

[0147] Where m∶n=1∶1;

[0148] The specific optimization process is as follows:

[0149] During the training phase of the noise generator, the generator with a noise-free module is trained first. In order to narrow the gap between the reconstructed frame and the real frame, the discriminator is fixed, and the generator is constrained at the appearance level. The objective function is as follows:

[0150]

[0151] Next, the corresponding discriminator is trained while the generator is fixed, enabling the discriminator to classify reconstructed frames as anomalous frames and real frames as normal frames, thus accurately classifying reconstructed and real frames. After label smoothing, the label outlier value 0 representing "normal" and the label outlier value 1 representing "abnormal" are replaced with 0.05 and 0.95 respectively. The loss function is:

[0152]

[0153] Where i and j are the indices of the frames, D(·)∈[0,1], and L(·,·) represents the absolute value of the difference between the two.

[0154] After obtaining the trained generator and discriminator, a noise module is added to the generator, the discriminator is fixed, and a constraint function is added at the appearance level to increase the distance between the generated frame and the real frame. The parameters of the noise module are then updated, ultimately enabling the noise generator to generate pseudo-abnormal frames. The objective function is shown below:

[0155]

[0156] Where θ n These are the parameters for the noise module.

[0157] In step S3, the reconstruction generator is optimized using maximum and minimum constraint functions, specifically as follows:

[0158] Regarding pseudo-abnormal frames, maximum constraints are used to widen the distance between the reconstructed pseudo-abnormal frames and the real frames. Intensity constraints are used to increase the inter-pixel distance between the reconstructed pseudo-abnormal frames and the real frames. Gradient constraints are used to increase the difference between the distance between adjacent pixels in the reconstructed pseudo-abnormal frames and the distance between adjacent pixels in the real frames. At the same time, optical flow constraints are added to constrain it at the motion level.

[0159] The motion constraint is the optical flow constraint, and its formula is shown below:

[0160]

[0161] Where, x t This represents the t-th frame;

[0162] The maximum constraint formula is shown below:

[0163]

[0164] λ gc , λ sc , λ opt These are the weights for gradient, intensity, and optical flow, respectively.

[0165] In terms of real frames, the minimum constraint is used to shorten the distance between reconstructed real frames. At the appearance level, the inter-frame distance and intra-frame distance between reconstructed real frames are shortened. At the motion level, the optical flow similarity between reconstructed real frames and the previous real frame is improved.

[0166] The minimum constraint formula is shown below:

[0167]

[0168] Based on the above, in order to verify the effectiveness of the present invention, this embodiment applies the above solution to an example, as follows:

[0169] Evaluation experiments were conducted on two publicly available datasets: UCSD Ped1 & Ped2 and CUHK Avenue. Both datasets contain training and testing sets; the videos in the testing set contain anomalous events. During training, the training sets from UCSD Ped1 & Ped2 and CUHK Avenue were input, enabling the noise generator to output pseudo-anomaly frames. The reconstruction generator then used these training frames and the generated pseudo-anomaly frames to improve its ability to reconstruct normal frames and its ability to detect anomalies. During testing, test frames were input into the trained reconstruction generator, and a corresponding reconstruction score was obtained. This score was used to distinguish whether the input frame was normal or anomalous.

[0170] The entire experiment was conducted on an NVIDIA GeForce RTX 3080Ti (12GB GPU memory) using the PyTorch deep learning framework. The experimental results were evaluated using the area under the curve (AUC) metric. AUC is a metric used to evaluate the performance of a binary classifier. It represents the area under the ROC curve (a line graph showing the probability of correctly classifying a positive sample versus misclassifying a negative sample) at a given probability threshold. A higher AUC value indicates better classifier performance; a value of 1 represents perfect classification. In video anomaly detection algorithms, a higher AUC value indicates more accurate anomaly identification.

[0171] In this embodiment, the detection method of the present invention is compared with several advanced video anomaly detection methods, as shown in Table 1. Experimental results show that the video anomaly detection method based on dual generators and channel attention mechanism provided by the present invention has a higher AUC, which is significantly better than other comparative methods.

[0172] Experiments on the Ped1 dataset show that this invention improves anomaly detection accuracy. Specifically, compared to R-VAE, DDGAN, and Attention Prediction, the AUC of this invention is improved by 10.7%, 2.9%, and 1.8%, respectively. On the Ped2 dataset, although the AUC of the two state-of-the-art methods, ASTNet and SSMTL, reaches 97%, there is still a difference compared to this invention. The AUCs obtained by other methods are significantly lower than those of this method. This also confirms that this invention still has certain advantages in detecting anomalies in Ped2.

[0173] On the Avenue dataset, ASTNet's AUC is only 0.5% higher than that of this invention. SSMTL's AUC is significantly higher than that of this invention, presumably due to the self-supervised learning and teacher-student model used in this method, but this method cannot enable the model to have some awareness of anomalies in advance during the training phase. Other state-of-the-art methods have significantly lower AUCs than this invention.

[0174] Table 1 Comparison of AUC with other methods

[0175]

[0176]

[0177] The experimental results show that because the reconstruction generator was trained using pseudo-video frames, it had a certain understanding of anomalies in advance. Therefore, when judging video frames, it reduced the probability of abnormal frames being reconstructed normally, thus improving the AUC of the present invention.

[0178] To investigate the effects of noise module A, noise module B, and the second-order channel attention module on the detection accuracy of this invention, ablation experiments were conducted under different conditions on the Ped1, Ped2, and Avenue datasets, as shown in Table 2.

[0179] Table 2 Comparison of AUC for each module

[0180]

[0181] On these datasets, the AUC of the dual-generator GAN detection model, which includes both a noise generator and a reconstruction generator, is higher than that of the GAN detection model with only a reconstruction generator. The AUC of the dual-generator GAN detection model employing a second-order channel attention module is also higher than that of the dual-generator GAN detection model without a second-order channel attention module.

[0182] Specifically, using the model without noise module A, noise module B, and the second-order channel attention module as the baseline, adding only noise module A improves the AUC of the three datasets (Ped1, Ped2, and Avenue, hereinafter the same) by 1.9%, 2.4%, and 0.4%, respectively. Adding only noise module B improves the AUC by 0.6%, 1.2%, and 2.2%, respectively. Adding both noise modules A and B improves the AUC by 2.6%, 3.3%, and 1.6%, respectively. Adding all three modules results in a significant performance improvement (+3.4%, +4.4%, and +2.5%).

[0183] Experiments on two public datasets demonstrate the superiority of the method of this invention and show that the method can effectively detect abnormal frames after pre-learning the pseudo-anomaly distribution. The experimental data verifies that the method of this invention can effectively improve the detection accuracy of video anomaly events.

Claims

1. A method for video anomaly detection based on dual generator and channel attention mechanism, characterized in that, The method comprises the following steps: S1: split the video set to obtain multiple frame-level sequences, and divide the frame-level sequences into normal training video frames and test video frames; S2: train the noise generator using the normal training video frames and the discriminator, generate pseudo abnormal frames through the trained noise generator; S3: train the reconstruction generator using the pseudo abnormal frames and the normal training video frames to obtain a trained reconstruction generator; S4: input the test video frames into the trained reconstruction generator to obtain reconstructed frames, calculate the reconstruction error of the reconstructed frames and the real frames, and classify the video frames according to the reconstruction error; The training method of the reconstruction generator in step S3 is that: the maximum constraint function and the minimum constraint function are added to the reconstructed pseudo abnormal frames and the reconstructed normal training video frames respectively, so that the reconstructed pseudo abnormal frames and the reconstructed real frames are in confrontation with the real frames, the distance between the reconstructed pseudo abnormal frames and the real frames is far away, and the distance between the reconstructed real frames and the real frames is close; The reconstruction generator in step S3 is composed of an autoencoder and a second-order channel attention module, wherein the autoencoder is composed of an encoder and a decoder, and adopts U-net as the network structure, and the specific training process is as follows: the pseudo abnormal frames and the real frames are input into the reconstruction generator, first enter the encoder module, and after multiple convolution and pooling processing, the latent features of different sizes and different channel numbers are obtained at each layer; Secondly, each layer of latent features is input into the second-order channel attention module to learn the correlation information with higher mutual dependence and higher closeness, and features with high information correlation are obtained; each layer of features is spliced with each layer of reconstructed features obtained after the decoder upsampling, and finally the reconstructed frames with the same size as the input frames are obtained; after obtaining the reconstructed pseudo abnormal frames and the reconstructed real frames, the maximum and minimum constraints are performed between them and the real frames, and the real frames are in confrontation, the distance between the reconstructed pseudo abnormal frames and the real frames is maximized, and the distance between the reconstructed real frames and the real frames is minimized.

2. The video anomaly detection method based on dual generator and channel attention mechanism according to claim 1, characterized in that, The noise generator in step S2 comprises a noise module, a generator and a second-order channel attention module, and the noise generator is trained twice, the first time is to train the generator without the noise module, and the second time is to train the noise generator with the noise module, and the training method is as follows: the generator is trained through mutual confrontation with the discriminator, so that the generated frames cannot be distinguished as abnormal by the discriminator, after obtaining the trained generator, the parameters of the generator are fixed, the noise module is added to the generator, and the generator is generated again to confront the discriminator, the parameters of the noise module are updated, and the noise generator is obtained.

3. The video anomaly detection method based on dual generator and channel attention mechanism according to claim 2, characterized in that, The generator in step S2 adopts an autoencoder network composed of an encoder and a decoder, and the training method of the generator is as follows: When the normal training video frames are input into the generator, first enter the encoder part composed of a pooling layer, a convolution layer and an activation function repeatedly, and after multiple size reduction and channel doubling, latent features are obtained; Secondly, the latent features are input into the second-order channel attention module to obtain features with closely related global features and local features; Finally, the obtained features are input into the decoder to reconstruct the latent features, and images with the same size as the input frames are output, so that high-quality reconstructed frames are generated; The obtained reconstructed frame is input into the discriminator together with the real frame, so that the discriminator and the generator are in confrontation, and the reconstructed frame generated by the generator is determined as a real frame by the discriminator.

4. The video anomaly detection method based on dual generator and channel attention mechanism according to claim 3, characterized in that, The noise module in the step S2 is composed of a self-encoder with a three-layer full connection structure, including a noise module A and a noise module B, and a training method of the noise module for the noise generator is as follows: Random noise is input into the noise module, three convolution, batch normalization and activation function operations are performed, and noise suitable for generating a pseudo abnormal frame is obtained; After obtaining the trained generator, the noise module is added to the generator, a skip connection is adopted after the first convolution and pooling of the encoder, and the noise module A is added, the noise module B is added after obtaining the latent feature, and the latent feature with noise is reconstructed to obtain a pseudo abnormal frame; The pseudo abnormal frame and the real frame are input into the discriminator, so that the discriminator and the generator are in confrontation, and the distance between the two frames is maximized, wherein the size of the noise module A and the noise module B is the same as the corresponding feature size.

5. The video anomaly detection method based on dual generator and channel attention mechanism according to claim 4, characterized in that, The specific process of obtaining the feature with high information correlation degree through the second-order channel attention module in the step S3 is as follows: After the feature map is input into the second-order channel attention module, first, global covariance pooling operation is performed, the input feature is changed into a CxHxW matrix X, and the corresponding covariance matrix ∑ is calculated, and the formula is as follows: Wherein, I and O are respectively an sxs unit matrix and a full 1 matrix; ∑ is used for covariance normalization, since the matrix is a symmetric positive definite matrix, the eigenvalue decomposition formula is as follows: ∑ = UΛU T U is an orthogonal matrix, and Lambda refers to a diagonal matrix of eigenvalues when not increasing; The covariance normalization is converted into the power of the eigenvalue, and the formula is as follows: a denotes a positive real number, Λ α denotes a diagonal matrix, when a = 1, no normalization is needed, when a < 1, it nonlinearly shrinks eigenvalues greater than 1.0; Secondly, enter the channel attention, pass through two layers of convolution, the first layer of convolution is used for channel dimension reduction, and the second layer of convolution is used for channel restoration, to obtain the final second-order channel attention vector omega, and the formula is as follows: ω = f(W U δ(W D z)) W U and W D are the weights of the convolutional layer, the feature channels are C / r and C, f(·) represents the sigmoid function, and δ represents the RELU function; Let The channel statistics z e R C×1 By contracting The cth dimension of z is given by where H GCP (·) denotes a global covariance pooling function; Finally, the second-order channel attention vector is multiplied with the input feature according to the channel to obtain a new internal information related feature.

6. The video anomaly detection method based on dual generator and channel attention mechanism according to claim 5, characterized in that, The step S2 uses apparent constraints to optimize the noise generator, the apparent constraints are divided into gradient constraints L gc and intensity constraints L sc The formula is as follows: Wherein, a, b represent the horizontal and vertical coordinates of the video frame pixels; The apparent constraint L gc can be obtained from the gradient constraint L sc and the intensity constraint L app and is given by the following equation: L app = mL gc + nL sc Wherein m:n = 1:1; The specific optimization process is as follows: In the training of the noise generator stage, first, the generator without noise module is trained, in order to narrow the distance between the reconstructed frame and the real frame, the discriminator is fixed, and the generator is constrained on the surface, and the objective function is as follows: Secondly, the corresponding discriminator is trained, the generator is fixed, so that the discriminator can classify the reconstructed frame as an abnormal frame and the real frame as a normal frame, so as to accurately classify the reconstructed frame and the real frame; after label smoothing processing, the label abnormal value 0 representing "normal" and the label abnormal value 1 representing "abnormal" are replaced by 0.05 and 0.95 respectively; and the loss function is as follows: Wherein, i, j are the indexes of the frame, D(·)∈[0,1], and L(·,·) represents the absolute value of the difference between the two. After obtaining the trained generator and discriminator, a noise module is added to the generator, the discriminator is fixed, a constraint function is added to the apparent layer to pull away the distance between the generated frame and the real frame, the parameters of the noise module are updated, and finally the noise generator generates pseudo abnormal frames. The objective function is as follows: where θ n is a parameter of the noise module.

7. The video anomaly detection method based on dual generator and channel attention mechanism according to claim 6, characterized in that, In step S3, the maximum constraint function and the minimum constraint function are used to optimize the reconstruction generator, specifically: In terms of pseudo abnormal frames, the maximum constraint is used to pull away the distance between the reconstructed pseudo abnormal frame and the real frame, the intensity constraint is used to increase the pixel frame distance between the reconstructed pseudo abnormal frame and the real frame, the gradient constraint is used to increase the difference between the distance of adjacent pixel points in the reconstructed pseudo abnormal frame and the distance of adjacent pixel points in the real frame, and the optical flow constraint is added to constrain it at the motion level; The motion constraint is the optical flow constraint, and its formula is as follows: wherein x t denotes the t-th frame; The maximum constraint formula is as follows: λ gc , λ sc , λ opt are the weights of the gradient, intensity and optical flow, respectively; In terms of real frames, the minimum constraint is used to pull close the distance between the reconstructed real frame and the real frame, the apparent layer is used to pull close the inter-frame distance and the intra-frame distance between the reconstructed real frame and the real frame, and the motion level is used to improve the optical flow similarity between the reconstructed real frame and the real frame and the previous time real frame. The minimum constraint formula is as follows:

Images