Video anomaly detection method based on infrared visible light image feature fusion

Abstract

The invention belongs to the field of computer vision, provides a video anomaly detection algorithm based on infrared visible light image feature fusion, and aims to solve the problem of video anomaly detection in an insufficient illumination condition or a low-visibility environment. A convolutional neural network encoder is utilized to extract visible light features of a video frame, a multi-stage memory encoder is utilized to perform enhancement and hierarchical memory on significant features in an infrared image to obtain enhanced infrared features, and finally a result is output through an upper sampling layer of another convolutional neural network. Deep fusion is carried out on the infrared and visible light features to obtain fusion features, and finally, a decoder corresponding to the convolutional neural network encoder is guided to reconstruct a target frame according to the fusion features and the enhanced infrared features. And detecting and judging whether an abnormal behavior exists or not through the target frame. The method is mainly applied to video anomaly detection system design and manufacturing occasions.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and relates to a video anomaly detection method and system based on the fusion of infrared and visible light multimodal features. The method is applicable to all-weather monitoring scenarios in low-light environments. Background Technology

[0002] With the rapid development of smart cities, intelligent security, traffic monitoring, and other application scenarios, the demand for real-time video data processing and intelligent recognition of abnormal behavior is becoming increasingly urgent. Traditional video anomaly detection methods rely on manual monitoring, which is not only inefficient and costly, but also prone to missed detections and cannot meet the needs of scenarios requiring rapid identification and response. In recent years, the rapid development of computer vision technology and deep learning methods have demonstrated enormous potential in automated video anomaly detection, making fully automated video anomaly behavior detection with high detection accuracy and fast response speed possible.

[0003] Currently, deep learning-based video anomaly detection models primarily rely on visible light modalities. They perform exceptionally well under ideal lighting conditions, providing high-resolution, color-rich images that align with human visual habits, facilitating scene detail capture and target recognition. However, visible light video anomaly detection models are highly dependent on ambient lighting conditions. In low light, strong backlight, fog, or target camouflage, their imaging quality deteriorates drastically or even fails, leading to the loss of crucial information or difficulty in target identification, severely impacting model robustness and detection accuracy. Furthermore, the autoencoder structures used in most existing models exhibit excessive generalization ability, often successfully reconstructing anomalous samples even when faced with abnormal data, resulting in missed detections. Particularly in complex scenes, when the feature distributions of anomalous and normal samples are similar, the model's discriminative ability weakens, failing to effectively distinguish between the two, thus affecting detection accuracy. To overcome these challenges and improve the model's perception capabilities under low light and adverse weather conditions, video anomaly detection models fusing infrared and visible light modal information are receiving increasing attention. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention aims to propose a video anomaly detection algorithm based on infrared and visible light image feature fusion. This algorithm addresses the problem of video anomaly detection in environments with insufficient lighting or low visibility. By constructing an infrared and visible light collaborative reconstruction system, cross-modal feature fusion is achieved, overcoming the limitations of traditional single-modal detection in low-light scenarios. The technical solution adopted in this invention is a video anomaly detection method based on infrared and visible light image feature fusion. First, a convolutional neural network encoder extracts visible light features from video frames. Simultaneously, a multi-level memory encoder enhances and hierarchically memorizes salient features in the infrared image to obtain enhanced infrared features. The multi-level memory encoder consists of an alternating connection of an encoding module and a memory module based on another convolutional neural network. Finally, the result is output through the upsampling layer of the other convolutional neural network. Then, the infrared and visible light features are deeply fused to obtain fused features. Finally, the decoder corresponding to the convolutional neural network encoder is guided to reconstruct the target frame based on the fused features and enhanced infrared features. Anomaly behavior inference: An anomaly score is generated based on the original frame and the reconstructed target frame. When the anomaly score exceeds a preset threshold, it indicates that anomaly behavior has been detected.

[0005] The detailed steps are as follows:

[0006] Step 1: Data Preprocessing: Preprocess the dataset, which contains both normal and abnormal samples; preprocess the input visible light modal video frame sequence. and infrared modal video frame sequences The video frames are normalized to the range of [-1,1] and the resolution is adjusted to 256×256. B represents the batch size, H represents the image height, W represents the image width, and C represents the number of feature channels.

[0007] Step 2, Feature Extraction and Fusion: Using an encoder Extract visible light features from video frames Infrared modal features were extracted using a multi-level memory encoder (MMAE). Calculate and store enhanced infrared modal features Then, the visible light features and infrared features are fused using attention features to generate the fused feature F. l According to the fusion feature F l and enhanced infrared signature Generate the latent space Z;

[0008] Step 3, Network Training: Introduce enhanced infrared modal features stored in the memory module. Establish the empirical distribution of the latent variable z By minimizing the true Bayesian posterior distribution between the two distributions Introducing fusion feature F l and real frame Vk We establish the KL divergence after approximate Bayesian analysis of the latent variable z to guide the convolutional neural network visible light feature encoder and convolutional neural network infrared feature encoder to learn the distribution of the latent space Z.

[0009] Step 4, Frame Reconstruction: Decoder z0 is obtained by sampling from the generated latent space Z, and z0 is combined with infrared features splicing, decoding, and generating reconstructed frames By minimizing the reconstruction frame and real frame V k The gap between them guides the decoder Learn the image reconstruction process;

[0010] Step 5: Abnormal Behavior Inference: Based on the original frame V k and reconstructed frames Generate total score S for anomalies k When the total abnormal score exceeds the preset threshold τ, it indicates that abnormal behavior has been detected.

[0011] In step 2, during the infrared modal feature extraction stage, the encoder Using a convolutional neural network as its basic structure, visible light features are extracted by stacking convolutional layers, batch normalization layers, and ReLU activation layers. The multi-level memory encoder consists of alternating encoding modules and memory modules, and finally outputs the result through an upsampling layer. Each encoding module consists of two convolutional blocks and a 2×2 max pooling layer. Each convolutional block consists of a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation layer. The memory enhancement module of each layer is a learnable memory matrix. N l C represents the number of memory slots in the l-th layer. l Represents the output dimension of the l-th layer, where each row vector within the matrix... This is called a memory item;

[0012] When extracting infrared features using MMAE, the encoder MMAE extracts infrared modal video frame sequences I. 1:k Extracting its features layer by layer Furthermore, for the features extracted by the l-th encoding module Memory enhancement operations are also required to generate enhanced infrared modal features. The enhancement process is as follows: First, calculate the feature map. With memory matrix The cosine similarity is then transformed into a feature vector for a single spatial location in the feature map. With memory items cosine similarity

[0013]

[0014] in,(·) T This represents the transpose operation, where j represents the index of a memory item in the l-th layer memory enhancement module; next, the learnable threshold λ is calculated. l To filter out low-relevance memory entries, only those with a high degree of matching with the current feature vector are retained; the threshold generation formula is as follows:

[0015]

[0016] Where, θ l Let be learnable parameters initialized to zero, and α be a hyperparameter used to control the threshold adjustment range. Then, sparsification is performed on the similarity by using the ReLU function to set terms with similarity below the threshold to zero, resulting in sparsed similarity.

[0017]

[0018] Finally, the sparse similarity is converted into normalized weights. To avoid division by zero errors, a minimum value ∈=10 needs to be added to the sparse similarity. -8 Then normalize it:

[0019]

[0020] Among them, normalized weights This indicates the proportion of each memory item's contribution to the current feature; the weighted relevant memory items are then compared with the current feature. The original features are reassembled and enhanced through residual connections to obtain the enhanced features.

[0021]

[0022] In the feature fusion stage, an attention-based feature fusion mechanism is adopted: First, the input visible light features are processed... Perform a global average pooling operation, infrared features Maintaining the same size as the input features, we then perform a concatenation operation. Will and When pieced together, they form a merged feature matrix:

[0023]

[0024] Then, the fusion weight matrix M is calculated. fUsed to adjust the weights between visible light and infrared features:

[0025]

[0026] Among them, W a This represents the weight matrix learned in the previous step, processed by the Sigmoid function, resulting in M. f The elements in the formula represent the fusion weights at the corresponding positions and their values ​​are all between [0,1]. Finally, the calculated weights are used to weight the visible light and infrared features:

[0027]

[0028] Here, ⊙ represents element-wise multiplication, and 1 represents a matrix with all elements being 1.

[0029] During network training in steps 3 and 4, directly calculating the KL divergence between the two distributions is transformed into maximizing the variational lower bound ELBO problem, which is expressed as:

[0030]

[0031] Rewriting ELBO as a loss function, which manifests as a separation loss. Compact loss and self-encoding loss Weighted form:

[0032]

[0033] Where J represents the batch size and L represents the number of memory module levels, Let m represent the original infrared features input to the k-th layer memory module in the j-th sample. n and m s These are the memory bank All memory items m l With infrared features The cosine similarity is taken as the memory terms corresponding to the maximum and second largest values, α is the boundary hyperparameter, and λ is the memory term. sep , λ com and λ CVAE To balance the hyperparameters of the losses in each part; the decoder Using a deconvolutional neural network as its basic structure, it receives sampled values ​​z0 of the latent space Z and infrared feature conditions. The input features are progressively upsampled by stacking deconvolution layers, batch normalization layers, and ReLU activation layers, ultimately reconstructing the target frame.

[0034] In step 5, the total abnormal score S k By S pand S m It consists of two parts. First, the original frame V is calculated using the peak signal-to-noise ratio (PSNR). k and reconstructed frames The image index between them is defined as:

[0035]

[0036] Where N is the total number of pixels in the video frame, v i This represents the pixel value of the corresponding pixel point i. The PSNR value is the maximum value of all pixels in this video frame. The anomaly score S is obtained by normalizing the PSNR value. p :

[0037]

[0038] Then, using infrared features With the most recent item m n L2 batch normalized distance as outlier score S m :

[0039]

[0040] The anomaly scores from the two parts are weighted to obtain the total frame anomaly score S for this video frame. k :

[0041] S k =w p S p +w m S m

[0042] Among them, w p and w m As weighting coefficients, iterate through all video frames of the test data and calculate the total frame anomaly score S frame by frame. k Anomaly detection is performed based on a preset threshold τ. When the following occurs:

[0043] S k >τ

[0044] When this happens, it is identified as an abnormal frame, indicating that abnormal behavior has been detected.

[0045] The features and beneficial effects of this invention are:

[0046] (1) This invention proposes a video anomaly detection algorithm based on the fusion of infrared and visible light features, which aims to solve the problem of video anomaly detection in environments with insufficient lighting or low visibility. By constructing an infrared and visible light collaborative reconstruction system, cross-modal feature fusion is achieved, breaking through the limitations of traditional single-modal detection in low-light scenarios.

[0047] (2) A multi-level memory encoder is proposed, which enhances and hierarchically memorizes salient features in infrared images, solving the problem of missed detection caused by the excessive generalization ability of traditional autoencoders and improving the sensitivity of abnormal behavior detection. The multi-level memory module of this encoder enhances the ability to distinguish between normal and abnormal samples, enabling the model to pay more attention when abnormal behavior occurs, thereby improving the accuracy of video anomaly detection and its adaptability in complex scenes. In addition, the enhanced infrared features stored in the multi-level memory module are used as prior conditions of the conditional variational encoder and input into the visible light decoder, guiding the decoder to generate visible light reconstruction results guided by infrared features, improving the detection performance of the model under low light or poor weather conditions. Attached Figure Description

[0048] Figure 1 : A schematic diagram of a video anomaly detection algorithm based on infrared and visible light image feature fusion.

[0049] Figure 2 Schematic diagram of a multi-level memory encoder.

[0050] Figure 3 : Frame reconstruction visualization results. Detailed Implementation

[0051] First, a multi-level encoder architecture with a hierarchical memory mechanism is used to effectively capture multi-scale feature representations in infrared video. Then, a feature fusion module is used to deeply fuse infrared and visible light features. Finally, the target frame is reconstructed based on the fused features and enhanced infrared features, thereby improving the reconstruction quality of visible light images under low light conditions and enhancing the model's anomaly detection capability under low light conditions.

[0052] In the infrared feature extraction stage, a multi-level memory encoder is proposed to enhance and hierarchically memorize salient features in infrared images. In the feature fusion stage, an attention-based feature fusion method is employed to adaptively adjust the contribution of the two modalities in a deep network. In the image reconstruction stage, enhanced infrared modal features are injected as conditional priors into the latent spatial distribution modeling process. The distribution of latent variables is constrained by infrared features and fused features, guiding the decoder to generate visible light reconstruction results guided by infrared features. This invention can significantly improve the image reconstruction effect and video anomaly detection performance of the model in low-light environments.

[0053] The technical solution adopted in this invention is a video anomaly detection method based on infrared and visible light image feature fusion. First, a convolutional neural network encoder is used to extract visible light features from video frames. Simultaneously, a multi-level memory encoder is used to enhance and hierarchically memorize salient features in the infrared image to obtain enhanced infrared features. The multi-level memory encoder is composed of an encoding module and a memory module based on another convolutional neural network, which are alternately connected. Finally, the result is output through the upsampling layer of the other convolutional neural network. Then, the infrared and visible light features are deeply fused to obtain fused features. Finally, the decoder corresponding to the convolutional neural network encoder is guided to reconstruct the target frame based on the fused features and enhanced infrared features. Anomaly behavior inference: an anomaly score is generated based on the original frame and the reconstructed target frame. When the anomaly score exceeds a preset threshold, it indicates that an anomaly behavior has been detected.

[0054] The detailed steps are as follows:

[0055] Step 1: Data Preprocessing: Preprocess the dataset, which contains both normal and abnormal samples; preprocess the input visible light modal video frame sequence. and infrared modal video frame sequences The video frames are normalized to the range of [-1,1] and the resolution is adjusted to 256×256. B represents the batch size, H represents the image height, W represents the image width, and C represents the number of feature channels.

[0056] Step 2, Feature Extraction and Fusion: Using an encoder Extract visible light features from video frames Infrared modal features were extracted using a multi-level memory encoder (MMAE). Calculate and store enhanced infrared modal features Then, the visible light features and infrared features are fused using attention features to generate the fused feature F. l According to the fusion feature F l and enhanced infrared signature Generate the latent space Z;

[0057] Step 3, Network Training: Introduce enhanced infrared modal features stored in the memory module. Establish the empirical distribution of the latent variable z By minimizing the true Bayesian posterior distribution between the two distributions Introducing fusion feature F l and real frame V k We establish the KL divergence after approximate Bayesian analysis of the latent variable z to guide the convolutional neural network visible light feature encoder and convolutional neural network infrared feature encoder to learn the distribution of the latent space Z.

[0058] Step 4, Frame Reconstruction: Decoder z0 is obtained by sampling from the generated latent space Z, and z0 is combined with infrared features splicing, decoding, and generating reconstructed frames By minimizing the reconstruction frame and real frame V k The gap between them guides the decoder Learn the image reconstruction process;

[0059] Step 5: Abnormal Behavior Inference: Based on the original frame V k and reconstructed frames Generate total score S for anomalies k When the total abnormal score exceeds the preset threshold τ, it indicates that abnormal behavior has been detected.

[0060] In step 2, during the infrared modal feature extraction stage, the encoder Using a convolutional neural network as its basic structure, visible light features are extracted by stacking convolutional layers, batch normalization layers, and ReLU activation layers. The multi-level memory encoder consists of alternating encoding modules and memory modules, and finally outputs the result through an upsampling layer. Each encoding module consists of two convolutional blocks and a 2×2 max pooling layer. Each convolutional block consists of a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation layer. The memory enhancement module of each layer is a learnable memory matrix. N l C represents the number of memory slots in the l-th layer. l Represents the output dimension of the l-th layer, where each row vector within the matrix... This is called a memory item;

[0061] When extracting infrared features using MMAE, the encoder MMAE extracts infrared modal video frame sequences I. 1:k Extracting its features layer by layer Furthermore, for the features extracted by the l-th encoding module Memory enhancement operations are also required to generate enhanced infrared modal features. The enhancement process is as follows: First, calculate the feature map. With memory matrix The cosine similarity is then transformed into a feature vector for a single spatial location in the feature map. With memory items cosine similarity

[0062]

[0063] in,(·) TThis represents the transpose operation, where j represents the index of a memory item in the l-th layer memory enhancement module; next, the learnable threshold λ is calculated. l To filter out low-relevance memory entries, only those with a high degree of matching with the current feature vector are retained; the threshold generation formula is as follows:

[0064]

[0065] Where, θ l Let be learnable parameters initialized to zero, and α be a hyperparameter used to control the threshold adjustment range. Then, sparsification is performed on the similarity by using the ReLU function to set terms with similarity below the threshold to zero, resulting in sparsed similarity.

[0066]

[0067] Finally, the sparse similarity is converted into normalized weights. To avoid division by zero errors, a minimum value ∈=10 needs to be added to the sparse similarity. -8 Then normalize it:

[0068]

[0069] Among them, normalized weights This indicates the proportion of each memory item's contribution to the current feature; the weighted relevant memory items are then compared with the current feature. The original features are reassembled and enhanced through residual connections to obtain the enhanced features.

[0070]

[0071] In the feature fusion stage, an attention-based feature fusion mechanism is adopted: First, the input visible light features are processed... Perform a global average pooling operation, infrared features Maintaining the same size as the input features, we then perform a concatenation operation. Will and When pieced together, they form a merged feature matrix:

[0072]

[0073] Then, the fusion weight matrix M is calculated. f Used to adjust the weights between visible light and infrared features:

[0074]

[0075] Among them, Wa This represents the weight matrix learned in the previous step, processed by the Sigmoid function, resulting in M. f The elements in the formula represent the fusion weights at the corresponding positions and their values ​​are all between [0,1]. Finally, the calculated weights are used to weight the visible light and infrared features:

[0076]

[0077] Here, ⊙ represents element-wise multiplication, and 1 represents a matrix with all elements being 1.

[0078] During network training in steps 3 and 4, directly calculating the KL divergence between the two distributions is transformed into maximizing the variational lower bound ELBO problem, which is expressed as:

[0079]

[0080] Rewriting ELBO as a loss function, which manifests as a separation loss. Compact loss and self-encoding loss Weighted form:

[0081]

[0082] Where J represents the batch size and L represents the number of memory module levels, Let m represent the original infrared features input to the k-th layer memory module in the j-th sample. n and m s These are the memory bank All memory items m l With infrared features The cosine similarity is taken as the memory terms corresponding to the maximum and second largest values, α is the boundary hyperparameter, and λ is the memory term. sep , λ com and λ CVAE To balance the hyperparameters of the losses in each part; the decoder Using a deconvolutional neural network as its basic structure, it receives sampled values ​​z0 of the latent space Z and infrared feature conditions. The input features are progressively upsampled by stacking deconvolution layers, batch normalization layers, and ReLU activation layers, ultimately reconstructing the target frame.

[0083] In step 5, the total abnormal score S k By S p and S m It consists of two parts. First, the original frame V is calculated using the peak signal-to-noise ratio (PSNR). k and reconstructed frames The image index between them is defined as:

[0084]

[0085] Where N is the total number of pixels in the video frame, v i This represents the pixel value of the corresponding pixel point i. The PSNR value is the maximum value of all pixels in this video frame. The anomaly score S is obtained by normalizing the PSNR value. p :

[0086]

[0087] Then, using infrared features With the most recent item m n L2 batch normalized distance as outlier score S m :

[0088]

[0089] The anomaly scores from the two parts are weighted to obtain the total frame anomaly score S for this video frame. k :

[0090] S k =w p S p +w m S m

[0091] Among them, w p and w m As weighting coefficients, iterate through all video frames of the test data and calculate the total frame anomaly score S frame by frame. k Anomaly detection is performed based on a preset threshold τ. When the following occurs:

[0092] S k >τ

[0093] When this happens, it is identified as an abnormal frame, indicating that abnormal behavior has been detected.

[0094] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0095] The purpose of this invention is to propose a video anomaly detection method based on infrared and visible light image feature fusion. First, a multi-level encoder architecture with a hierarchical memory mechanism is used to effectively capture multi-scale feature representations in infrared video. Then, a feature fusion step is employed to deeply fuse infrared and visible light features to obtain fused features. Finally, the fused features and enhanced infrared features guide the reconstruction of the target frame. This method can effectively improve the image reconstruction and anomaly detection performance of the model under low-light conditions.

[0096] The specific implementation steps are as follows:

[0097] Step 1, Data Preprocessing: Preprocess the dataset. This involves processing the visible light modal video frame sequences from the input system. and infrared modal video frame sequences The video frames are normalized to the range of [-1,1] and the resolution is adjusted to 256×256. B represents the batch size, H represents the image height, W represents the image width, and C represents the number of feature channels.

[0098] Step 2, Feature Extraction and Fusion: Using an encoder Extract visible light features from video frames Infrared modal features are extracted using a multi-level memory encoder. Calculate and store enhanced infrared modal features Then, the visible light features and infrared features are input into the attention feature fusion module to generate the fused feature F. l ;

[0099] Step 3, Network Training: Introduce enhanced infrared modal features stored in the memory module. Establish the true posterior distribution of the latent variable z Introducing fusion feature F l and real frame V k Establish an approximate posterior distribution of the latent variable z. The encoder network is guided to learn the distribution of the latent space Z by minimizing the KL divergence between two distributions;

[0100] Step 4, Frame Reconstruction: Encoder z0 is obtained by sampling from the generated latent space Z, and z0 is combined with infrared features splicing, decoding, and generating reconstructed frames By minimizing the reconstruction frame and real frame V k The gap between them guides the decoder network to learn the image reconstruction process.

[0101] In step 1, the datasets used include: the public datasets USCDPed2, CUHK Avenue, and ShanghaiTech, and the self-made datasets SADAY and SANight. SADAY was collected under normal daytime lighting conditions and contains 23 training videos and 25 test videos, recording a total of 40 anomalous events such as people carrying firearms, illegal border crossings, and throwing objects. SANight was collected under low-light nighttime conditions and contains 10 training videos and 16 test videos, recording a total of 30 anomalous events such as people loitering, robbery, and throwing objects.

[0102] In step 2, this invention proposes a multi-level memory encoder, overcoming the shortcomings of traditional encoders that often successfully reconstruct anomalous samples even when faced with abnormal data, leading to missed detections of anomalous events. It aims to extract salient features from infrared images and perform layered memory encoding, thereby improving the model's detection accuracy. This multi-level memory encoder is based on a convolutional neural network and consists of alternating and sequential encoding and memory modules, unlike visible light convolutional neural networks without memory modules. The final output is achieved through an upsampling layer. Each encoding module consists of two convolutional blocks and a 2×2 max-pooling layer. Each convolutional block consists of a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation layer. The memory enhancement module in each layer is a learnable memory matrix. N l C represents the number of memory slots in the l-th layer. l Represents the output dimension of the l-th layer, where each row vector within the matrix... These are called memory terms, representing the prototype features of normal data. Memory matrices at different levels store infrared features at different scales. Shallow-level memory matrices store local structural features such as object contours, human posture, and texture information, while deep-level memory matrices store overall structural features such as behavior patterns and spatial locations, effectively compensating for the information loss in visible light images under low-light conditions.

[0103] When extracting infrared features in MMAE, the encoding module extracts infrared modal video frame sequences I. 1:k Extracting its features layer by layer Furthermore, for the features extracted by the l-th encoding module Memory enhancement operations are also required to generate enhanced infrared modal features. The process is as follows: First, calculate the feature map. With memory matrix The cosine similarity is then transformed into a feature vector for a single spatial location in the feature map. With memory items cosine similarity

[0104]

[0105] in,(·) T This represents the matrix transpose operation, and j represents the index of a memory item in the l-th layer of the memory enhancement module. Similarity This reflects the degree of matching between the feature vector and each memory term. Next, a learnable threshold λ is calculated. l This filters out low-relevance memory entries, retaining only those with a high degree of matching to the current feature vector. The threshold generation formula is as follows:

[0106]

[0107] Where, θ l The parameters are learnable and initialized to zero, and α is a hyperparameter used to control the threshold adjustment range. Then, sparsification is performed on the similarity using the ReLU function to set terms with similarity below the threshold to zero, resulting in sparsed similarity.

[0108]

[0109] The ReLU function sets similarity below a threshold to zero, retaining positively correlated terms and effectively removing memory terms irrelevant to the current feature, keeping only positively correlated terms. Finally, the sparse similarity is converted into normalized weights. This is used to quantify the contribution of each memory item to the current feature. To avoid division by zero errors, a minimum value ∈=10 needs to be added to the sparse similarity. -8 Then normalize it:

[0110]

[0111] Among them, normalized weights This represents the proportion of each memory item's contribution to the current feature. If the similarity between a feature and all memory items is below a threshold, the weights degenerate into a uniform distribution. The memory module will weight the relevant memory items and the current features. The original features are reassembled and enhanced through residual connections to obtain the enhanced features.

[0112]

[0113] The memory module is a hierarchical memory enhancement architecture for anomaly detection in infrared video. By dynamically capturing the characteristic patterns of infrared modalities, it overcomes the generalization bottleneck of traditional autoencoders. Simultaneously, it achieves alignment of infrared and visible light at the feature level, effectively adapting to the spatiotemporal complexity of video.

[0114] In the feature fusion stage, an attention-based feature fusion module was employed to enhance the fusion capability of multimodal features. First, the input visible light features... Perform a global average pooling operation to convert the feature map into a single value for each channel, for infrared features. Maintain the same size as the input features. Next, perform a concatenation operation. Will and When pieced together, they form a merged feature matrix:

[0115]

[0116] Then, the fusion weight matrix M is calculated. f Used to adjust the weights between visible light and infrared features:

[0117]

[0118] Among them, W a This represents the weight matrix learned in the previous step. After processing with the Sigmoid function, the resulting M... f The elements in the expression represent the fusion weights at corresponding positions and their values ​​range from [0,1]. Finally, the calculated weights are used to weight the visible light and infrared features:

[0119]

[0120] Here, ⊙ represents element-wise multiplication, and 1 represents a matrix with all elements equal to 1. This module effectively fuses the image features of the visible light and infrared modes by dynamically adjusting the weights between them, enhancing the model's ability to model complex dependencies. In anomaly detection tasks, especially when image behavior is unclear in the visible light mode, it can improve the model's performance.

[0121] In the network training process of steps 3 and 4, directly calculating the KL divergence between two distributions is difficult, and directly using pixel value differences to guide network training is inconvenient. The problem is modeled as follows:

[0122]

[0123] This indicates that based on visible light video frame V 1:k and infrared video frame I 1:k Recover the reconstructed frame This problem has a variational lower bound (ELBO), which can be expressed as:

[0124]

[0125] In the above equation, the first term of RHS represents the constraint on the encoder to make the approximate posterior distribution of the latent space Z as close as possible to its prior distribution. The second term represents the constraint on the decoder to make the image recovered by sampling from the latent space Z as close as possible to the real image. This transforms ELBO into a loss function. The training objective of MMAE is to encourage differentiation between memory terms while ensuring that each feature vector can be accurately represented by the memory module. The former uses separation loss. This indicates that the aim is to encourage maximizing the distance between the nearest and second-nearest memory items, thereby motivating the model to remember more differentiated information. The latter uses compaction loss. This is intended to ensure that each feature vector is tightly represented by at least one memory term. Separation loss. It can be represented as:

[0126]

[0127] Where J represents the number of test batches, and L represents the number of memory module levels. Let m represent the original infrared features input to the k-th layer memory module in the j-th sample. n and m s These are the memory bank All memory items m l With infrared features The cosine similarity is calculated by taking the terms corresponding to the maximum and second-largest values, where α is the boundary hyperparameter. Compact loss. It can be represented as:

[0128]

[0129] The autoencoder employs two loss functions for constraint: the KL divergence loss constrains the distribution of the latent space, ensuring it aligns with the conditional distribution of infrared features, thus constraining the encoder; the reconstruction error loss measures the difference between the reconstructed frame and the real frame, constraining the decoder. By jointly optimizing these two losses, the conditional variational encoder can generate visible light reconstruction results that better match the true distribution. It can be represented as:

[0130]

[0131] Model comprehensive training loss It can be represented as:

[0132]

[0133] Where, λ sep , λ com and λ CVAETo balance the hyperparameters of the various loss components, the Adam optimizer is used for parameter updates during training, along with a learning rate decay strategy to improve model convergence. The overall training loss is calculated as follows: By comprehensively considering the constraints and learning capabilities of each part of the autoencoder, a model that can maintain anomaly detection performance under low light or poor weather conditions can be trained by minimizing the loss.

[0134] An example of a complete training and testing process of this invention is as follows:

[0135] a. Network parameter initialization: The number of training epochs is set to 200, the number of training batches B is set to 64, and the initial learning rate of the MMAE module is set to 2×10⁻⁶. -5 The learning rate of the conditional variational encoder is set to 2×10. -4 The Adam optimizer momentum parameters are set to (0.9, 0.999). For the UCSDPed2, CHUK Avenue, ShanghaiTech, SADay, and SANight datasets, the memory terms are set to {32, 32, 200, 200, 200}, and the fusion weights w p and w m The values ​​are set to (0.9, 0.1), (0.9, 0.1), (0.8, 0.2), (0.8, 0.2), and (0.8, 0.2) respectively. The threshold τ is set to {0.7, 0.7, 0.6, 0.6, 0.6} respectively. The hyperparameter λ... sep , λ com , λ CVAE Set them to {0.9,1,1,1}, {0.1,0.6,0.3,0.3}, {0.1,0.06,0.07,0.05}, {0.1,0.06,0.07,0.05}, and {0.1,0.06,0.07,0.05} respectively.

[0136] b. Data Loading: Select a dataset, traverse the folders under its specified path and read the files, and preprocess the video frames: ① For datasets lacking infrared data, use optical flow features extracted from visible light data as substitute information for infrared modes; ② Resize each frame to 256×256; ③ Normalize the pixel values ​​in all frames to [-1,1]; ④ For each video, take the first k-1 frames as training frames and the kth frame as the ground truth frame; ⑤ Randomly divide the dataset into training set, validation set and test set according to the scaling parameter.

[0137] c. Feature extraction and fusion: using an encoder Extract visible light features from each video frame in the training and validation sets. The infrared modal features of each video frame are extracted and stored using MMAE. Then, the visible light and infrared features are input into the AFF to generate the fused feature F. l ;

[0138] d. Network training: Introducing enhanced infrared modal features stored in a memory module. Establish the true posterior distribution of the latent variable z Introducing fusion feature F l and real frame V k Establish an approximate posterior distribution of the latent variable z. The encoder network is guided to learn the distribution of the latent space Z by minimizing the KL divergence between two distributions;

[0139] e. Frame reconstruction: z0 is obtained by sampling from the latent space Z generated by the encoder network, and z0 is combined with infrared features. splicing, input decoder In the process, reconstructed frames are generated. By minimizing the reconstruction frame and real frame V k The gap between them guides the decoder network to learn the image reconstruction process;

[0140] f. Model tuning and saving: Minimize the overall training loss of the model. The model is trained under constraints. After each training round, the model is validated using validation set data, and the model with the smallest overall validation loss is selected. The model is saved.

[0141] g. Model Testing: After feature extraction and fusion of the test set data, the encoder network generates a latent space Z based on the fused features. The decoder network samples z0 from the latent space Z, and then combines z0 with the infrared features... splicing, decoding, and generating reconstructed frames

[0142] h. Abnormal behavior reasoning: Based on the original frame V k and reconstructed frames Generate total score S for anomalies k Abnormal frames are determined based on a threshold τ. When S occurs... k When the value is greater than τ, it is determined that abnormal behavior has been detected.

[0143] i. Visualize the results.

[0144] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A video anomaly detection method based on infrared and visible light image feature fusion, characterized in that, First, a convolutional neural network encoder is used to extract visible light features from the video frame. Simultaneously, a multi-level memory encoder enhances and hierarchically memorizes salient features in the infrared image to obtain enhanced infrared features. The multi-level memory encoder consists of an alternating connection of an encoding module and a memory module based on another convolutional neural network. The final result is output through the upsampling layer of the other convolutional neural network. Then, the infrared and visible light features are deeply fused to obtain fused features. Finally, the decoder corresponding to the convolutional neural network encoder is guided to reconstruct the target frame based on the fused features and enhanced infrared features. Abnormal behavior inference: An abnormal total score is generated based on the original frame and the reconstructed target frame. When the abnormal total score exceeds a preset threshold, it indicates that abnormal behavior has been detected.

2. The video anomaly detection method based on infrared and visible light image feature fusion as described in claim 1, characterized in that, The detailed steps are as follows: Step 1: Data Preprocessing: Preprocess the dataset, which contains both normal and abnormal samples; preprocess the input visible light modal video frame sequence. and infrared modal video frame sequences The video frames are normalized to the range of [-1,1] and the resolution is adjusted to 256×256. B represents the batch size, H represents the image height, W represents the image width, and C represents the number of feature channels. Step 2, Feature Extraction and Fusion: Using an encoder Extract visible light features from video frames Infrared modal features were extracted using a multi-level memory encoder (MMAE). Calculate and store enhanced infrared modal features Then, the visible light features and infrared features are fused using attention features to generate the fused feature F. l According to the fusion feature F l and enhanced infrared signature Generate the latent space Z; Step 3, Network Training: Introduce enhanced infrared modal features stored in the memory module. Establish the empirical distribution of the latent variable z By minimizing the true Bayesian posterior distribution between the two distributions Introducing fusion feature F l and real frame V k We establish the KL divergence after approximate Bayesian analysis of the latent variable z to guide the convolutional neural network visible light feature encoder and convolutional neural network infrared feature encoder to learn the distribution of the latent space Z. Step 4, Frame Reconstruction: Decoder z0 is obtained by sampling from the generated latent space Z, and z0 is combined with infrared features splicing, decoding, and generating reconstructed frames By minimizing the reconstruction frame and real frame V k The gap between them guides the decoder Learn the image reconstruction process; Step 5: Abnormal Behavior Inference: Based on the original frame V k and reconstructed frames Generate total score S for anomalies k When the total abnormal score exceeds the preset threshold τ, it indicates that abnormal behavior has been detected.

3. The video anomaly detection method based on infrared and visible light image feature fusion as described in claim 2, characterized in that, In step 2, during the infrared modal feature extraction stage, the encoder Using a convolutional neural network as its basic structure, visible light features are extracted by stacking convolutional layers, batch normalization layers, and ReLU activation layers. The multi-level memory encoder consists of alternating encoding modules and memory modules, and finally outputs the result through an upsampling layer. Each encoding module consists of two convolutional blocks and a 2×2 max pooling layer. Each convolutional block consists of a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation layer. The memory enhancement module of each layer is a learnable memory matrix. N l C represents the number of memory slots in the l-th layer. l Represents the output dimension of the l-th layer, where each row vector within the matrix... This is called a memory item; When extracting infrared features using MMAE, the encoder MMAE extracts infrared modal video frame sequences I. 1:k Extracting its features layer by layer Furthermore, for the features extracted by the l-th encoding module Memory enhancement operations are also required to generate enhanced infrared modal features. The enhancement process is as follows: First, calculate the feature map. With memory matrix The cosine similarity is then transformed into a feature vector for a single spatial location in the feature map. With memory items cosine similarity in,(·) T This represents the transpose operation, where j represents the index of a memory item in the l-th layer memory enhancement module; next, the learnable threshold λ is calculated. l This is used to filter out low-relevance memory entries and retain only those that have a high degree of matching with the current feature vector. The threshold generation formula is as follows: Where, θ l Let be learnable parameters initialized to zero, and α be a hyperparameter used to control the threshold adjustment range. Then, sparsification is performed on the similarity by using the ReLU function to set terms with similarity below the threshold to zero, resulting in sparsed similarity. Finally, the sparse similarity is converted into normalized weights. To avoid division by zero errors, a minimum value ∈=10 needs to be added to the sparse similarity. -8 Then normalize it: Among them, normalized weights This indicates the proportion of each memory item's contribution to the current feature; the weighted relevant memory items are then compared with the current feature. The original features are reassembled and enhanced through residual connections to obtain the enhanced features. In the feature fusion stage, an attention-based feature fusion mechanism is adopted: First, the input visible light features are processed... Perform a global average pooling operation, infrared features Maintaining the same size as the input features, we then perform a concatenation operation. Will and When pieced together, they form a merged feature matrix: Then, the fusion weight matrix M is calculated. f Used to adjust the weights between visible light and infrared features: Among them, W a This represents the weight matrix learned in the previous step, processed by the Sigmoid function, resulting in M. f The elements in the formula represent the fusion weights at the corresponding positions and their values ​​are all between [0,1]. Finally, the calculated weights are used to weight the visible light and infrared features: Here, ⊙ represents element-wise multiplication, and 1 represents a matrix with all elements being 1.

4. The video anomaly detection method based on infrared and visible light image feature fusion as described in claim 2, characterized in that, During network training in steps 3 and 4, directly calculating the KL divergence between the two distributions is transformed into maximizing the variational lower bound ELBO problem, which is expressed as: Rewriting ELBO as a loss function, which manifests as a separation loss. Compact loss and self-encoding loss Weighted form: Where J represents the batch size and L represents the number of memory module levels, Let m represent the original infrared features input to the k-th layer memory module in the j-th sample. n and m s These are the memory bank All memory items m l With infrared features The cosine similarity is taken as the memory terms corresponding to the maximum and second largest values, α is the boundary hyperparameter, and λ is the memory term. sep , λ com and λ CVAE To balance the hyperparameters of the losses in each part; the decoder Using a deconvolutional neural network as its basic structure, it receives sampled values ​​z0 of the latent space Z and infrared feature conditions. The input features are progressively upsampled by stacking deconvolution layers, batch normalization layers, and ReLU activation layers, ultimately reconstructing the target frame.

5. The video anomaly detection method based on infrared and visible light image feature fusion as described in claim 2, characterized in that, In step 5, the total abnormal score S k By S p and S m It consists of two parts. First, the original frame V is calculated using the peak signal-to-noise ratio (PSNR). k and reconstructed frames The image index between them is defined as: Where N is the total number of pixels in the video frame, v i This represents the pixel value of the corresponding pixel point i. The PSNR value is the maximum value of all pixels in this video frame. The anomaly score S is obtained by normalizing the PSNR value. p : Then, using infrared features With the most recent item m n L2 batch normalized distance as outlier score S m : The anomaly scores from the two parts are weighted to obtain the total frame anomaly score S for this video frame. k : S k =w p S p +w m S m Among them, w p and w m As weighting coefficients, iterate through all video frames of the test data and calculate the total frame anomaly score S frame by frame. k Anomaly detection is performed based on a preset threshold τ. When the following occurs: S k >t When this happens, it is identified as an abnormal frame, indicating that abnormal behavior has been detected.

Images