A target detection method for surveillance video based on multi-frame high-frequency difference enhancement

Abstract

This invention discloses a target detection method for surveillance videos based on multi-frame high-frequency difference enhancement, comprising the following steps: Step 1: First, a series of surveillance videos are selected, and these videos are segmented and frames are extracted. Frame extraction is performed by sampling one frame every ten frames as a training set. Then, data preprocessing and enhancement operations are performed on the input continuous image frames. Step 2: A multi-frame image reconstruction module is constructed to reconstruct images based on the low-level feature maps of the current frame and the reference frame. The parameters of the reconstruction module are optimized by calculating the mean square error (MSE) loss between the original image and the reconstructed image. This target detection method for surveillance videos based on multi-frame high-frequency difference enhancement cleverly combines continuous frame high-frequency difference maps and an adaptive fusion enhancement feature strategy. By extracting and fusing high-frequency motion features from the reconstruction of continuous frames, a powerful target detection framework is constructed, which can significantly improve the target detection performance in surveillance videos.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, specifically to a method for target detection in surveillance videos based on multi-frame high-frequency difference enhancement. Background Technology

[0002] With the rapid development of computer vision and artificial intelligence technologies, object detection, as a fundamental and key technology, has been widely used in many fields such as autonomous driving, video surveillance, and medical image analysis. The rapid popularization of deep convolutional neural networks in the field of static image object detection has shown great superiority in performance compared with traditional methods, and has gradually played a corresponding role in object detection tasks based on surveillance videos.

[0003] The core task of target detection in surveillance videos is to accurately locate and identify targets in each frame of the video. This is particularly important for practical applications such as intelligent security monitoring, traffic control, and behavior analysis. Traditional video target detection methods, such as inter-frame difference and background difference methods, mainly detect moving targets by calculating the pixel differences between consecutive frames or the differences between the background and the current frame. While these methods are effective in simple scenes, their detection performance is unstable when dealing with complex backgrounds, lighting changes, and target occlusion. Furthermore, their capture of moving targets is relatively coarse, making it difficult to meet the requirements for high precision.

[0004] Deep learning-based object detection algorithms, such as Faster R-CNN, SSD, and YOLO series, have demonstrated excellent performance in the field of static image object detection and are gradually being applied in video detection tasks. However, object detection in surveillance videos is more complex than static image detection. Targets in surveillance videos are affected by motion blur, occlusion, uneven lighting, and changes in appearance. Existing object detection methods often lack robustness when dealing with complex scenes with significant lighting changes and rapid dynamic changes in targets, making it difficult to meet the requirements of high accuracy and real-time performance.

[0005] Most current mainstream target detection technologies for surveillance videos rely on keyframe detection and optical flow information to improve detection accuracy and speed by analyzing the correlation between video frames. For example, Depth Forwarding (DFF) extracts features from only keyframes using DeeSFeatureFlow and passes these features to adjacent frames via optical flow, reducing redundant computation but also leading to a decrease in detection accuracy. Forward-Growth Automated Approach (FGFA) enhances the utilization of temporal information through multi-frame feature aggregation, but its high computational cost affects the model's real-time performance. The Thickness-Based Optical Process (THP) method improves detection accuracy and speed through sparse recursive feature aggregation and optical flow estimation, but still relies on keyframe selection. In complex scenes with large lighting variations or target occlusion, keyframe strategies are prone to failure, and the computational complexity of optical flow estimation affects real-time performance. Furthermore, these methods are still insufficiently adaptable to target motion blur, occlusion, and uneven lighting in surveillance videos, making it difficult to achieve stable detection results in complex surveillance environments. To address the limitations of existing technologies in surveillance video scenarios, a detection method that can effectively utilize multi-frame temporal information while possessing high accuracy and high real-time performance is urgently needed. Summary of the Invention

[0006] The purpose of this invention is to provide a target detection method for surveillance videos based on multi-frame high-frequency difference enhancement, in order to solve the problems that, although the current deep learning-based target detection algorithms mentioned in the background have made some progress, they still face problems such as an imbalance between computational efficiency and detection accuracy, and insufficient generalization ability for detecting weak targets when facing surveillance target detection tasks in complex dynamic scenes.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a target detection method for surveillance video based on multi-frame high-frequency difference enhancement, comprising the following steps, characterized in that:

[0008] Step 1: First, select a series of surveillance videos and segment and extract frames from these videos. The frame extraction method is to sample one frame every ten frames as the training set. Then, perform data preprocessing and enhancement operations on the input continuous image frames.

[0009] Step 2: Construct a multi-frame image reconstruction module to reconstruct images based on the low-level feature maps of the current frame and the reference frame. Optimize the parameters of the reconstruction module by calculating the mean squared error (MSE) loss between the original image and the reconstructed image.

[0010] Step 3: Construct a high-frequency difference extraction module to extract high-frequency information from the reconstructed image and the original image, and subtract the high-frequency information of the corresponding frames and perform a modulo operation respectively;

[0011] Step 4: Construct a high-frequency difference fusion enhancement module. By constructing a convolutional network, high-frequency difference information of continuous frames is fused, and global background information and local saliency information of continuous frames are integrated to model and enhance the low-level features of the current frame and feed them into the multi-scale fusion module and the detection head.

[0012] Step 5: Input the surveillance video images from the test set into the trained model to obtain the classification confidence score and prediction box information of the detected targets.

[0013] Preferably, step 1 mainly includes the following steps:

[0014] (1) Perform interval sampling on the video frames, and extract training frames from the video starting from the first frame of the video at the set interval;

[0015] (2) The specific methods of image preprocessing and enhancement are to perform random expansion, cropping, color distortion, flipping, resizing and other operations on the input image.

[0016] Preferably, the specific method of the multi-frame image reconstruction module in step 2 is as follows:

[0017] (1) Input the data-enhanced continuous video frames into the backbone network, which is based on ResNet50, and obtain the output features of the last three stages in the network;

[0018] (2) Perform three consecutive upsampling operations on the low-level features, halving the number of channels each time. Finally, generate a three-channel reconstructed image through convolution, calculate the difference between the reconstructed image and the original input image, and optimize the reconstruction effect by using MSE loss constraints.

[0019] Preferably, the specific method for constructing the high-frequency difference extraction module in step 3 is as follows:

[0020] (1) Process the high-frequency components of the current frame, reference frame, current frame reconstruction image and reference frame reconstruction image one channel at a time. By applying two-dimensional Fourier transform to the image of each channel, the image is transformed from the spatial domain to the frequency domain, and then the zero-frequency component is shifted.

[0021] (2) Based on the center position of the image, create a high-pass filter mask and apply the high-pass filter mask to the image in the frequency domain to preserve the high-frequency components in the image;

[0022] (3) Calculate the difference between the high-frequency components of the input image and the reconstructed image to obtain the difference map in the frequency domain. Then perform inverse Fourier transform and modulus operation on it to obtain the actual high-frequency difference map.

[0023] Preferably, in step 4, a high-frequency difference fusion enhancement module is constructed to better fuse the features of the current frame and the reference frame:

[0024] (1) This module stitches the high-frequency difference maps of the current frame and the previous two frames together and integrates them into a single-channel high-frequency difference map through convolution operation. This fusion step provides more complete spatial details, such as edge and texture information, than a single frame by stitching together the high-frequency difference maps of multiple frames. In this way, the network can capture important features in the image more fully, improve the understanding and processing ability of complex scenes, and enhance the overall spatial feature perception. By processing the stitched high-frequency difference map through convolution operation, the network can adaptively adjust the degree of attention to the changes between different frames. This adaptive ability enables the model to dynamically adjust the feature fusion method according to the specific scene, ensuring good performance in complex temporal change scenarios.

[0025] (2) In order to further enhance the effect of temporal feature fusion and improve feature diversity, feature information is fused using a similar method for the feature maps of the current frame and the reference frame. Specifically, we perform global pooling on the fused feature map and further extract high-level features through linear and activation layers. At the same time, we apply average pooling and extract features through linear and activation layers. After the features are extracted by pooling, a weighted operation is performed and multiplied pixel by pixel with the fused high-frequency difference map to obtain a pixel-level enhancement matrix.

[0026] Furthermore, during model training, the loss function is:

[0027] in, , and Weighting parameters to balance different losses;

[0028] L mse The reconstruction loss is the mean square error (MSE) of the original image, L box To predict the L1 loss of the bounding box, L cls This is a classification loss function that incorporates IoU.

[0029] Compared with existing technologies, the beneficial effects of this invention are as follows: This surveillance video target detection method based on multi-frame high-frequency difference enhancement cleverly combines continuous frame high-frequency difference maps and adaptive fusion enhancement feature strategies. By reconstructing and fusing high-frequency motion features from continuous frames, it constructs a powerful target detection framework that can significantly improve target detection performance in surveillance videos. Its specific structure is as follows:

[0030] (1) The multi-frame image reconstruction module restores the shallow feature maps of consecutive video frames to reconstructed images of the same size as the original images through reconstruction learning. This module is based on multi-level upsampling operations, which enables the reconstructed images of different frames and the original images to be compared and fused at the same scale, laying the foundation for subsequent high-frequency information difference extraction.

[0031] (2) The core of the high-frequency difference extraction module lies in analyzing the high-frequency components of the original image and the reconstructed image. First, Fast Fourier Transform (FFT) is applied to the original image and the reconstructed image respectively. Then, the high-frequency components of both are extracted through a high-pass filter to generate a high-frequency difference map. This high-frequency difference map captures the high-frequency information features related to target motion, edge details, artifacts, etc., that cannot be accurately recovered during the reconstruction process. By utilizing high-frequency differences, the model can more sensitively perceive the motion changes of weak targets.

[0032] (3) The fusion enhancement module integrates high-frequency difference information of continuous frames and utilizes global background information and local salient features to enable the model to effectively capture changes in the time dimension. This helps to model the temporal correlation in the video image sequence, thereby capturing key changes in dynamic scenes more completely, improving the detection capability of weak targets and the generalization capability of complex scenes. The structure is simple and effective and easy to deploy. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating the overall framework of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. Please refer to... Figure 1 This invention provides a technical solution: a target detection method for surveillance video based on multi-frame high-frequency difference enhancement, comprising three steps: a multi-frame image reconstruction module, a high-frequency difference extraction module, and an adaptive fusion enhancement module. In this invention, X∈R 3×H×W This represents an RGB image, where H and W are the height and width of the input image, respectively, and the output is... Includes categories and bounding box Simultaneously, a reconstruction loss is generated to optimize the image reconstruction module.

[0035] Step 1: Select a series of surveillance videos and segment and extract frames from these videos. The frame extraction method is to sample one frame every ten frames as the training set. Then, preprocess and data augment the input continuous image frames. The specific method is to perform random expansion, cropping, color distortion, flipping, resizing and other operations on the input images. The images after these data processing operations will be directly used as the input of the backbone network.

[0036] Step 2: Shallow feature reconstruction of consecutive image frames. The multi-frame image reconstruction module is mainly based on the shallow feature map S3 in the last three stage feature maps S3, S4, and S5 output by the backbone network. This module consists of three upsampling layers, one convolutional layer, and one activation function. The upsampling layer comprises a transposed convolution with a 4x4 kernel, a stride of 2, and padding of 1; two 3x3 convolutions with padding of 1; and a ReLU activation function. Through this process, the number of channels in the feature map is gradually halved, while the spatial resolution is doubled, ultimately resulting in the reconstructed image.

[0037] This module optimizes the parameters of the reconstruction process by minimizing the reconstruction loss (MSE), so that the reconstructed image is as close as possible to the original input image.

[0038] Step 3: High-frequency difference extraction module. The purpose of this module is to extract the high-frequency component differences between the expected reconstruction images of adjacent frames through Fourier transform, thereby capturing important changes such as rapidly changing target boundaries and detail edges in surveillance video images.

[0039] Step 3.1 Fourier transform and frequency domain centering processing, denoted as image representations of frame t and frame t-1. , The corresponding reconstruction diagram is as follows: , Perform a two-dimensional Fast Fourier Transform (FFT) on each channel of the original and reconstructed images in frame t to transform the images from the spatial domain to the frequency domain, yielding the following results: and For each channel of the t-1 frame image, perform the same operation to obtain... and Next, the frequency domain data is centered by using the `fftshift` function to move the low-frequency components to the center of the spectrum.

[0040] ,

[0041] ,

[0042] Perform the same operation on each channel of frame t-1 to obtain and .

[0043] Step 3.2 Construct the high-pass filter mask. Construct the mask using the following formula:

[0044]

[0045] Here, i and j are the row and column indices of the spectrum, respectively. The radius r controls the size of the high-frequency portion to be retained; in the specific experiment, it was set to 0.1. A high-pass filter mask is applied to the frequency domain image to suppress low-frequency components while retaining the high-frequency portion.

[0046]

[0047] The same process is applied to each channel of frame t-1 to obtain... and Then, the difference is calculated in the frequency domain:

[0048]

[0049] To convert the high-frequency difference from the frequency domain to the spatial domain, we first apply inverse centering to the difference spectrum to obtain... Then, a two-dimensional inverse Fourier transform is performed to obtain the high-frequency difference map:

[0050] Then, the mean is calculated along the channel dimension. Finally, since the Fourier transform may produce complex values, the modulus of the high-frequency difference map is taken to obtain the actual single-channel high-frequency difference map. :

[0051]

[0052] in:

[0053] This represents the final high-frequency difference plot. , This represents the high-frequency difference value of the c-th channel. C is the number of channels, with a value of 3.

[0054] Step 4: High-Frequency Difference Fusion Enhancement Module. This module adaptively fuses high-frequency difference maps from multiple frames and combines them with the feature map of the current frame, enhancing the model's understanding of the global scene and its ability to perceive small targets within the scene.

[0055] This module receives four inputs: the S3-level feature map of frame t. S3 level feature map of frame t-1 S3 level feature map of frame t-2 And the high-frequency difference diagram of the three frames. To fuse multi-frame difference maps together and more comprehensively preserve inter-frame variation information, a convolutional network is constructed to fuse multi-frame difference maps and retain key variation regions. First, the high-frequency difference maps of each frame are stitched together along the channels to obtain... Then A single-channel fused high-frequency difference map is obtained by applying a 3x3 convolution kernel and a ReLU activation function. .

[0056] in, .

[0057] Again Bilinear interpolation is performed to obtain , .

[0058] because To enhance the diversity of features in the temporal dimension by integrating spatial variation information from multiple frames, we concatenate and fuse the shallow feature maps S3 from multiple frames and employ max pooling to obtain local saliency information. Average pooling is used to capture global stability information. Then generate time series weights. , and apply T to The above generates a pixel-by-pixel high-frequency difference enhancement matrix H. The weight vector T is beneficial for further establishing temporal correlations between multiple frames, enhancing the overall motion understanding of the target, and improving the detection capability of targets with occlusion and motion blur.

[0059] Ultimately, the pixel-wise attention matrix H is used to... Enhancement will be achieved by using H and Pixel-by-pixel multiplication yields the enhanced feature map. ,Right now .

[0060] Step 5: Model convergence and testing process: Supervise model training until convergence using the total loss function L.

[0061] The total loss function L of this invention consists of three parts: the mean square error (MSE) of the original image, the reconstruction loss L... mse L1 loss for predicting bounding boxes and classification loss L1 with IoU introduced. cls Reconstruction loss is commonly used to evaluate the performance of generative models in data reconstruction tasks; it measures the difference between the model's reconstructed output and the original data. In image processing applications, reconstruction loss is typically calculated using MSE (Mean Separation of Elements). The formula for reconstruction loss based on MSE is as follows: Where N represents the batch_size.

[0062] L1 loss is often used to calculate the deviation between the predicted bounding box and the target bounding box, directly measuring the difference between the two in coordinate space.

[0063]

[0064] Where N is the number of bounding boxes. Represents the actual information in the box. This indicates the predicted bounding box information.

[0065] The classification loss function of this invention combines IoU weights and the FocalLoss concept to optimize the classification results of bounding boxes. First, it generates weights for loss calculation by calculating the IoU value between the predicted and ground truth boxes. Then, it combines the predicted classification score with Sigmoid activation and optimizes using a weighted binary cross-entropy loss. The formula for its loss function is as follows:

[0066]

[0067] in

[0068] It is the model output for the i-th sample, representing the model's raw scores for different categories. It is the true class label of the i-th sample. The category labels are weighted using IoU values ​​to measure the degree of match of the target bounding box. It combines FocalLoss and IoU to dynamically adjust the loss weights for samples. The Lcls loss function of this invention combines the predicted score and the target score. The predicted score is determined by the model's classification... The target score is converted into a probability value by the Sigmoid activation function, representing the model's predicted probability for each class. The target score is then used to generate a weighted supervision signal to guide model learning by combining the IoU value with the One-Hot encoding of the true class label. The final loss function is based on a weighted binary cross-entropy loss of the predicted and target scores, and is dynamically adjusted for samples of varying difficulty by combining IoU and FocalLoss weights.

[0069] The development language is Sython, the development environment is based on the Ubuntu system, and the deep learning framework used is Sytorch. The dataset is the UA-DETRAC vehicle detection dataset. This dataset contains 10 hours of video footage shot at 24 different locations in Beijing and Tianjin, China, using a Canon EOS 550D camera. The video was recorded at 25 frames per second (fSs) with a resolution of 960×540 pixels. The UA-DETRAC dataset contains over 140,000 frames and 8,250 manually labeled vehicles, totaling 1.21 million labeled object bounding boxes. The training set contains approximately 82,085 images, and the test set contains approximately 56,167 images.

[0070] Model training

[0071] Using the Sytorch framework, steps S1-S4 are performed according to... Figure 1The model was trained for 60 eSochs with a batch size of 1, using the AdamW optimizer and a base learning rate of 1e. -4 The weight decays to 1e -4 The model uses average precision (AS) as the evaluation metric, recording the AS metric on the validation set and the parameters of the current model for each ESoch.

[0072] Model testing

[0073] Using the model with the highest average precision (AS) during training as the test model, we evaluated the performance of our proposed method on the UA-DETRAC dataset across multiple scenes. The mean accuracy (AS) score was calculated following the method described in the paper "The SASCAL Visual Object Classes Challenge: ARetros Secret," and tested on the DETRAC test set. A confidence threshold of 0.7 was set for each image prediction; setting a higher confidence threshold reduces low-confidence predictions, thereby improving the reliability of the prediction results.

[0074] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A target detection method for surveillance video based on multi-frame high-frequency difference enhancement, comprising the following steps, characterized in that: Step 1: First, select a series of surveillance videos and segment and extract frames from these videos. The frame extraction method is to sample one frame every ten frames as the training set. Then, perform data preprocessing and enhancement operations on the input continuous image frames. Step 2: Construct a multi-frame image reconstruction module to reconstruct images based on the low-level feature maps of the current frame and the reference frame. Optimize the parameters of the reconstruction module by calculating the mean squared error (MSE) loss between the original image and the reconstructed image. The specific method of the multi-frame image reconstruction module in step 2 is as follows: (1) Input the data-enhanced continuous video frames into the backbone network, which is based on ResNet50, and obtain the output features of the last three stages in the network; (2) Perform three consecutive upsampling operations on the low-level features, halving the number of channels each time. Finally, generate a three-channel reconstructed image through convolution, calculate the difference between the reconstructed image and the original input image, and optimize the reconstruction effect by using MSE loss constraints. Step 3: Construct a high-frequency difference extraction module to extract high-frequency information from the reconstructed image and the original image, and subtract the high-frequency information of the corresponding frames and perform a modulo operation respectively; The specific method for constructing the high-frequency difference extraction module in step 3 is as follows: (1) Process the high-frequency components of the current frame, reference frame, current frame reconstruction image and reference frame reconstruction image one channel at a time. By applying two-dimensional Fourier transform to the image of each channel, the image is transformed from the spatial domain to the frequency domain, and then the zero-frequency component is shifted. (2) Based on the center position of the image, create a high-pass filter mask and apply the high-pass filter mask to the image in the frequency domain to preserve the high-frequency components in the image; (3) Calculate the difference between the high-frequency components of the input image and the reconstructed image to obtain the difference map in the frequency domain, and then perform inverse Fourier transform and modulus operation on it to obtain the actual high-frequency difference map. Step 4: Construct a high-frequency difference fusion enhancement module. By constructing a convolutional network, high-frequency difference information of continuous frames is fused, and global background information and local saliency information of continuous frames are integrated to model and enhance the low-level features of the current frame and feed them into the multi-scale fusion module and the detection head. Step 4 involves constructing features for the high-frequency difference fusion enhancement module, including: (1) This module stitches the high-frequency difference maps of the current frame and the previous two frames together and integrates them into a single-channel high-frequency difference map through convolution operation. This fusion step provides edge and texture information by stitching together the high-frequency difference maps of multiple frames. (2) The feature maps of the current frame and the reference frame are obtained by the fusion step. The fused feature map is subjected to global pooling operation and high-level features are extracted through linear layer and activation layer. At the same time, average pooling operation is applied and feature extraction is performed through linear layer and activation layer. After the features are extracted by pooling, a weighted operation is performed and the fused high-frequency difference map is multiplied pixel by pixel to obtain the pixel-level enhancement matrix. Furthermore, during model training, the loss function is: in, , and Weighting parameters to balance different losses; L mse The reconstruction loss is the mean square error (MSE) of the original image, L box To predict the L1 loss of the bounding box, L cls For classification loss functions that incorporate IoU; Step 5: Input the surveillance video images from the test set into the trained model to obtain the classification confidence score and prediction box information of the detected targets.

2. The method for target detection in surveillance video based on multi-frame high-frequency difference enhancement according to claim 1, characterized in that: Step 1 includes the following steps: (1) Perform interval sampling on the video frames, and extract training frames from the video starting from the first frame of the video at the set interval; (2) The specific methods of image preprocessing and enhancement are to perform random expansion, cropping, color distortion, flipping and resizing operations on the input image.

Images