Monitoring video target detection method based on multi-frame high-frequency difference enhancement

Abstract

The invention discloses a monitoring video target detection method based on multi-frame high-frequency difference enhancement, and the method comprises the following steps: 1, firstly, selecting a series of monitoring videos, carrying out the segmentation and frame extraction of the monitoring videos, and sampling one frame every ten frames as a training set, carrying out data preprocessing and enhancement operation on the input continuous image frames; and 2, constructing a multi-frame image reconstruction module to reconstruct an image according to the low-layer feature maps of the current frame and the reference frame, and optimizing parameters of the reconstruction module by calculating mean square error (MSE) loss between the original image and the reconstructed image. According to the monitoring video target detection method based on multi-frame high-frequency difference enhancement, a continuous frame high-frequency difference chart and a self-adaptive fusion enhancement feature strategy are ingeniously combined, high-frequency motion features are extracted and fused through reconstruction of continuous frames, and a powerful target detection framework is constructed; the target detection performance in the monitoring video can be obviously improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a surveillance video target detection method based on multi-frame high-frequency difference enhancement. Background Art

[0002] With the rapid development of computer vision and artificial intelligence technology, target detection, as a basic and key technology, has been widely used in many fields such as autonomous driving, video surveillance, and medical image analysis. The rapid popularization of deep convolutional neural networks in the field of static image target detection has shown great superiority in performance compared to traditional methods, and has gradually played a corresponding role in target detection tasks based on surveillance videos.

[0003] The core task of surveillance video target detection is to accurately locate and identify the target from each surveillance video frame, which is particularly important for practical application scenarios such as intelligent security monitoring, traffic control, and behavior analysis. Traditional video target detection methods, such as inter-frame difference method and background difference method, mainly detect moving targets by calculating the pixel difference between consecutive frames or the difference between the background and the current frame. Although these methods are effective in simple scenes, the detection effect is unstable when dealing with complex backgrounds, lighting changes, and target occlusion. The capture of moving targets is relatively rough, and it is difficult to meet the requirements of high precision.

[0004] Object detection algorithms based on deep learning, such as FasterR-CNN, SSD, and YOLO series, have demonstrated excellent performance in the field of static image object detection and have gradually been applied to video detection tasks. However, surveillance video object detection is more complex than static image detection. Objects in surveillance videos are affected by motion blur, occlusion, uneven lighting, and appearance changes. Existing object detection methods often show insufficient robustness when dealing with complex scenes with obvious lighting changes and rapid dynamic changes of objects, and it is difficult to meet the requirements of high precision and real-time performance.

[0005] Most of the current mainstream surveillance video target detection technologies rely on keyframe detection and optical flow information, and improve detection accuracy and speed by analyzing the association between video frames. For example, DFF only extracts features from keyframes through deep feature flow, and transfers these features to adjacent frames through optical flow, which reduces redundant calculations, but also brings about the problem of reduced detection accuracy. FGFA enhances the utilization of temporal information through multi-frame feature aggregation, but its high computational cost affects the real-time performance of the model. The THP method improves detection accuracy and speed through sparse recursive feature aggregation and optical flow estimation, but still relies on keyframe selection. In complex scenes with large illumination changes or target occlusion, the keyframe strategy is prone to failure, and the computational complexity of optical flow estimation affects real-time performance. In addition, these methods are still insufficient in adapting to target motion blur, occlusion and uneven illumination in surveillance videos, and it is difficult to obtain stable detection results in complex surveillance environments. In order to solve the limitations of existing technologies in surveillance video scenarios, a detection method that can effectively utilize multi-frame temporal information and has high accuracy and high real-time performance is urgently needed. Summary of the invention

[0006] The purpose of the present invention is to provide a surveillance video target detection method based on multi-frame high-frequency difference enhancement, so as to solve the problem that the current target detection algorithm based on deep learning proposed in the above background technology has made certain progress, but when facing the surveillance target detection task in complex dynamic scenes, it still faces the problems of imbalance between computational efficiency and detection accuracy, insufficient generalization ability of weak target detection, etc.

[0007] To achieve the above object, the present invention provides the following technical solution: a surveillance video target detection method based on multi-frame high-frequency difference enhancement, comprising the following steps, characterized in that: Step 1: First, select a series of surveillance videos and segment and extract frames from them. The frame extraction method is to sample one frame every ten frames as a training set. Then, perform data preprocessing and enhancement operations on the input continuous image frames. Step 2: Construct a multi-frame image reconstruction module to reconstruct the image based on the low-level feature maps of the current frame and the reference frame, and optimize the parameters of the reconstruction module by calculating the mean square error (MSE) loss between the original image and the reconstructed image; Step 3: Construct a high-frequency difference extraction module to extract the high-frequency information of the reconstructed image and the original image, and subtract the high-frequency information of the corresponding frames and perform a modulus operation; Step 4: Construct a high-frequency difference fusion enhancement module, which fuses the high-frequency difference information of consecutive frames by building a convolutional network, and integrates the global background information and local salient information modeling of consecutive frames to enhance the low-level features of the current frame and feeds them to the multi-scale fusion module and the detection head; Step 5: Input the surveillance video images in the test set into the trained model to obtain the classification confidence score and prediction box information of the detected target.

[0008] Preferably, the step 1 mainly includes the following steps: (1) Sampling video frames at intervals and extracting training frames from the video at set intervals starting from the first frame of the video; (2) The specific methods of image preprocessing and enhancement are to perform random operations such as expansion, cropping, color distortion, flipping, and resizing on the input image.

[0009] Preferably, the specific method of the multi-frame image reconstruction module in step 2 is: (1) Input the continuous video frames after data enhancement into the backbone network, which is implemented based on ResNet50, to obtain the output features of the last three stages in the network; (2) The low-level features are upsampled three times in a row, and the number of channels is halved each time. Finally, a three-channel reconstructed image is generated through convolution. The difference between the reconstructed image and the original input image is calculated, and the MSE loss constraint is used to optimize the reconstruction effect.

[0010] Preferably, the specific method of constructing the high-frequency difference extraction module in step 3 is: (1) Processing the high-frequency components of the current frame, the reference frame, the current frame reconstruction image, and the reference frame reconstruction image channel by channel, by applying a two-dimensional Fourier transform to the image of each channel, converting the image from the spatial domain to the frequency domain, and then shifting the zero-frequency component; (2) Create a high-pass filter mask based on the center position of the image and apply the high-pass filter mask to the image in the frequency domain to retain the high-frequency components in the image; (3) Calculate the difference in high-frequency components between the input image and the reconstructed image to obtain a difference map in the frequency domain, and then perform an inverse Fourier transform and modulo operation on it to obtain the actual high-frequency difference map.

[0011] Preferably, in step 4, a high-frequency difference fusion enhancement module is constructed to better fuse the features of the current frame and the reference frame: (1) This module concatenates the high-frequency difference maps of the current frame and the previous two frames, and integrates them into a single-channel high-frequency difference map through convolution operations. This fusion step provides more complete spatial details than a single frame, such as edge and texture information, by concatenating high-frequency difference maps of multiple frames. In this way, the network can more fully capture important features in the image, improve its understanding and processing capabilities of complex scenes, and enhance the overall spatial feature perception. By processing the concatenated high-frequency difference maps through convolution operations, the network can adaptively adjust the degree of attention to changes between different frames. This adaptive capability enables the model to dynamically adjust the feature fusion method according to the specific scene, ensuring good performance in complex time-series change scenes; (2) In order to further enhance the effect of temporal feature fusion and improve feature diversity, a similar method is used to fuse feature information of the feature maps of the current frame and the reference frame. Specifically, we perform a global pooling operation on the fused feature map and further extract high-level features through linear layers and activation layers. At the same time, we apply an average pooling operation and also perform feature extraction through linear layers and activation layers. After the features are extracted through pooling, they are weighted and multiplied pixel by pixel with the fused high-frequency difference map to obtain a pixel-level enhancement matrix. Furthermore, during the model training process, the loss function is: in, , and A weighting parameter to weigh different losses; Lmse is the original image mean square error (MSE) reconstruction loss, Lbox is the L1 loss of the predicted bounding box, and Lcls is the classification loss function combined with IoU.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: the surveillance video target detection method based on multi-frame high-frequency difference enhancement cleverly combines the continuous frame high-frequency difference map and the adaptive fusion enhancement feature strategy, and constructs a powerful target detection framework by reconstructing and extracting the high-frequency motion features of the continuous frames, which can significantly improve the target detection performance in the surveillance video. The specific structure is as follows: (1) The multi-frame image reconstruction module restores the shallow feature maps of continuous video frames to reconstructed images of the same size as the original images through reconstruction learning. This module is based on multi-level upsampling operations, so that the reconstructed images and original images of different frames can be compared and fused at the same scale, laying the foundation for the subsequent extraction of high-frequency information differences; (2) The core of the high-frequency difference extraction module is to analyze the high-frequency parts of the original image and the reconstructed image. First, the fast Fourier transform (FFT) is applied to the original image and the reconstructed image respectively, and then the high-frequency parts of the two are extracted through a high-pass filter to generate a high-frequency difference map. This high-frequency difference map captures the target motion, edge details, artifacts and other related high-frequency information features that cannot be accurately restored during the reconstruction process. By utilizing high-frequency differences, the model can more keenly perceive the motion changes of weak targets; (3) The fusion enhancement module integrates the high-frequency difference information of consecutive frames and utilizes the global background information and local salient features, so that the model can effectively capture the changes in the temporal dimension. This helps to model the temporal correlation in the surveillance video image sequence, thereby more completely capturing the key changes in dynamic scenes, improving the detection ability of weak targets and the generalization ability of complex scenes. The structure is simple and effective, and it is easy to deploy. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is the overall framework flow chart of the present invention. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] See also Figure 1 The present invention provides a technical solution: a surveillance video target detection method based on multi-frame high-frequency difference enhancement, which is divided into three steps: a multi-frame image reconstruction module, a high-frequency difference extraction module and an adaptive fusion enhancement module. In the present invention, X∈R 3×H×W Represents an RGB image, where H and W are the height and width of the input image, respectively, and the output Included categories and bounding box , and a reconstruction loss is generated to optimize the image reconstruction module.

[0016] Step 1: Select a series of surveillance videos and segment and extract frames from them. The frame extraction method is to sample one frame every ten frames as the training set, and then preprocess and enhance the input continuous image frames. The specific method is to randomly expand, crop, color distort, flip, resize and other operations on the input images. The images after these data processing operations will be directly used as the input of the backbone network.

[0017] Step 2: Reconstruct shallow features of continuous image frames. The multi-frame image reconstruction module is mainly based on the shallow feature map S3 in the last three stages of feature maps S3, S4, and S5 output by the backbone network. The module consists of three upsampling modules, one convolution layer and one activation function. The upsampling module consists of a transposed convolution with a convolution kernel of 4x4, a step size of 2 and a padding of 1, two convolution kernels of 3x3 and a padding of 1, and a ReLU activation function. After this process, the number of channels of the feature map is gradually halved, the spatial resolution is doubled, and the reconstructed image is finally obtained. ,This module optimizes the parameters of the reconstruction process by minimizing the reconstruction loss (MSE) so that the reconstructed image is as close to the original input image as possible.

[0018] Step 3: High-frequency difference extraction module. The purpose of this module is to extract the high-frequency component differences between the expected reconstruction images of adjacent frames through Fourier transform, so as to capture the fast-changing target boundaries and important changes such as detail edges in the surveillance video images.

[0019] Step 3.1 Fourier transform and frequency domain centering, denoted as frame t and frame t-1 , , and the corresponding reconstruction graph is denoted by , Perform two-dimensional fast Fourier transform (FFT) on each channel of the original image and reconstructed image of frame t, and convert the image from the spatial domain to the frequency domain. The results are and , for each channel of the t-1 frame image, the same operation is performed to obtain and Then, the frequency domain data is centered and the fftshift function is used to move the low-frequency components to the center of the spectrum. , Perform the same operation on each channel of frame t-1 to obtain and .

[0020] Step 3.2: Construct the high-pass filter mask. Use the following formula to construct the mask:

[0021] Where i and j are the row and column indices of the spectrum, respectively. The radius r controls the size of the high-frequency portion retained, and is set to 0.1 in the specific experiment. Applying the high-pass filter mask to the frequency domain image suppresses the low-frequency components and retains the high-frequency components.

[0022] The same processing is performed on each channel of the t-1 frame to obtain and . The difference is then calculated in the frequency domain: To convert the high-frequency differences from the frequency domain to the spatial domain, we first apply inverse centering to the difference spectrum to obtain , and then perform a two-dimensional inverse Fourier transform to obtain the high-frequency difference map: Then the mean is calculated along the channel dimension. Finally, since the Fourier transform may produce complex values, the high-frequency difference map is modulo to obtain the actual single-channel high-frequency difference map. : in: represents the final high-frequency difference map, , Represents the high-frequency difference value of the cth channel. C is the number of channels, and its value is 3. Step 4: High-frequency difference fusion enhancement module. This module adaptively fuses the high-frequency difference maps of multiple frames and combines them with the feature map of the current frame to enhance the model's understanding of the global scene and its perception of small targets in the scene.

[0023] This module receives four inputs, namely the S3-level feature map of frame t , S3-level feature map of frame t-1 , S3-level feature map of frame t-2 And the high-frequency difference map of the three frames In order to fuse the multi-frame difference maps together and preserve the change information between frames more comprehensively, a convolutional network is constructed to fuse the multi-frame difference maps and preserve the key change areas. First, the high-frequency difference maps of each frame are spliced ​​along the channel to obtain , and then After a 3x3 convolution kernel and ReLU activation function, a single-channel fused high-frequency difference map is obtained. .

[0024] in, Again Perform bilinear interpolation to obtain , because In order to improve the diversity of features in the temporal dimension, we integrate the spatial variation information of multiple frames and merge the shallow feature maps S3 of multiple frames and use maximum pooling to obtain local salient information. , using average pooling to capture global stability information , and then generate the timing weight And apply T to The above is used to generate a pixel-by-pixel high-frequency difference enhancement matrix H. The weight vector T is conducive to further establishing the temporal association between multiple frames, enhancing the understanding of the overall motion of the target, and improving the detection ability of targets such as occlusion and motion blur.

[0025] Finally, the pixel-by-pixel attention matrix H is used to By combining H and Multiply pixel by pixel to get the enhanced feature map ,Right now Step 5: Model convergence and testing process: Use the total loss function L to supervise model training until convergence.

[0026] The total loss function L of the present invention consists of three parts: the mean square error (MSE) reconstruction loss Lmce of the original image, the L1 loss of the predicted bounding box, and the classification loss Lcls introduced by IoU. Reconstruction loss is often used to evaluate the performance of generative models in data reconstruction tasks. It measures the difference between the model reconstruction output and the original data. In image processing applications, reconstruction loss is usually calculated using MSE. The reconstruction loss formula based on MSE is as follows: Where N represents the batch_size.

[0027] L1 loss is often used to calculate the deviation between the predicted bounding box and the target bounding box, directly measuring the difference between the two in coordinate space. Where N is the number of bounding boxes, Represents the real frame information, Represents the predicted bounding box information.

[0028] The classification loss function of the present invention combines the IoU weight and FocalLoss ideas to optimize the classification results of the bounding box. First, the IoU value between the predicted box and the real box is calculated to generate the weight for loss calculation. Then, combined with the predicted classification score activated by Sigmoid, the weighted binary cross entropy loss is used for optimization. The calculation formula of its loss function is: in

[0029] is the model output of the i-th sample, which represents the original score of the model for different categories. is the true category label of the i-th sample. The category labels are weighted with the IoU value to measure the matching degree of the target box. It is a combination of FocalLoss and IoU, which is used to dynamically adjust the loss weight of the sample. The Lcls loss function of the present invention combines the prediction score and the target score. The prediction score is obtained by combining the classification of the model The Sigmoid activation function converts it into a probability value, which represents the model's predicted probability for each category. The target score generates a weighted supervision signal for guiding model learning by combining the IoU value with the One-Hot encoding of the true category label. The final loss function is based on the weighted binary cross entropy loss of the predicted score and the target score, and dynamically adjusts samples of different difficulty by combining the IoU and FocalLoss weights. The development language is Python, the system of the development environment is Ubuntu, the deep learning framework is Pytorch, and the dataset is: UA-DETRAC vehicle detection dataset. The dataset contains 10 hours of video shot at 24 different locations in Beijing and Tianjin, China using a Canon EOS 550D camera. The video is recorded at 25 frames per second with a resolution of 960×540 pixels. There are more than 140,000 frames and 8,250 manually annotated vehicles in the UA-DETRAC dataset, with a total of 1.21 million labeled object bounding boxes, including about 82,085 images in the training set and about 56,167 images in the test set.

[0030] Model Training Use the Pytorch framework to perform steps 1-4 according to Figure 1 The model was trained for 60 epochs, the batch size was set to 1, the AdamW optimizer was used, and the basic learning rate was set to 1e -4 , weight decay is 1e -4 The model uses average precision (AP) as the evaluation indicator, and records the AP indicator of the validation set and the parameters of the current model for each epoch.

[0031] Model Testing The model with the largest AP during training was used as the test model. We evaluated the performance of our proposed method on the UA-DETRAC dataset in multiple scenarios. The calculation of the average precision (AP) score follows the method described in the paper "The Pascal Visual Object Classes Challenge: A Retrospective" and is tested on the DETRAC test set. The confidence threshold for each image prediction is set to 0.7; setting a higher confidence threshold can reduce low-confidence prediction results, thereby improving the reliability of the prediction results.

[0032] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A surveillance video target detection method based on multi-frame high-frequency difference enhancement, comprising the following steps, characterized in that: Step 1: First, select a series of surveillance videos and segment and extract frames from them. The frame extraction method is to sample one frame every ten frames as a training set. Then, perform data preprocessing and enhancement operations on the input continuous image frames. Step 2: Construct a multi-frame image reconstruction module to reconstruct the image based on the low-level feature maps of the current frame and the reference frame, and optimize the parameters of the reconstruction module by calculating the mean square error (MSE) loss between the original image and the reconstructed image; Step 3: Construct a high-frequency difference extraction module to extract the high-frequency information of the reconstructed image and the original image, and subtract the high-frequency information of the corresponding frames and perform a modulus operation; Step 4: Construct a high-frequency difference fusion enhancement module, which fuses the high-frequency difference information of consecutive frames by building a convolutional network, and integrates the global background information and local salient information modeling of consecutive frames to enhance the low-level features of the current frame and feeds it to the multi-scale fusion module and the detection head; Step 5: Input the surveillance video images in the test set into the trained model to obtain the classification confidence score and prediction box information of the detected target.

2. The method for monitoring video target detection based on multi-frame high-frequency difference enhancement according to claim 1, characterized in that: The step 1 mainly includes the following steps: (1) Sampling video frames at intervals and extracting training frames from the video at set intervals starting from the first frame of the video; (2) The specific methods of image preprocessing and enhancement are to perform random operations such as expansion, cropping, color distortion, flipping, and resizing on the input image.

3. The method for monitoring video target detection based on multi-frame high-frequency difference enhancement according to claim 1, characterized in that: The specific method of the multi-frame image reconstruction module in step 2 is: (1) Input the continuous video frames after data enhancement into the backbone network, which is implemented based on ResNet50, to obtain the output features of the last three stages in the network; (2) The low-level features are upsampled three times in a row, and the number of channels is halved each time. Finally, a three-channel reconstructed image is generated through convolution. The difference between the reconstructed image and the original input image is calculated, and the MSE loss constraint is used to optimize the reconstruction effect.

4. The method for monitoring video target detection based on multi-frame high-frequency difference enhancement according to claim 1, characterized in that: The specific method of constructing the high-frequency difference extraction module in step 3 is: (1) Processing the high-frequency components of the current frame, the reference frame, the current frame reconstruction image, and the reference frame reconstruction image channel by channel, by applying a two-dimensional Fourier transform to the image of each channel, converting the image from the spatial domain to the frequency domain, and then shifting the zero-frequency component; (2) Create a high-pass filter mask based on the center position of the image and apply the high-pass filter mask to the image in the frequency domain to retain the high-frequency components in the image; (3) Calculate the difference in high-frequency components between the input image and the reconstructed image to obtain a difference map in the frequency domain, and then perform an inverse Fourier transform and modulo operation on it to obtain the actual high-frequency difference map.

5. The method for monitoring video target detection based on multi-frame high-frequency difference enhancement according to claim 1, characterized in that: In step 4, a high-frequency difference fusion enhancement module is constructed to better fuse the features of the current frame and the reference frame: (1) This module concatenates the high-frequency difference maps of the current frame and the previous two frames, and integrates them into a single-channel high-frequency difference map through convolution operations. This fusion step provides more complete spatial details than a single frame, such as edge and texture information, by concatenating high-frequency difference maps of multiple frames. In this way, the network can more fully capture important features in the image, improve its understanding and processing capabilities of complex scenes, and enhance the overall spatial feature perception. By processing the concatenated high-frequency difference maps through convolution operations, the network can adaptively adjust the degree of attention to changes between different frames. This adaptive capability enables the model to dynamically adjust the feature fusion method according to the specific scene, ensuring good performance in complex time-series change scenes; (2) In order to further enhance the effect of temporal feature fusion and improve feature diversity, a similar method is used to fuse feature information of the feature maps of the current frame and the reference frame. Specifically, we perform a global pooling operation on the fused feature map and further extract high-level features through linear layers and activation layers. At the same time, we apply an average pooling operation and also perform feature extraction through linear layers and activation layers. After the features are extracted through pooling, they are weighted and multiplied pixel by pixel with the fused high-frequency difference map to obtain a pixel-level enhancement matrix. Furthermore, during the model training process, the loss function is: in, , ,and It is a weighted parameter for weighing different losses; Lmse is the original image mean square error (MSE) reconstruction loss, Lbox is the L1 loss of the predicted bounding box, and Lcls is the classification loss function combined with IoU.

Images