A Method for Detecting Moving Vehicles in Satellite Videos Based on Multi-Frame Difference

Abstract

The present invention discloses a method for detecting moving vehicles in satellite videos based on multi-frame difference, which includes the following steps: establishing a moving vehicle detection data set and dividing it into a training set and a test set; constructing an improved YOLOv5 network model; introducing a frame difference module into the improved YOLOv5 network model; modifying the data reading method of the frame difference module to form a YOLOv5 network model that takes into account spatio-temporal information; inputting the training set into the YOLOv5 network model that takes into account spatio-temporal information for training, and obtaining the optimal model after training; inputting the test set into the optimal model and outputting the vehicle target detection result. The present invention combines the temporal feature information of satellite videos, performs detection in the form of continuous frame triples, and through the frame difference module, improves the model's detection ability for moving vehicles in satellite videos. The combination with data augmentation methods such as Mosaic makes more full use of spatio-temporal information. In the case of insufficient appearance features of single-frame targets, inter-frame information is used to improve the detection accuracy of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of moving vehicle detection, and particularly relates to a method for detecting moving vehicles in satellite video based on multi-frame difference. Background Art

[0002] In recent years, the emergence of deep learning technology has provided new ideas and research methods for the field of image target detection. Due to its powerful image learning ability and feature abstraction ability, it has received attention in many image processing fields. A video satellite is a new type of earth observation satellite that can continuously observe a certain area in a "staring" manner and record the observation information in the form of video, which is conducive to the monitoring of moving targets. The so-called "staring" means that as the satellite moves, the optical imaging system always stares at a certain target area and can continuously observe the changes within the field of view. Compared with traditional surveillance videos, using satellite videos to detect vehicles can provide more comprehensive analysis results on a large scale. Compared with high-resolution satellite image data, satellite videos have a higher temporal resolution and can detect vehicles more continuously. However, due to the high perspective characteristics of satellite videos, when using satellite videos for vehicle detection, the following problems exist: (1) Due to the limitation of the imaging spatial resolution of satellite videos, vehicles are only a few to a dozen pixels in size in the video, and there is less detailed information about the vehicles; (2) During the imaging process, the movement of the sensor causes displacement changes of targets such as buildings and trees, resulting in many pseudo moving targets in the detection process and making the background more complex; (3) Some vehicles in the satellite video sequence have a similar contrast to the background.

[0003] Current satellite video moving vehicle detection algorithms are mainly divided into two categories: ① Traditional target detection methods. The optical flow method usually requires a large amount of calculation and is easily affected by changes in illumination, which limits its application in satellite videos; most background difference methods and frame difference methods are easily affected by background movement and contrast changes, and these factors are common in satellite videos. Traditional target detection methods often need to manually modify parameters for different scenarios according to experience, and the detection performance depends on manually designed features. It is difficult to achieve high detection accuracy in different complex scenarios using unified parameters. ② Target detection methods based on deep learning. This method overcomes the problems of traditional methods affected by illumination and background movement to a certain extent. However, there are also some deficiencies. The moving vehicle targets in satellite videos are small and blurred, and have a low contrast with the background. Detecting using only a single frame image has poor accuracy. In the selection of data augmentation methods, simply performing data augmentation methods such as rotation, scaling, and cropping on the current image cannot effectively increase the complexity of the data and is difficult to handle the detection of moving vehicles in complex scenarios.

[0004] In order to overcome the problems of illumination and background movement in the detection of moving vehicle targets in satellite videos, in existing algorithms, the optical flow method calculates the motion field for each pixel point in a video or image sequence and analyzes the temporal changes and correlations of the motion field to determine the motion changes of each pixel point's position, and then detects moving targets from the background. Li et al. first processed the image with the Canny edge detection operator, then calculated the displacement of each pair of edges based on the bidirectional optical flow method and established a displacement histogram of all matching edges, and finally matched the edge points of the moving target based on edge constraints to determine the moving target area. The optical flow method can detect relatively complete information of moving targets and has good anti-interference ability for video jitter, but the disadvantage is that the computational complexity is large, it is difficult to meet the real-time requirement, and it is also sensitive to illumination and not applicable to scenarios with large illumination changes.

[0005] The background subtraction method models the background through multiple frames of images, and then performs a difference operation between the current frame image and the background image to obtain moving targets. The process of background subtraction usually consists of three steps: 1) Background initialization. Background initialization is also known as background estimation, background generation, and background restoration, which is used to initialize the background model to form a video frame without foreground targets; 2) Background update. During the process of moving target detection, due to uncertain factors such as illumination, motion blur, and pseudo-moving targets, the background image often changes. In order to suppress the influence of uncertain factors on target detection and achieve accurate detection of moving targets, it is necessary to continuously update the background during the moving target detection process; 3) Foreground detection. Calculate the foreground by comparing the current video frame and the background model. The calculation result is a binary mask, where white pixels represent the foreground and black represents the background. In some background subtraction algorithms, post-processing steps are performed after foreground detection to refine the detected moving targets. Ahmadi et al. used the background subtraction method and the Otsu threshold (Otsu threshold segmentation method) for each frame to detect moving targets. The background subtraction method is the most widely used method in traditional moving target detection algorithms. Its advantages are simple calculation, easy implementation, higher accuracy compared to other traditional moving target detection algorithms, and most background subtraction algorithms can achieve real-time processing. However, the disadvantage is that it is sensitive to occlusion, illumination changes, and background jitter in the scene. Especially in the case of background jitter, the background subtraction method will generate a large number of pseudo-moving targets.

[0006] Chen et al. used the background difference method based on the K-Nearest Neighbor (KNN) model to obtain detection results with high recall rate, and then used a simple convolutional neural network to suppress false targets. Although the model is simple, it largely depends on the results of the background difference method and does not utilize the powerful feature extraction and expression capabilities of the neural network. At the same time, since this strategy cascades the two methods, the detection speed is slow. Therefore, there is an urgent need for a moving vehicle detection method that can overcome the difficulty of effectively using data augmentation strategies to improve data complexity and solve the problem of poor detection accuracy in single-frame scenarios. Summary of the Invention

[0007] To solve the problems existing in the prior art, the present invention provides a moving vehicle detection method for satellite videos based on multi-frame difference. By introducing a Frame Difference Module (FDM), the inter-frame difference information between adjacent frames is utilized to enrich the feature information of vehicles in the current frame; the Mosaic data augmentation method is improved to enhance the model's ability to handle complex scenarios; and it is effectively combined with the frame difference module. Using continuous frame triples as the model input and introducing the Focal Loss function, the model focuses more on difficult-to-classify samples, thereby achieving high-precision detection of moving vehicles in satellite videos and solving the problems mentioned in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: A moving vehicle detection method for satellite videos based on multi-frame difference, including the following steps:

[0009] S1. Process the collected satellite video moving vehicle data, establish a moving vehicle detection data set, and divide the data set into a training set and a test set;

[0010] S2. Construct an improved YOLOv5 network model;

[0011] S3. Introduce a frame difference module into the improved YOLOv5 network model, place it in the first layer of the feature extraction part, obtain differential time information by performing inter-frame difference on adjacent frames through convolution, and fuse it with the spatial information of the current frame to enhance the features of vehicle targets in the current frame;

[0012] S4. Modify the data reading method of the frame difference module to form a YOLOv5 network model that takes into account spatio-temporal information;

[0013] S5. Input the training set data into the YOLOv5 network model that takes into account spatio-temporal information for training, and obtain the optimal model after training;

[0014] S6. Input the test set data into the optimal model for model testing, and output the vehicle target detection results.

[0015] Preferably, in step S1, the processing of the collected satellite video moving vehicle data specifically includes data preprocessing, data calibration, and data format conversion.

[0016] Preferably, in step S2, the improved YOLOv5 network model specifically refers to replacing the cross-entropy loss function of the original YOLOv5 model with the Focal Loss function.

[0017] Preferably, in step S3, the construction method of introducing the frame difference module specifically includes:

[0018] Let V = {P1, P2,...P t ,..., P n} be the satellite video, and P t be the t-th frame of the satellite video;

[0019] Input the consecutive frames V t = {P t-1 , P t , P t+1} into the improved YOLOv5 network, and extract the spatial information of the consecutive frames through a two-layer convolution. Through the spatial feature calculation formula:

[0020] F t = C1·P t ,

[0021] Obtain the spatial features {F t-1 , F t , F t+1} corresponding to the consecutive frames;

[0022] Perform pairwise differences on the obtained spatial features. The calculation formula of the difference features:

[0023] F (t-1，t+1) = C2·|F t-1 - F t+1 |

[0024] Then, obtain the difference features {F (t-1，t) , F (t-1，t+1) , F (t，t+1)} through a 3×3 convolution;

[0025] Fuse the difference features and form temporal features through a 3×3 convolution. The calculation of the temporal features:

[0026] F (t-1，t，t+1) = C3·Concat(F (t-1，t) , F (t-1，t+1), F (t，t+1) );

[0027] Fuse the temporal features with the spatial features of the current frame, and form the final fused features that combine temporal and spatial information through a 3×3 convolution. The calculation of the fused features is as follows:

[0028] FS t = C4·Concat(F t , F (t-1，t，t+1) )

[0029] In the formula, Concat() is the concatenation operation. C1 is a 6×6 convolution with a stride of 2 and a 3×3 convolution with a stride of 1. C2, C3, and C4 are 3×3 convolutions with a stride of 1. The convolution structure is the same as the basic convolution structure of the YOLOv5 model, including a 2D convolution, batch normalization, and the SiLU activation function.

[0030] Preferably, the step S4 specifically includes: changing the single-frame input to a continuous-frame triple input, and then enhancing the data by combining conventional data augmentation and Mosaic data augmentation.

[0031] Preferably, the conventional data augmentation includes data rotation and data scaling.

[0032] Preferably, the optimal model is determined as the optimal model when the F1 accuracy index of the Nth epoch is the largest in the historical values, and the iteration continues until the end of model training. If the difference between the current epoch and the epoch of the historical optimal model is greater than a fixed threshold, the model training is terminated early, and the final optimal model is output. The fixed threshold is set to 100.

[0033] The beneficial effects of the present invention are as follows:

[0034] 1) The method for detecting moving vehicles in satellite videos based on multi-frame difference proposed by the method of the present invention uses deep learning methods, combines the temporal feature information of satellite videos, and performs detection in the form of continuous-frame triples, achieving high accuracy and high real-time performance, and realizing the end-to-end processing of detecting moving vehicles in satellite videos.

[0035] 2) The frame difference module proposed by the present invention can significantly improve the model's ability to detect moving vehicles in satellite videos while slightly increasing the detection time and video memory occupancy. The combination with data augmentation methods such as Mosaic enables the model to make more full use of spatio-temporal information, further improving the detection accuracy of the model. When the appearance features of single-frame targets are insufficient, inter-frame information is used to improve the detection accuracy of the model.

[0036] 3) The data augmentation optimization proposed by the present invention can better improve the recall rate of the model and will not significantly reduce the precision rate of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the improved YOLOv5 model structure;

[0038] Figure 2 It is a schematic diagram of the structure of the Frame Difference Module (FDM);

[0039] Figure 3 It is a schematic diagram of the principle of combining conventional data augmentation methods;

[0040] Figure 4 It is a schematic diagram of the principle of combining Mosaic data augmentation methods;

[0041] Figure 5 It is a schematic diagram of the optimal model training and selection;

[0042] Figure 6 It is a schematic diagram of the technical idea step flow of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0044] Please refer to Figures 1-6 , the present invention provides a technical solution: a method for detecting moving vehicles in satellite videos based on multi-frame difference, including the following steps:

[0045] S1. Process the collected satellite video moving vehicle data, establish a moving vehicle detection data set, and divide the data set into a training set and a test set;

[0046] S2. Construct an improved YOLOv5 network model,

[0047] S3. Introduce a frame difference module into the improved YOLOv5 network model, place it in the first layer of the feature extraction part, obtain differential time information by performing frame-by-frame difference on adjacent frames using convolution, and fuse it with the spatial information of the current frame to enhance the features of the vehicle target in the current frame;

[0048] S4. Modify the data reading method of the frame difference module to form a YOLOv5 network model that takes into account spatio-temporal information;

[0049] S5. Input the training set data into the YOLOv5 network model that takes into account spatio-temporal information for training, and obtain the optimal model after the training ends.

[0050] S6. Input the test set data into the optimal model for model testing, and output the vehicle target detection results.

[0051] Aiming at the problem of poor detection accuracy for single-frame images, the present invention proposes a YOLOv5 satellite video moving vehicle detection method based on continuous frame triples. By introducing a Frame Difference Module (FDM), using the inter-frame difference information between adjacent frames, the feature information of the vehicle in the current frame is enriched, thereby enhancing the extraction ability of moving targets and reducing the interference of stationary vehicle targets.

[0052] Aiming at the problem of being difficult to effectively utilize data augmentation strategies to increase data complexity, the present invention designs an improved Mosaic data augmentation method to improve the model's ability to handle complex scenarios; and in order to effectively combine with the frame difference module, continuous frame triples are used as the model input; the Focal Loss function is introduced to make the model more focused on difficult-to-classify samples. Thus, high-precision detection of moving vehicles in satellite videos is achieved.

[0053] The technical solution of the present invention mainly includes a data augmentation method, a frame difference module, and a YOLOv5 network model. The improved YOLOv5 network model is as Figure 1 shown. Using the YOLOv5 network model as the basis, a Frame Difference Module (FDM) is introduced. By using the inter-frame difference information between adjacent frames, the feature information of the vehicle in the current frame is enriched; considering that when using frame difference, the consistency between adjacent frames needs to be maintained, directly introducing the frame difference module will make the model unable to use data augmentation methods. By changing the data reading method of the YOLOv5 model from single-frame input to continuous frame triple input, the frame difference module is combined with conventional data augmentation methods such as rotation and scaling, as well as the Mosaic data augmentation method, effectively utilizing the spatio-temporal information of moving vehicles in satellite videos and improving the model's detection ability for moving vehicles in satellite videos.

[0054] (1) Optimization of the loss function:

[0055] For the input image, the size of moving vehicles in satellite videos is only a few to a dozen pixels, and the foreground information in a single image is less while the background information is more. As the training progresses, background information often accumulates continuously, thus drowning out some targets and affecting the detection accuracy of the model. In addition, compared with ordinary remote sensing images with tens of thousands of pictures, the data volume of the satellite video moving target detection dataset is small. During the model training process, overfitting phenomena are prone to occur.

[0056] To address the above problems, the present invention optimizes the YOLOv5 model, including replacing the cross-entropy loss function of the YOLOv5 model with the Focal Loss function and adding richer data augmentation methods, thereby improving the model's detection ability for moving vehicles. Focal Loss is a loss function that addresses the imbalance problem between hard examples and easy examples. The target is moving vehicles in satellite videos. Due to resolution limitations and vehicle movement, some vehicles will be blurred, which are hard examples compared to other moving vehicles. Therefore, it is necessary to introduce the Focal Loss function. Data augmentation is a method that can effectively improve the detection performance of small targets. By expanding the scale of the training dataset and enriching the diversity of the dataset, data augmentation can well improve the robustness and generalization ability of the detection model.

[0057] (2) Construction of the frame difference module:

[0058] Moving vehicles in satellite videos are just "mosaics" with only a few pixels in size, and the spatial information is not rich. However, satellite video data has a unique advantage compared to other remote sensing data, that is, it has a high temporal resolution. When the spatial information of a single frame is insufficient, if the temporal information can be used to enrich the features of moving vehicles.

[0059] Based on the YOLOv5 model, the present invention introduces a frame difference module, as Figure 2 shown. It is placed in the first layer of the feature extraction part. By using convolution to perform inter-frame difference on adjacent frames to obtain differential time information and fusing it with the spatial information of the current frame to enhance the features of the vehicle target in the current frame and improve the detection effect. Here, it is chosen to directly input the original image and then perform frame difference through the frame difference module, rather than performing frame difference first and then directly inputting the result of the frame difference into the model. The reason is that the frame difference module also participates in backpropagation and has learning ability. As the model is trained, the weights of the convolution in the frame difference module will also continuously adapt to the detection of moving vehicles in satellite videos. At the same time, compared with the method of directly subtracting images, using convolution for frame difference is more reasonable and effective.

[0060] Let V = {P1, P2,...P t ,..., P n} be a satellite video, and P t be the t-th frame of the satellite video. The construction method of the frame difference module is as follows:

[0061] For consecutive frames V t = {P t-1 , P t , P t+1}Input the improved YOLOv5 network, extract the spatial information of consecutive frames through a two-layer convolution, and use the spatial feature calculation formula:

[0062] F t = C1·P t ,

[0063] Obtain the spatial features {F t-1 , F t , F t+1} corresponding to the consecutive frames;

[0064] Perform pairwise differences on the obtained spatial features. The calculation formula for the difference features:

[0065] F (t-1，t+1) = C2·|F t-1 - F t+1 |

[0066] Then, obtain the difference features {F (t-1，t) , F (t-1，t+1) , F (t，t+1)} through a 3×3 convolution;

[0067] Fuse the difference features and form temporal features through a 3×3 convolution. The calculation of the temporal features:

[0068] F (t-1，t，t+1) = C3·Concat(F (t-1，t) , F (t-1，t+1) , F (t，t+1) );

[0069] Fuse the temporal features with the spatial features of the current frame and form the final fusion features that combine temporal and spatial information through a 3×3 convolution. The calculation of the fusion features:

[0070] FS t = C4·Concat(F t , F (t-1，t，t+1) )

[0071] In the formula, Concat() is the concatenation operation. C1 is a 6×6 convolution with a stride of 2 and a 3×3 convolution with a stride of 1. C2, C3, and C4 are 3×3 convolutions with a stride of 1. The convolution structure is the same as the basic convolution structure of the YOLOv5 model, including a 2D convolution, batch normalization, and the SiLU activation function.

[0072] (3) Data augmentation optimization:

[0073] After adding the frame difference module, although the inter-frame difference information can be effectively utilized, there is still a problem: ordinary deep learning models input images frame by frame, and data augmentation operations such as rotation and scaling are also performed on single images, and there is no association between them. After adding the frame difference module, if the original data augmentation method is still used, due to the inconsistent data augmentation methods for consecutive frame images, the difference information of consecutive frames cannot be used as useful information during inter-frame difference calculation.

[0074] To address this problem, the present invention further modifies the data reading method of the frame difference module to enable it to be combined with conventional data augmentation methods. Specifically, after adding the frame difference module to the model, the present invention takes three consecutive frames as a triple, changing the data reading method of the original model. That is, when facing data augmentation, the same data augmentation is performed within the same group, while the data augmentation between different groups remains random. This overcomes the problem of being unable to use data augmentation methods after adding the frame difference module. The specific principle is as Figure 3 shown.

[0075] On the basis of the conventional triple data augmentation, the Mosaic data augmentation method is further combined. Specifically, taking the upper left corner as the current frame sequence, and the other three parts respectively use three consecutive random frames in the dataset. This not only ensures the consistency of the frame difference information but also ensures the diversity of the data augmentation method, enhances the randomness of the data, enriches the dataset, and the specific principle is as Figure 4 shown.

[0076] The overall technical route of the present invention is as Figure 6 shown. First, the data processing methods for satellite video moving vehicle detection are studied, including data preprocessing, data calibration, data format conversion, etc.; then, the training method of the YOLOv5 model is studied, and the model is optimized for moving vehicles and improved considering spatio-temporal information on the basis of the YOLOv5 model to improve the accuracy of satellite video moving vehicle detection; the training set data is input into the YOLOv5 network model considering spatio-temporal information for training, as Figure 5 shown. After training, the optimal model is obtained. Then, the obtained optimal model is used to detect vehicle targets in the satellite video moving vehicle detection dataset and obtain the detection results; finally, the satellite video moving vehicle detection results are evaluated from both quantitative and qualitative perspectives to verify the effectiveness of the present invention. The overall accuracy (F1 score) is improved by 2.9% and 9.3% respectively on the SkySat and Jilin-1 datasets, enhancing the data complexity and improving the detection ability of the model.

[0077] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for detecting moving vehicles in satellite videos based on multi-frame difference, characterized in that, It includes the following steps: S1. Process the collected satellite video moving vehicle data, establish a moving vehicle detection data set, and divide the data set into a training set and a test set; S2. Construct an improved YOLOv5 network model; the improved YOLOv5 network model specifically refers to replacing the cross-entropy loss function of the original YOLOv5 model with the Focal Loss loss function; S3. Introduce a frame difference module into the improved YOLOv5 network model, place it in the first layer of the feature extraction part, obtain differential time information by performing frame difference on adjacent frames using convolution, and fuse it with the spatial information of the current frame to enhance the features of the vehicle target in the current frame; The specific construction method of introducing the frame difference module includes: Let \(V = \{P_1, P_2, \ldots P t , \ldots, P n \}\) be the satellite video, and \(P t \) be the \(t\)-th frame of the satellite video; Input the continuous frames V t ={P t-1 , P t , P t+1} into the improved YOLOv5 network, and extract the spatial information of the continuous frames through a two-layer convolution. Through the spatial feature calculation formula: F t = C1·P t , Obtain the spatial features {F t-1 , F t , F t+1} corresponding to consecutive frames; Perform pairwise differences on the obtained spatial features. The calculation formula for the differential features: F (t-1,t+1) = C · |F t-1 -F t+1 | Another 3×3 convolution is performed to obtain the differential features {F (t-1，t) , F (t-1，t+1) , F (t，t+1)}; Fuse the differential features and form temporal features through a 3×3 convolution. The calculation of the temporal features: F (t-1,t,t+1) = C3·Concat(F (t-1,t) , F (t-1,t+1) , F (t,t+1) ); Fuse the temporal features with the spatial features of the current frame and form the final fusion features that combine temporal and spatial information through a 3×3 convolution. The calculation of the fusion features: FS t = C4·Concat(F t , F (t-1，t，t+1) ) In the formula, Concat() is a concatenation operation. C1 is a 6×6 convolution with a stride of 2 and a 3×3 convolution with a stride of 1. C2, C3, and C4 are 3×3 convolutions with a stride of 1. The convolution structure is the same as the basic convolution structure of the YOLOv5 model, including a 2D convolution, batch normalization, and the SiLU activation function; S4. Modify the data reading method of the frame difference module to form a YOLOv5 network model that takes into account spatio-temporal information; change the single-frame input to a continuous frame triple input, and then enhance the data by combining conventional data augmentation and Mosaic data augmentation; S5. Input the training set data into the YOLOv5 network model that takes into account spatio-temporal information for training, and obtain the optimal model after training; S6. Input the test set data into the optimal model for model testing, and output the vehicle target detection result.

2. The method for detecting moving vehicles in satellite videos based on multi-frame difference according to claim 1, wherein: In step S1, the processing of the collected satellite video moving vehicle data specifically includes data preprocessing, data calibration, and data format conversion.

3. The satellite video moving vehicle detection method based on multi-frame difference according to claim 1, characterized in that: The conventional data augmentation includes data rotation and data scaling.

4. The satellite video moving vehicle detection method based on multi-frame difference according to claim 1, characterized in that: The optimal model is determined as the optimal model when the F1 accuracy index of the Nth epoch is the maximum historical value, and the iteration continues until the model training ends; if the difference between the current epoch and the epoch of the historical optimal model is greater than a fixed threshold, the model training is terminated early, and the final optimal model is output. The fixed threshold is set to 100.

Images