Satellite-borne video vehicle detection method

Abstract

The invention belongs to the technical field of satellite-borne video data vehicle extraction, and particularly relates to a satellite-borne video vehicle detection method. The method comprises the following steps: firstly, acquiring original spaceborne video data, and preprocessing to improve the quality of the spaceborne video data; carrying out preliminary vehicle detection on the preprocessed satellite-borne video data frame by frame; and finally, according to the preliminary vehicle detection result, an SORT algorithm is adopted to carry out vehicle tracking so as to complete moving vehicle detection, secondary fine detection is carried out on failed vehicle targets in the vehicle tracking process so as to obtain high-precision moving vehicle targets, and the secondary fine detection is that the failed targets are associated in the multi-target tracking stage to carry out emphasized detection, so that the accuracy of the target tracking is improved. Therefore, the purpose of key detection of key areas is achieved. Specifically, distance-based secondary matching is performed on a prediction frame and a detection frame which are not successfully associated, so that the situation of association failure caused by the fact that the intersection-to-parallel ratio is 0 due to sudden acceleration and deceleration or steering of the vehicle is eliminated, and the detection precision of the moving vehicle is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of vehicle extraction from satellite-borne video data, and in particular relates to a satellite-borne video vehicle detection method. Background Art

[0002] Satellite video can quickly provide city-level vehicle dynamic information and provide reliable data for urban informatics, smart cities, and urban development planning. In recent years, low-orbit video satellites have attracted the attention of more and more countries because they can quickly obtain large-scale high-resolution video data, and have gradually become the forefront of the current development of remote sensing technology. Vehicle extraction from satellite-borne video data is of great significance in urban traffic planning, road construction, and smart city construction. However, the characteristics of high-altitude staring shooting by video satellites also make the moving targets small in scale and similar in features, and pseudo-moving targets and noise interference are serious. How to accurately extract moving vehicles from satellite-borne videos is still a hot topic and difficulty in the current research of remote sensing processing and application.

[0003] At present, the processing of satellite videos at home and abroad is mainly focused on the two stages of detection and tracking. In the detection stage, most methods directly introduce computer vision to detect moving targets in close-range videos. Although such an introduction is feasible and effective, it is necessary to consider the unique properties of satellite videos. The main manifestations are that the satellite shooting position is high and the imaging range is wide, resulting in a large number of complex objects in the background, and moving targets appear as small, dim targets; the staring imaging characteristics of video satellites cause static targets between frames to appear in motion, that is, parallax pseudo-motion. Traditional moving target detection algorithms cannot effectively identify pseudo-moving targets, real moving vehicles, and noise, and cannot effectively detect vehicle targets that are too dim and small. This is also the difficulty in detecting moving vehicles in satellite videos.

[0004] In order to solve the problem of being unable to distinguish noise, pseudo-moving targets, and vehicle targets, many scholars at home and abroad have proposed a variety of prior knowledge to optimize target detection algorithms. For example, the paper "Rapid Detection of Moving Vehicle Targets from Remote Sensing Video Satellites" published by authors Kang Jinzhong, Wang Guizhou, etc. in Volume 24, Issue 9 of "Journal of Remote Sensing" in 2020 uses the frame difference method to detect the moving target area, accumulates the moving area to generate the area of ​​interest, and detects the moving vehicle on this basis. This method only considers vehicle targets with strong contrast, and still cannot effectively solve the problem of detecting dim and small vehicles.

[0005] In the tracking stage, the main problem is to solve the problem of inter-frame association of detected targets. Existing tracking methods are severely limited by the accuracy of detection results. When missed detection occurs, there will be problems such as tracking interruption or target switching, including detection interruption when the target just moves out of the image range, or poor detection accuracy within a frame, resulting in target tracking interruption due to failure to successfully detect the target. Tracking information is only used for inter-frame vehicle association, and tracking information and detection information are not effectively combined, making it difficult to achieve a better detection effect for moving vehicle targets. Summary of the invention

[0006] The purpose of the present invention is to provide a satellite video vehicle detection method to solve the problem of poor detection effect of moving vehicle targets caused by detection interruption when the target just moves out of the image range in the tracking method in the prior art.

[0007] In order to solve the above technical problems, the technical solutions and corresponding beneficial effects provided by the present invention are as follows:

[0008] The invention discloses a satellite video vehicle detection method, comprising the following steps: 1) acquiring original satellite video data and performing preprocessing to improve the quality of the satellite video data; 2) performing preliminary vehicle detection on the preprocessed satellite video data frame by frame; 3) performing vehicle tracking using a SORT algorithm according to the preliminary vehicle detection result to complete moving vehicle detection; wherein, if a prediction frame fails to be associated with a detection frame during the application of the SORT algorithm in a certain frame, firstly, the distance between the unsuccessfully associated prediction frame and all unsuccessfully associated detection frames is calculated, and the detection frame is the preliminary vehicle detection result: if the shortest distance is less than a distance threshold, then it is determined that the detection frame and the prediction frame are the same vehicle target, otherwise, it is determined that the vehicle target corresponding to the prediction frame is still a prediction target that fails to be associated; then, a Kalman filter in the SORT algorithm is updated according to the new vehicle target to be applied to vehicle tracking in the next frame.

[0009] The beneficial effects of the above technical solution are: when the detection result accuracy is poor, problems such as tracking interruption or target switching may occur when missed detection occurs. To address this phenomenon, the present invention performs distance-based secondary matching on the prediction frame and the detection frame that have not been successfully associated, eliminating the situation where the intersection-and-union ratio is 0 due to sudden acceleration, deceleration or turning of the vehicle, thereby improving the detection accuracy of moving vehicles. This method focuses on detecting targets that have failed to be associated in the multi-target tracking stage, which can achieve the purpose of focusing on key areas and has good robustness in the detection of moving vehicles in satellite videos.

[0010] Furthermore, in step 3), after determining that the target corresponding to the prediction box is still the prediction target that failed to be associated, it is necessary to open a window at the prediction position for the prediction target that failed to be associated, resample the window and perform binarization processing with different thresholds. If the image obtained after the binarization processing meets the following conditions, it is determined that the prediction box corresponds to the vehicle target:

[0011]

[0012] In the formula, They represent the maximum and minimum coordinate points of the circumscribed rectangle of the connected domain in the window, S u represents the area of ​​the u-th connected domain in the window, represents the aspect ratio of the uth connected domain in the window, x represents the total number of connected domains in the window, C u represents the compactness of the u-th connected domain in the window, and l u represents the side length of the uth connected domain in the window, A represents the area threshold, and A R represents the aspect ratio threshold, and C represents the compactness range.

[0013] The beneficial effects of the above technical solution are as follows: since the current satellite video background contains a large amount of noise and pseudo-moving targets, moving vehicles appear dim and tiny. For predicted targets that fail to be associated, the present invention opens a window with a side length of i at the predicted position to perform multi-threshold binary vehicle detection, so as to effectively improve the detection stability, and then judges the moving vehicle target based on edge connectivity, area, aspect ratio and compactness in the image obtained after binarization, and uses tracking information to focus on detecting dim and tiny moving vehicles, thereby improving the detection effect of low-contrast vehicles.

[0014] Furthermore, the side length of the window is determined by the image resolution of the satellite-borne video data and the actual vehicle size.

[0015] The beneficial effect of the above technical solution is: taking into account the actual situation, the window size is determined by the image resolution of the satellite video data and the actual vehicle size, so as to ensure that the window fully contains the complete vehicle while making the vehicle as large as possible and containing fewer irrelevant targets.

[0016] Furthermore, in step 2), the means for performing preliminary vehicle detection on the preprocessed satellite video data frame by frame is: first, using the trained deep learning model to extract the road mask on the first frame of the satellite video data; and then using the trained moving target detection model to perform preliminary moving vehicle detection in the extracted road mask area.

[0017] The beneficial effects of the above technical solution are: using the road mask can eliminate the noise and pseudo-moving target interference in the background, improve the detection accuracy, and achieve accurate preliminary moving vehicle detection.

[0018] Furthermore, the deep learning model is a D-LinkNet network model.

[0019] The beneficial effects of the above technical solution are: D-LinkNet has achieved a good balance in performance, stability and complexity, and the encoder of D-LinkNet adopts ResNet34, which can effectively solve the problem of image feature information loss during the convolution process.

[0020] Furthermore, the moving target detection model is a KNN network model.

[0021] The beneficial effect of the above technical solution is that the KNN network model combines parameter-free probability density estimation with the KNN classification concept, and can effectively distinguish between background areas and foreground areas.

[0022] Furthermore, the preprocessing in step 1) includes inter-frame image stabilization processing and / or noise removal processing.

[0023] The beneficial effect of the above technical solution is that the above processing can effectively improve the quality of the original satellite video data and lay the foundation for the execution of subsequent steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of the satellite-borne video vehicle detection method of the present invention;

[0025] Figure 2(a) to Figure 2(d) They are the 11th, 13th, 15th, and 17th frames of images in the detection interruption example;

[0026] Figure 3 It is a schematic diagram of the compensation detection process based on tracking of the present invention. DETAILED DESCRIPTION

[0027] The main process of the satellite video vehicle detection method of the present invention is: (1) obtaining the original satellite video data and preprocessing it to improve the quality of the satellite video data. (2) performing preliminary vehicle detection on the preprocessed satellite video data. (3) using the SORT algorithm to track the vehicle based on the preliminary vehicle detection results, and performing secondary precision detection on the failed vehicle targets during the vehicle tracking process to obtain high-precision moving vehicle targets. The secondary precision detection is to focus on the detection of targets that failed to be associated in the multi-target tracking stage to achieve the purpose of focusing on key areas. The specific means is to perform secondary association based on distance between the prediction frame and the detection frame that have not been successfully associated, and eliminate the situation where the intersection-to-union ratio is 0 due to the sudden acceleration, deceleration or turning of the vehicle. Specifically, the distance between the prediction frame that has not been successfully associated and all the detection frames that have not been successfully associated is solved. If the shortest distance meets the distance threshold T l If the detection frame and the prediction frame are the same vehicle target, otherwise it is still a prediction target with failed association. After the vehicle target is detected, it is added to the detection target of the tth frame, and the Kalman filter is updated, so as to complete the detection of all frames in a cycle. In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0028] An embodiment of a satellite-borne video vehicle detection method:

[0029] The specific process of this embodiment is as follows Figure 1 As shown:

[0030] (1) Obtain the original satellite video data and preprocess it to improve the quality of the satellite video data.

[0031] The pre-processing means in this step may be: inter-frame image stabilization and denoising processing, so as to improve the video quality to facilitate the next step of vehicle detection.

[0032] (2) Perform preliminary vehicle detection on the preprocessed satellite video data.

[0033] When performing preliminary vehicle detection in this step, the network can be used to construct a road mask to eliminate background noise and interference from pseudo-moving targets and improve detection accuracy. Therefore, the specific means of this step are: first, use the trained deep learning model to extract the road mask on the first frame of the satellite video data to construct the constraints for vehicle detection; then use the trained moving target detection model to perform preliminary moving vehicle detection in the extracted road mask area.

[0034] In the past decade, domestic and foreign scholars have proposed a large number of methods for road extraction from satellite remote sensing images. With the rapid development of deep learning, road extraction using the network has become the mainstream algorithm at this stage. Among them, D-LinkNet has achieved a good balance in performance, stability and complexity. It fully considers narrowness, connectivity, complexity and long span. By using pooling layers and jump connections, the receptive field of feature points in the central part of the network is increased, and the connection method of the central area can be regarded as a parallel mode. In addition, the encoder of D-LinkNet uses ResNet34, which can effectively solve the problem of image feature information loss in the convolution process. Each multi-channel feature mapping layer is represented by a rectangular block. Therefore, in this embodiment, D-LinkNet is selected as a deep learning model for road mask extraction. In the loss function construction part, since the DICE coefficient loss (DICE) is not affected by a large number of background pixels, the present invention adopts a binary cross entropy (BCE) combined with the DICE coefficient loss to improve the qualitative model training.

[0035] After the mask is constructed, moving vehicles are detected in the constrained area. The four most common moving target detection algorithms are compared, as shown in Table 1. Finally, the KNN moving target detection algorithm is selected as the moving target detection model. This algorithm model shows the best effect in the stage of moving vehicle detection in satellite video. It combines parameter-free probability density estimation with KNN classification ideas. KNN moving target detection integrates background segmentation algorithm into machine learning to improve classification effect. The KNN algorithm can learn each frame of the image and effectively distinguish the background area from the foreground area. Compared with the traditional segmentation algorithm, the KNN background segmentor is more robust and has better segmentation effect.

[0036] Table 1

[0037]

[0038] (3) The SORT algorithm is used to track the preliminary vehicle detection results to complete the moving vehicle detection.

[0039] The continuity of most tracking algorithms in satellite-borne video is severely limited by the detection accuracy. Influenced by this, the present invention takes the opposite approach and hopes to detect through information reverse compensation in the tracking stage, that is, to focus on the detection of targets that failed to be associated in the multi-target tracking stage, so as to achieve the purpose of focusing on the detection of key areas.

[0040] Specifically, the SORT algorithm is used to associate vehicles between frames. Because the vehicles in the satellite video frames are all moving slowly and have few features, the SORT algorithm uses IOU to calculate the similarity of targets between frames, which is simple and effective. The Kalman filter is used to define the moving vehicle as a vector with position and velocity, and the covariance matrix is ​​used to infer the state of the next frame from the current frame of the target, that is, the trajectory position of the moving vehicle. The Kalman filter prediction equation is as follows:

[0041]

[0042] In the formula, represents the state quantity of the target k-th frame, A is the state transfer matrix, P is the state covariance matrix, and Q is the process noise covariance matrix. After associating the detection result of the next frame, the target state is updated. The update equation at this time is shown in formula (2):

[0043]

[0044] In the formula, Z k is the state value observation vector, H is the state observation matrix, R is the measurement noise covariance matrix, K is the Kalman gain, and I is the unit matrix.

[0045] After using Kalman filtering to obtain the predicted positions of n vehicles in the tth frame, data is associated with the m vehicles detected in the tth frame. The data association matrix can be obtained as follows:

[0046]

[0047] Where C is the data association matrix; c mn It represents the correlation between the mth detection box and the nth prediction box. The correlation is measured by the intersection-over-union (IOU) ratio between the two boxes. Then the Hungarian algorithm finds the optimal solution for the data correlation matrix and completes the correlation of moving vehicles between frames.

[0048] Obviously, when n is greater than m, some targets will be interrupted during the tracking phase. Ideally, this phenomenon should occur when the target just moves out of the image range, such as Figure 2(a) to Figure 2(d) The detection of vehicle A in the 17th frame is interrupted; however, if the target tracking is interrupted due to poor detection accuracy in this frame and the target is not successfully detected, it is unacceptable. Figure 2(a) to Figure 2(d) Vehicle B in the figure is interrupted due to missed detection in the 13th and 15th frames. To this end, the present invention proposes to focus on detecting the target area where data association fails to be performed to compensate for the poor detection effect in the first stage. The process is as follows: Figure 3 shown.

[0049] First, the unmatched prediction frames and detection frames are matched again based on distance to eliminate the situation where the intersection-over-union ratio is 0 due to the sudden acceleration, deceleration or turning of the vehicle. Specifically, the distance between the unmatched prediction frame and all unmatched detection frames is solved. If the shortest distance meets the distance threshold T l Then the detection box and the prediction box are considered to be the same target, otherwise it is still a predicted target with failed association. The discriminant is shown in formula (4):

[0050]

[0051] Among them, (x t ,y t ) represents the coordinates of the center point of the tth unassociated prediction box, (x i ,y i ) represents the coordinates of the center point of the i-th unassociated detection frame, p and q represent the total number of unassociated prediction frames and detection frames, respectively. It should be noted that the detection frame here corresponds to the preliminary vehicle detection result obtained in step (2).

[0052] Then, for the predicted targets that failed to be associated, a window with a side length of i is opened at the predicted position for multi-threshold binary vehicle detection. Statistics show that the vehicles that failed the first detection are mostly small in size and dim in color, usually occupying only 8-16 pixels. The window side length i can be determined comprehensively based on the image resolution and the actual vehicle size. While fully ensuring that the window contains the complete vehicle, the vehicle should be as large as possible and contain fewer irrelevant targets. After determining the window, the window is resampled to 1024×1024 pixels, and the binarization processing with thresholds of th1, th2, th3, th4, and th5 is performed in the window. Each binarization process needs to be processed, but for the same target, it needs to be processed according to different threshold sizes. When the large threshold meets the conditions, the subsequent thresholds do not need to be verified again, and the target can be considered to meet the requirements. Multi-threshold processing can adapt to the different grayscale differences between vehicles and roads in different regions, different shooting times, and different weather conditions, and effectively improve the detection stability. Then, in multiple binary images, the edge connectivity, area (A), aspect ratio (A R ) and compactness (C) to determine the vehicle target. The specific judgment formula is as follows:

[0053]

[0054] In the formula, They represent the maximum and minimum coordinate points of the circumscribed rectangle of the connected domain, S u represents the area of ​​the u-th connected domain in the window, represents the aspect ratio of the u-th connected domain in the window, C urepresents the compactness of the uth connected domain in the window, and x represents the total number of connected domains in the window. If formula (5) is satisfied, the target in the target box can be considered to be a moving vehicle. u The calculation formula is as follows:

[0055]

[0056] In the formula, l u Represents the edge length of the u-th connected domain.

[0057] After the vehicle target is detected, it is added to the detection target of the tth frame and the Kalman filter is updated. This cycle completes the detection of all frames.

[0058] In summary, in view of the problems of complex background, serious interference of noise and pseudo-moving targets, and weak and small moving vehicle targets that are difficult to detect in satellite-borne videos, the present invention proposes a satellite-borne video small vehicle detection method that combines road information and tracking information. Compared with the traditional method, the present invention uses a road mask to eliminate the interference of noise and pseudo-moving targets in the background, and uses tracking information to focus on the detection of weak and small moving vehicles, which improves the detection effect of low-contrast vehicles and effectively enhances the tracking continuity. In the experimental part, the data set was re-annotated, and 2120 moving vehicle targets were added to make it closer to the actual situation, providing an important real data set for the study of satellite-borne vehicle detection. In the vehicle detection and tracking experiment, the method of the present invention reduced the phenomenon of missed detection and false detection in the detection stage, and the detection accuracy and recall rate were improved; the accuracy and identity maintenance ability in the tracking stage were improved, which verified the effectiveness of the method. Therefore, the method of the present invention has good robustness in the detection of moving vehicles in satellite videos.

Claims

1. A satellite-based video vehicle detection method, characterized in that: The steps include: 1) Obtain the original satellite video data and perform preprocessing to improve the quality of the satellite video data; 2) Perform preliminary vehicle detection on the pre-processed satellite video data frame by frame; 3) Based on the preliminary vehicle detection results, the SORT algorithm is used to track the vehicle to complete the moving vehicle detection; Among them, if the prediction box and the detection box fail to be associated during the application of the SORT algorithm in a certain frame, the distance between the unsuccessfully associated prediction box and all unsuccessfully associated detection boxes is first calculated, and the detection box is the preliminary vehicle detection result: if the shortest distance is less than the distance threshold, it is determined that the detection box and the prediction box are the same vehicle target, otherwise it is determined that the vehicle target corresponding to the prediction box is still the prediction target that failed to be associated; then the Kalman filter in the SORT algorithm is updated according to the new vehicle target to be applied to vehicle tracking in the next frame.

2. The satellite-borne video vehicle detection method according to claim 1, characterized in that: In step 3), after determining that the target corresponding to the prediction box is still the prediction target that failed to be associated, it is necessary to open a window at the prediction position for the prediction target that failed to be associated, resample the window and perform binarization processing with different thresholds. If the image obtained after the binarization processing meets the following conditions, it is determined that the prediction box corresponds to the vehicle target: In the formula, They represent the maximum and minimum coordinate points of the circumscribed rectangle of the connected domain in the window, S u represents the area of ​​the u-th connected domain in the window, represents the aspect ratio of the uth connected domain in the window, x represents the total number of connected domains in the window, C u represents the compactness of the u-th connected domain in the window, and l u represents the side length of the u-th connected domain, A represents the area threshold, and A R represents the aspect ratio threshold, and C represents the compactness range.

3. The satellite-borne video vehicle detection method according to claim 2, characterized in that: The side length of the window is determined by the image resolution of the satellite video data and the actual vehicle size.

4. The satellite-borne video vehicle detection method according to claim 1, characterized in that: The means of performing preliminary vehicle detection on the preprocessed satellite video data frame by frame in step 2) is: firstly, using the trained deep learning model to extract a road mask on the first frame of the satellite video data; The trained moving object detection model is then used to perform preliminary moving vehicle detection in the extracted road mask area.

5. The satellite-borne video vehicle detection method according to claim 4, characterized in that: The deep learning model is a D-LinkNet network model.

6. The satellite-borne video vehicle detection method according to claim 4, characterized in that: The moving target detection model is a KNN network model.

7. The satellite-borne video vehicle detection method according to claim 1, characterized in that: The preprocessing in step 1) includes inter-frame image stabilization processing and / or noise removal processing.

Images