Near real-time target detection and tracking methods for satellite remote sensing imagery

Abstract

This invention relates to a near real-time target detection and tracking method for satellite remote sensing imagery, comprising: inputting consecutive frames of satellite remote sensing imagery into a target detector for target detection to obtain the actual position of the target in the current frame; inputting the actual position of the target in the current frame into a Kalman filter to predict the target's position range in the next frame; matching the actual position of the same target in the current frame with the predicted position range in the next frame, updating the Kalman filter, and outputting the matching result; generating the target motion trajectory based on the matching result, predicting the target's motion region in the next frame, and inputting it into the target detector to obtain the target's actual position in the next frame. This invention can synchronously and dynamically update the parameters associated with target detection and tracking, improving the accuracy of the target detection and tracking association.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection and tracking, and particularly relates to a near-real-time target detection and tracking method for satellite remote sensing images. BACKGROUND

[0002] Target tracking is a technique that uses the context information of a video or image sequence to model the appearance and motion information of a target, thereby predicting the motion state of the target and locating the target position. It is an important basic problem in computer vision and has important theoretical research significance and application value. It has wide application in intelligent video monitoring systems, intelligent human-computer interaction, intelligent transportation, and visual navigation systems. In recent years, with the development of remote sensing earth observation technology, such as video satellite technology and unmanned aerial vehicle technology, target tracking technology based on remote sensing video has broad application prospects in intelligent traffic management, video monitoring, and other fields.

[0003] With the continuous improvement of people's safety and efficiency requirements, the complexity of computer vision tasks is continuously increasing, and the effective application of visual tracking technology in various industries also puts forward the demand for multi-target tracking. For example, video monitoring and automatic driving have been put into product use. These images usually contain the activity trajectories of multiple targets. Neither single-frame detection of frame sequences nor single-target tracking (SOT) for all targets in the frame can meet the task requirements. Because target tracking needs to establish the connection between the target in the previous image and the target in the next image, using a detection algorithm in a single frame can only achieve recognition but cannot establish the connection between the previous and the next. Using a single-target tracking algorithm between adjacent images can only achieve the connection of a single target but cannot achieve recognition. The replacement of new and old targets and identity recognition are major difficulties that multi-object tracking (MOT) needs to solve. The recognition method and the association strategy affect the performance of the MOT tracking algorithm.

[0004] Satellite single-band image data has low resolution and low frame rate, the target features are not obvious, and it is greatly affected by clouds. The high false alarm rate of target detection will greatly affect the subsequent target tracking association. The most widely used target detection framework Deep SORT in the industry adopts the method of detecting first and then tracking. However, the detector and the target tracking association module are relatively independent, and the setting and accuracy of the detector greatly affect the correctness and accuracy of the subsequent target tracking association. At the same time, the detector and the tracking module are in a series structure, and the parameters and association results cannot be effectively utilized. In the overall implementation process, the false alarms detected cannot be effectively removed. SUMMARY

[0005] To solve the technical problems existing in the prior art, the present application aims to provide a near real-time target detection and tracking method for satellite remote sensing images, which can dynamically update parameters related to target detection and tracking in synchronization and improve the accuracy of target detection and tracking.

[0006] To achieve the above-mentioned application purposes, the technical scheme of the present application is as follows:

[0007] The present application provides a near real-time target detection and tracking method for satellite remote sensing images, which comprises:

[0008] The continuous frames of satellite remote sensing images are input into a target detector frame by frame for target detection, so as to obtain the actual position of the target in the current frame.

[0009] The actual position of the target in the current frame is input into a Kalman filter, so as to predict the position range of the target in the next frame.

[0010] The actual position of the same target in the current frame and the predicted position range of the target in the next frame are matched, so as to update the Kalman filter and output the matching result.

[0011] According to the matching result, the motion trajectory of the target is generated, the motion area of the target in the next frame is predicted, and the target detector is input, so as to obtain the actual position of the target in the next frame.

[0012] According to one aspect of the present application, each frame of the satellite remote sensing images contains a plurality of different targets.

[0013] According to one aspect of the present application, the continuous frames of satellite remote sensing images are input into a target detector frame by frame for target detection, so as to obtain the actual position of the target in the current frame, which comprises:

[0014] The continuous frames of satellite remote sensing images are input into a target detector frame by frame, and a plurality of targets of the satellite remote sensing image of the current frame are detected, so as to output the predicted position and the confidence degree corresponding to each target, and the predicted position corresponding to the highest confidence degree is the actual position of each target in the current frame.

[0015] According to one aspect of the present application, the actual position of the target in the current frame is input into a Kalman filter, so as to predict the position range of the target in the next frame, which comprises:

[0016] The Kalman filter is initialized, and a tracker is allocated to each target detector.

[0017] The actual positions of a plurality of targets in the current frame are input into the Kalman filter, and the motion trajectories of the plurality of targets are tracked by using the trackers respectively, so as to output the predicted position ranges of the plurality of targets in the next frame.

[0018] According to one aspect of the present application, in the process of matching the actual position of the same target in the current frame and the predicted position range of the next frame, the following is included:

[0019] The confidence of the target is used as the basis for classification matching calculation. If the confidence is greater than 0.7, the actual position of the same target in the current frame and the predicted position range of the next frame are matched in cascade; if the confidence is less than or equal to 0.7, the actual position of the same target in the current frame and the predicted position range of the next frame are matched in IOU.

[0020] According to one aspect of the present application, the cascade matching uses the cosine distance and Mahalanobis distance of the target appearance as the cost matrix, and introduces a loss function Margin Loss to increase the matching accuracy.

[0021] According to one aspect of the present application, the IOU matching is realized by using the Hungarian algorithm.

[0022] According to one aspect of the present application, the Kalman filter is updated, and the matching result is output, including:

[0023] Based on the error of the predicted position range and the actual position of multiple targets in the next frame, the Kalman filter and the tracker are iteratively updated until the motion trajectory of the multiple targets and the predicted position range in the next frame are respectively closer to the actual motion trajectory and the predicted position.

[0024] The predicted position range and the motion trajectory ID of the target in the next frame are output for the matching successful target, and the motion trajectory of the matching failed target is deleted.

[0025] According to one aspect of the present application, the target motion trajectory is generated according to the matching result, the motion area of the target in the next frame is predicted, and the target detector is input, so as to obtain the actual position of the target in the next frame, including:

[0026] The motion direction and the motion speed of the target are generated by trajectory association according to the position coordinates and the motion trajectory ID of the matching successful target, and the predicted motion area of the target in the next frame is output;

[0027] The motion trajectory ID and the corresponding predicted motion area are weighted, the satellite remote sensing image of the current frame is segmented, and the target detector is input according to the motion area weight for detection, so as to generate the predicted position and the confidence of the target in the next frame;

[0028] The predicted position and the predicted motion area of the target in the next frame are matched, and the confidence is updated based on the matching result until the actual position of the target in the next frame is obtained.

[0029] According to one aspect of the present application, the predicted position of the target in the next frame and the predicted motion region are matched, the confidence is updated based on the matching result, including:

[0030] If the predicted position of the target in the next frame is in the predicted motion region, the confidence is updated by weighting according to the following formula:

[0031] P=N*P0

[0032] Wherein, P0 is the confidence output by the target detector, and N is the weight of the predicted motion region.

[0033] Compared with the prior art, the present application has the following beneficial effects:

[0034] According to the scheme of the present application, an improved near real-time target detection and tracking association method is provided, which can dynamically update the parameters of target detection and tracking association in synchronization, improve the accuracy of target detection and tracking association, and thus improve the utilization rate of satellite remote sensing image data. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0036] Figure 1 An implementation flowchart of a near real-time target detection and tracking method for satellite remote sensing images disclosed by the embodiments of the present application is schematically shown;

[0037] Figure 2 A schematic diagram of a target motion trajectory and a predicted motion region disclosed by the embodiments of the present application is schematically shown;

[0038] Figure 3 A schematic diagram of a predicted motion region disclosed by the embodiments of the present application is schematically shown. DETAILED DESCRIPTION

[0039] The description of the embodiments of the present application should be combined with the corresponding drawings, and the drawings should be regarded as a part of the complete description. In the drawings, the shape or thickness of the embodiments can be enlarged, and the illustration is simplified or facilitated. Furthermore, the parts of the structures in the drawings will be described separately, and it should be noted that the elements not shown in the drawings or not described by words are in the form known by those skilled in the art.

[0040] The description of the embodiments herein, any reference to direction or orientation is merely intended for convenience of description and is not intended in any way to limit the scope of the present application. The following description of the preferred embodiments will be made with reference to the accompanying drawings, in which features, which can occur in various combinations and permutations, are provided, and the application is not limited to the preferred embodiments. The scope of the application is defined by the claims.

[0041] As shown in Figure 1 The embodiment discloses a near real-time target detection and tracking method for satellite remote sensing images, and the specific implementation process includes the following steps:

[0042] Step 100, input the continuous frames of satellite remote sensing images into the target detector frame by frame for target detection to obtain the actual position of the target in the current frame.

[0043] Each frame of the satellite remote sensing image contains a plurality of different targets. The method simultaneously detects and tracks a plurality of targets on the continuous frames of satellite remote sensing images.

[0044] In one embodiment, the specific implementation process of step 100, which inputs the continuous frames of satellite remote sensing images into the target detector frame by frame for target detection to obtain the actual position of the target in the current frame, includes: inputting the continuous frames of satellite remote sensing images into the target detector frame by frame, detecting a plurality of targets of the satellite remote sensing image of the current frame, outputting the predicted position and the confidence degree corresponding to each target, and the predicted position corresponding to the highest confidence degree being the actual position of each target in the current frame.

[0045] Step 200, input the actual position of the target in the current frame into the Kalman filter to predict the position range of the target in the next frame.

[0046] In one embodiment, the specific implementation process of step 200, which inputs the actual position of the target in the current frame into the Kalman filter to predict the position range of the target in the next frame, includes: initializing the Kalman filter, and assigning a tracker to each target detector; inputting the actual positions of a plurality of targets in the current frame into the Kalman filter, tracking the motion trajectories of the plurality of targets respectively by using the trackers, and outputting the predicted position ranges of the plurality of targets in the next frame.

[0047] Step 300, match the actual position of the same target in the current frame and the predicted position range of the next frame, update the Kalman filter, and output the matching result.

[0048] In one embodiment, in the specific implementation process of matching the actual position of the same target in the current frame and the predicted position range of the next frame in step 300, the confidence of the target is taken as the basis for calculating the classification matching, if the confidence is greater than 0.7, the actual position of the same target in the current frame and the predicted position range of the next frame are matched in cascade; if the confidence is less than or equal to 0.7, the actual position of the same target in the current frame and the predicted position range of the next frame are matched in IOU.

[0049] Specifically, the cosine distance and Mahalanobis distance of the target appearance are taken as the cost matrix in the cascade matching, and the loss function Margin Loss is introduced to increase the matching accuracy. The IOU matching is realized by using the Hungarian algorithm.

[0050] In one embodiment, in the specific implementation process of updating the Kalman filter and outputting the matching result in step 300, the following steps are included: based on the error of the predicted position range and the actual position of the plurality of targets in the next frame, the Kalman filter and the tracker are iteratively updated until the motion trajectory of the plurality of targets and the predicted position range of the next frame are respectively closer to the actual motion trajectory and the predicted position; the corresponding predicted position range of the next frame and the motion trajectory ID of the target are output for the target whose matching is successful, and the motion trajectory of the target whose matching is failed is deleted.

[0051] In step 400, the target motion trajectory is generated according to the matching result, the motion area of the target in the next frame is predicted, and the target detector is input to obtain the actual position of the target in the next frame.

[0052] In one embodiment, in the specific implementation process of generating the target motion trajectory according to the matching result, predicting the motion area of the target in the next frame, and inputting the target detector to obtain the actual position of the target in the next frame in step 400, the following steps are included: the motion direction and the motion speed of the target are generated according to the position coordinates and the motion trajectory ID of the target whose matching is successful, and the predicted motion area of the target in the next frame is output; the motion trajectory ID and the corresponding predicted motion area are weighted, the satellite remote sensing image of the current frame is segmented, and the target detector is input for detection according to the weight of the motion area, so as to generate the predicted position and the confidence of the target in the next frame; the predicted position and the predicted motion area of the target in the next frame are matched, the confidence is updated based on the matching result, and the actual position of the target in the next frame is obtained.

[0053] Specifically, the time before and after relationship of the point coordinates is determined according to the generated motion trajectory ID, and the motion trajectory curve of the target is expressed by using a high-order polynomial:

[0054] f(x)=p n (x)=a0+a1(x-x0)+a2(x-x0)2 +…+a n (x-x0) n

[0055] Using the least square method to express the error, the error function is:

[0056]

[0057] Using Gaussian elimination method to optimize the Loss function, the target motion trajectory curve is fitted.

[0058] As shown in Figure 2 , due to the error caused by the resolution size of the image, satellite jitter, and the target speed change between the continuous frames of the image, the actual motion area of the target will have errors. Here, the target motion direction is regarded as the normal line, and the-45 degree to 45 degree of the target motion direction is designed as area 1, the-90 degree to-45 degree as area 2, and the 45 degree to 90 degree as area 3. The target motion direction is a vector (α, β), and the error caused by external reasons is (x', y'). The position coordinates of the target in the next frame are calculated as (x1, y1) using the expression of the target motion trajectory. The target motion speed is the slope of the target motion trajectory curve at this point, denoted as S. The function expression of the target moving direction is y=kx+b.

[0059] The target activity area is a semicircle of-180 degrees to 180 degrees, with the center coordinates of (x1+ax', y1+by'), and the radius of S.

[0060] According to the expression of the circle, the target activity range is:

[0061]

[0062] Wherein, R=S, D is the perpendicular line of the target moving direction, then

[0063] As shown in Figure 3 , the expression of area 1 is: The expression of area 2 is: The expression of area 3 is: Wherein, D1 and D'1 are the straight lines obtained by counterclockwise rotating 45 degrees and clockwise rotating 45 degrees of the straight line of the target moving direction respectively, and their expressions are:

[0064]

[0065]

[0066] Wherein, x1 is the coordinate of the last position of the target on the motion trajectory.

[0067] Region 1, region 2, region 3 are weighted. The weight of region 1 is set to 1.5, the weight of region 2 and region 3 is set to 1.2, and the weight of other regions is set to 1.

[0068] The specific implementation process of matching the predicted position of the target in the next frame and the predicted motion region and updating the confidence based on the matching result in the above step comprises: judging whether the predicted position of the target in the next frame is in the predicted motion region, and if yes, updating the confidence according to the following formula:

[0069] P=N*P0

[0070] Wherein, P0 is the confidence output by the target detector, and N is the weight of the predicted motion region.

[0071] In the embodiment, the method is an improved near real-time target detection and tracking association method, which can dynamically update the parameters of target detection and tracking association in synchronization, improve the accuracy of target detection and tracking association, and thus improve the utilization rate of satellite remote sensing image data.

[0072] The serial numbers of the above steps involved in the method of the present application do not mean the order of the execution of the method, and the execution order of each step should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0073] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A near real-time target detection and tracking method for satellite remote sensing images, comprising: inputting each frame of continuous frames of satellite remote sensing images into a target detector for target detection to obtain an actual position of a target in a current frame; inputting the actual position of the target in the current frame into a Kalman filter to predict a position range of the target in a next frame; matching the actual position of the same target in the current frame and the predicted position range of the target in the next frame to update the Kalman filter and output a matching result; generating a target motion trajectory according to the matching result, predicting a motion area of the target in the next frame, and inputting the motion area into the target detector to obtain an actual position of the target in the next frame, comprising: performing trajectory association according to position coordinates and a motion trajectory ID of a target with a successful match to generate a motion direction and a motion speed of the target, outputting a predicted motion area of the target in the next frame; weighting the motion trajectory ID and the corresponding predicted motion area, segmenting the satellite remote sensing image of the current frame, inputting the satellite remote sensing image into the target detector for detection according to the motion area weight, and generating a predicted position and a confidence of the target in the next frame; and matching the predicted position of the target in the next frame and the predicted motion area, and updating the confidence based on the matching result until an actual position of the target in the next frame is obtained; wherein matching the predicted position of the target in the next frame and the predicted motion area and updating the confidence based on the matching result comprises: determining whether the predicted position of the target in the next frame is within the predicted motion area, and if so, weighting and updating the confidence according to the following formula: P = N * P0 wherein P0 is the confidence output by the target detector, and N is the weight of the predicted motion area.

2. The method of claim 1, wherein, Each frame of the satellite remote sensing images contains multiple different targets.

3. The method of claim 2, wherein, Inputting each frame of continuous frames of satellite remote sensing images into a target detector for target detection to obtain an actual position of a target in a current frame comprises: inputting each frame of continuous frames of satellite remote sensing images into a target detector to detect multiple targets of the satellite remote sensing image of the current frame, outputting a predicted position and a confidence corresponding to each target, and taking the predicted position corresponding to the highest confidence as the actual position of each target in the current frame.

4. The method of claim 3, wherein, Inputting the actual position of the target in the current frame into a Kalman filter to predict a position range of the target in a next frame comprises: initializing the Kalman filter and assigning a tracker to each target detector; inputting the actual positions of multiple targets in the current frame into the Kalman filter to track motion trajectories of the multiple targets respectively using the trackers and output a predicted position range of the multiple targets in the next frame.

5. The method of claim 4, wherein, In the process of matching the actual position of the same target in the current frame and the predicted position range of the target in the next frame, comprising: taking the confidence of the target as a basis for classification and matching calculation, and if the confidence is greater than 0.7, performing cascade matching on the actual position of the same target in the current frame and the predicted position range of the target in the next frame; and if the confidence is less than or equal to 0.7, performing IOU matching on the actual position of the same target in the current frame and the predicted position range of the target in the next frame.

6. The method of claim 5, wherein, The cascade matching uses cosine distance and Mahalanobis distance of target appearance as cost matrix, and introduces a loss function Margin Loss to increase matching accuracy.

7. The method of claim 5, wherein, The IOU matching uses Hungarian algorithm to realize.

8. The method of claim 4, wherein, Updating the Kalman filter, and outputting matching results, including: Based on the error between the predicted position range and the actual position of the plurality of targets in the next frame, iteratively updating the Kalman filter and the tracker until the motion trajectory of the plurality of targets and the predicted position range corresponding to the next frame are respectively closer to the actual motion trajectory and the predicted position; Outputting the predicted position range and the motion trajectory ID of the next frame corresponding to the matching successful target, and deleting the motion trajectory of the matching failed target.

Images