Remote sensing video multi-target tracking method and device

Abstract

The application provides a remote sensing video multi-target tracking method and device, and relates to the field of image processing. In the method, a plurality of image frames of a remote sensing video are acquired, and candidate image frames containing target objects are extracted; a detection position and attribute information of the target objects in the candidate image frames are detected; a pre-established trajectory set is acquired; a position of a historical object in the candidate image frames is predicted based on historical motion information; a position similarity of the detection position and the predicted position and an attribute similarity of a current attribute and a historical attribute are calculated, and a comprehensive similarity is obtained through fusion; the target object is cascadedly matched with a plurality of historical objects based on the comprehensive similarity and a loss count value; after successful matching, the attribute information and the motion information of the corresponding historical object are updated by using the detection result, and a final tracking trajectory of the target object is obtained according to the updated historical trajectory.

Description

Technical Field

[0001] This application relates to the fields of remote sensing image processing, computer vision, and target tracking technology, and more specifically, to a method and apparatus for multi-target tracking in remote sensing video. Background Technology

[0002] The main purpose of multi-target tracking is to first detect the target of interest in a given video or image sequence. The number of targets is usually multiple, and the identity (ID) of these multiple targets is maintained in subsequent frames, and their motion trajectories are recorded.

[0003] Currently, multi-target tracking methods for remote sensing videos typically employ a detection-tracking separation framework. This involves first obtaining the target position in each frame using a target detector, and then matching the detection results with historical trajectories using a data association algorithm. However, this separation framework suffers from the following problems in remote sensing scenarios:

[0004] (1) Remote sensing targets are characterized by small size and similar appearance. Existing methods rely on only a single appearance feature or motion feature to calculate similarity, which is difficult to fully express the feature information of remote sensing targets and is prone to identity jump and mismatch.

[0005] (2) In remote sensing videos, there are complex backgrounds and frequent occlusions. When the target disappears for a long time and then reappears, the uncertainty of the Kalman filter prediction increases, resulting in a low accuracy of re-association after occlusion.

[0006] (3) Existing multi-feature fusion methods mostly use simple feature splicing or weighted summation, lacking feature selection and fusion strategies tailored to the characteristics of remote sensing targets. Summary of the Invention

[0007] In view of the above problems, this application provides a method and apparatus for multi-target tracking in remote sensing video.

[0008] This application provides a method for multi-target tracking in remote sensing video, comprising: acquiring multiple image frames from the remote sensing video and extracting multiple candidate image frames containing target objects; detecting the object detection result of the target object contained in each candidate image frame, the object detection result including detection position and current attribute information; acquiring a pre-established trajectory set containing multiple historical tracking trajectories, each historical tracking trajectory representing the motion trajectory of a tracked historical object in a historical time period, the trajectory set storing the loss count value of each historical tracking trajectory, the historical motion information and historical attribute information of the corresponding historical object, the loss count value representing the number of consecutive candidate image frames in which a historical tracking trajectory has not been successfully matched before any candidate image frame. Based on historical motion information, predict the predicted position of the historical object corresponding to the historical motion information in the candidate image frame; calculate the positional similarity between the detected position and the predicted position, and calculate the attribute similarity between the current attribute information and the historical attribute information, and fuse the positional similarity and attribute similarity to obtain the comprehensive similarity; based on the comprehensive similarity and the loss count value, perform cascade matching between the target object and multiple historical objects; if the target object is successfully matched with any historical object, update the historical attribute information and historical motion information corresponding to any historical object based on the object detection result of the target object, and obtain the target tracking trajectory of the target object in the remote sensing video based on the updated historical tracking trajectory of any historical object.

[0009] According to an embodiment of this application, the trajectory set is established in the following manner: for candidate image frames in which any historical object appears for the first time among multiple candidate image frames, a Kalman filter is initialized based on the detection position corresponding to the candidate image frame, and a tracker is assigned to the historical object. The tracker is used to store the historical attribute information of the historical object; based on the tracker and the Kalman filter, the historical tracking trajectory of the historical object is obtained; and based on the historical tracking trajectory of each historical object, a trajectory set is established.

[0010] According to an embodiment of this application, predicting the predicted position of a historical object in a candidate image frame based on historical motion information includes: inputting the historical motion information of the historical object into a corresponding Kalman filter; and predicting the predicted position of the historical object in the candidate image frame through the state transition equation of the Kalman filter.

[0011] According to embodiments of this application, calculating the attribute similarity between current attribute information and historical attribute information includes: extracting current depth features, current texture features, and current color features from the current attribute information, and extracting historical depth features, historical texture features, and historical color features from the historical attribute information; calculating the depth feature distance between the current depth feature and the historical depth feature, the texture feature distance between the current texture feature and the historical texture feature, and the color feature distance between the current color feature and the historical color feature, respectively; and fusing the depth feature distance, texture feature distance, and color feature distance to obtain the attribute similarity.

[0012] According to an embodiment of this application, when a target object is successfully matched with any historical object, the historical attribute information and historical motion information corresponding to any historical object are updated based on the object detection result of the target object. This includes: updating the parameters of the Kalman filter corresponding to any historical object based on the detection position of the target object to obtain updated historical motion information; and updating the historical attribute information stored in the tracker corresponding to any historical object based on the current attribute information of the target object to obtain updated historical attribute information.

[0013] According to an embodiment of this application, a target object is cascaded with multiple historical objects based on a comprehensive similarity and a loss count value, including: sorting the multiple historical objects in ascending order of loss count values; and matching the target object with each historical object sequentially based on the comprehensive similarity in the sorted order.

[0014] According to an embodiment of this application, if the target object is not successfully matched, the method further includes: initializing a new Kalman filter based on the detection position of the target object, and assigning a new tracker to the target object; obtaining the target tracking trajectory of the target object based on the new tracker and the new Kalman filter, and storing the current attribute information corresponding to the target object into the new tracker.

[0015] According to an embodiment of this application, when the target object fails to match any historical object, the method further includes: for any historical tracking trajectory corresponding to a historical object that has not matched the target object in any candidate image frame, increasing the loss count value of the historical tracking trajectory; if the loss count value is less than a preset counting threshold, retaining the corresponding historical tracking trajectory and continuing to match the corresponding historical tracking trajectory with the target object in the next candidate image frame; before the loss count value reaches the counting threshold, if any historical object is rematched with the object detection result of the target object, resetting the loss count value of the historical tracking trajectory corresponding to any historical object to zero, and updating the historical tracking trajectory corresponding to any historical object based on the object detection result of the rematched target object; if the loss count value is greater than or equal to the counting threshold, deleting the corresponding historical tracking trajectory from the trajectory set.

[0016] According to an embodiment of this application, an unmatched historical tracking trajectory has a trajectory state, which includes an unconfirmed state and a confirmed state. The method further includes: for an unconfirmed historical tracking trajectory, if the historical tracking trajectory is successfully matched in a consecutive preset number of candidate image frames, updating the trajectory state of the historical tracking trajectory to a confirmed state; if the historical tracking trajectory is not successfully matched in a number of candidate image frames, deleting the historical tracking trajectory from the trajectory set.

[0017] This application, in another aspect, provides a remote sensing video multi-target tracking device, comprising: an image acquisition module for acquiring multiple image frames in a remote sensing video and extracting multiple candidate image frames containing target objects from the multiple image frames; an image detection module for detecting the object detection result of the target object contained in each candidate image frame, the object detection result including detection position and current attribute information; a set acquisition module for acquiring a pre-established set of trajectories containing multiple historical tracking trajectories, each historical tracking trajectory representing the motion trajectory of a tracked historical object in a historical time period, the trajectory set storing a loss count value for each historical tracking trajectory, the historical motion information and historical attribute information of the corresponding historical object, the loss count value representing the number of consecutive image frames in which a historical tracking trajectory has not been successfully matched before any candidate image frame; and a position pre-... The system comprises the following modules: a detection module, which predicts the position of a historical object in a candidate image frame based on historical motion information; a similarity calculation module, which calculates the positional similarity between the detected position and the predicted position, and the attribute similarity between the current attribute information and the historical attribute information, and fuses the positional similarity and attribute similarity to obtain a comprehensive similarity; an object matching module, which performs cascaded matching between the target object and multiple historical objects based on the comprehensive similarity and the loss count; and a trajectory tracking module, which, when the target object successfully matches any historical object, updates the historical attribute information and historical motion information corresponding to any historical object based on the object detection result of the target object, and obtains the target tracking trajectory of the target object in the remote sensing video based on the updated historical tracking trajectory of any historical object.

[0018] The remote sensing video multi-target tracking method and apparatus provided in this application can achieve the following beneficial effects:

[0019] (1) By extracting candidate image frames containing the target object for processing, the invalid calculation without target frames is avoided, thus improving the processing efficiency;

[0020] (2) By fusing position similarity and attribute similarity, the target characteristics are more comprehensively characterized, so that multiple targets can maintain stable matching performance under complex background, lighting changes and posture changes.

[0021] (3) By performing cascade matching according to the order of the lost count value from small to large, reliable trajectories with short disappearance time are given priority to matching opportunities, avoiding mismatches caused by inaccurate prediction after the target is occluded for a long time, reducing the number of target identity jumps, and improving the continuity and stability of tracking. Attached Figure Description

[0022] The above and other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0023] Figure 1 A flowchart illustrating a remote sensing video multi-target tracking method according to an embodiment of this application is shown schematically.

[0024] Figure 2 A flowchart illustrating attribute similarity calculation according to an embodiment of this application is shown schematically;

[0025] Figure 3 A schematic comparison diagram of the original image and the texture feature image according to an embodiment of this application is shown;

[0026] Figure 4 A flowchart illustrating the matching of target objects and historical objects according to an embodiment of this application is shown schematically.

[0027] Figure 5 A schematic block diagram of a remote sensing video multi-target tracking device according to an embodiment of this application is shown. Detailed Implementation

[0028] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0031] Figure 1 A flowchart illustrating a remote sensing video multi-target tracking method according to an embodiment of this application is shown schematically.

[0032] like Figure 1 As shown, the remote sensing video multi-target tracking method of this embodiment includes steps S110 to S170.

[0033] In step S110, multiple image frames are acquired from the remote sensing video, and multiple candidate image frames containing the target object are extracted from the multiple image frames.

[0034] Remote sensing video can be video data acquired by imaging sensors mounted on remote sensing platforms (such as satellites, drones, and aircraft) through continuous imaging of the Earth's surface. Its characteristics include: a top-down shooting perspective, small target size, complex backgrounds (such as the presence of clouds, oceans, forests, and buildings), and large frame intervals.

[0035] For example, remote sensing video can include continuous satellite footage of airports, drone footage of port vessels being monitored, and aircraft footage of ground reconnaissance.

[0036] An image frame can be any static image that makes up a remote sensing video. It is the basic unit of a remote sensing video and its characteristics include: each image frame is arranged in chronological order, each image frame corresponds to a shooting time, and each image frame is played continuously to form a remote sensing video.

[0037] For example, if a satellite video lasts for 10 seconds and has 25 frames per second, then the satellite video consists of a total of 250 image frames.

[0038] The target object can be a moving object of interest and with specific significance that needs to be detected and tracked in remote sensing video, and it can appear in some of the image frames in multiple image frames.

[0039] For example, the target could be an airplane in an airport, a ship in a port, or a vehicle on a road.

[0040] Candidate image frames (also called candidate frames) are image frames selected from multiple image frames in remote sensing video that contain one or more target objects, and are used as objects for subsequent detection and tracking processing.

[0041] In step S120, the object detection result of the target object contained in each candidate image frame is detected. The object detection result includes the detection location and current attribute information.

[0042] Object detection results can be a set of information about each target object output after processing candidate image frames through an object detection model. Each target object can correspond to one object detection result, and a candidate image frame can contain multiple object detection results.

[0043] The detection location can be the spatial location information of the target object in a candidate image frame, which can be represented by a bounding box. The current attribute information can be the feature information of the target object in a candidate image frame, used to describe the appearance characteristics of the target.

[0044] In step S130, a pre-established trajectory set containing multiple historical tracking trajectories is obtained. Each historical tracking trajectory represents the motion trajectory of a tracked historical object during a historical period. The trajectory set stores the loss count value of each historical tracking trajectory, the historical motion information and historical attribute information of the corresponding historical object. The loss count value represents the number of consecutive candidate image frames that a historical tracking trajectory has failed to match before any candidate image frame.

[0045] A historical tracking trajectory can be a path record formed by connecting all successfully matched location points of a tracked target object in chronological order within a historical time period. The historical time period can be the time period of candidate image frames preceding the current candidate image frame. A trajectory set can be a pre-established collection of multiple historical tracking trajectories used for unified management of all currently tracked target trajectories.

[0046] Historical objects can be objects that have been tracked before the current candidate image frame. These objects can be target objects or not; they are entities corresponding to historical tracking trajectories.

[0047] The motion trajectory can be a path formed by connecting all the successfully detected locations of a historical object in chronological order within a historical period, with each location point corresponding to a historical image frame.

[0048] The loss count can be the number of consecutive candidate image frames in which a historical tracking trajectory corresponding to a historical object fails to match any target object before the current candidate image frame. This can be used to determine whether the target object has permanently left the target area.

[0049] Historical motion information can be information describing the motion patterns and trends of historical objects. Historical motion information can include the state information of historical objects (such as position, velocity, etc.) and matrix information (such as the covariance matrix used to represent uncertainty).

[0050] Historical attribute information can be the appearance features of a historical object when it was successfully matched within a historical period, and it can be stored in the form of a feature set.

[0051] In step S140, based on historical motion information, the predicted position of the historical object corresponding to the historical motion information in the candidate image frame is predicted.

[0052] The predicted location can be the possible location of a historical object in each candidate image frame. It is an estimate of the historical object's location in the candidate image frame, rather than the actual detected location.

[0053] In step S150, the positional similarity between the detected position and the predicted position is calculated, and the attribute similarity between the current attribute information and the historical attribute information is calculated. The positional similarity and attribute similarity are then fused to obtain a comprehensive similarity.

[0054] In step S160, the target object is matched with multiple historical objects in a cascade manner based on the comprehensive similarity and the loss count.

[0055] While determining the overall similarity between the target object and any historical object, matching can be performed in batches according to the length of time the historical object disappeared (i.e., the size of the loss count), and in order of priority. The longer the historical object has been missing, the less accurate its predicted location is, and therefore it should not be matched in the same order as historical objects with shorter loss times.

[0056] In step S170, if the target object is successfully matched with any historical object, the historical attribute information and historical motion information corresponding to any historical object are updated based on the object detection result of the target object, and the target tracking trajectory of the target object in the remote sensing video is obtained according to the updated historical tracking trajectory of any historical object.

[0057] Understandably, a target tracking trajectory can be a record of the complete motion path of a target object in remote sensing video from its appearance to its disappearance (or to the current candidate frame). A target tracking trajectory can be a sequence of detection positions that were successfully matched across all historical image frames, arranged chronologically, and can record instances where the target object was lost in multiple image frames.

[0058] The remote sensing video multi-target tracking method based on the embodiments of this application fully expresses the multidimensional feature information of target objects contained in the remote sensing video by fusing position similarity and attribute similarity, and can maintain stable matching performance even under complex background and lighting conditions. By using lost count values ​​and comprehensive similarity, the method solves the problem of re-association after long-term occlusion of target objects, reduces the number of identity jumps of multiple target objects, and improves tracking continuity and stability.

[0059] In the embodiments of this application, the trajectory set is established in the following manner: for candidate image frames in which any historical object appears for the first time among multiple candidate image frames, a Kalman filter is initialized based on the detection position corresponding to the candidate image frame, and a tracker is assigned to the historical object. The tracker is used to store the historical attribute information of the historical object; based on the tracker and the Kalman filter, the historical tracking trajectory of the historical object is obtained; and based on the historical tracking trajectory of each historical object, a trajectory set is established.

[0060] Understandably, a Kalman filter can be an algorithm for estimating the state of a linear system. It can use the known state equations of the linear system, observe the system's input and input data, and make an optimal estimate of the dynamic system.

[0061] For example, a historical object whose detection location has been determined can be represented by the following formula;

[0062]

[0063] In the formula, For the state vector of the historical object, The location coordinates of the historical object. The speed of historical objects.

[0064] Kalman filters assume position and speed These two variables follow a normal distribution, meaning they are random and follow a Gaussian distribution, and each variable has a mean. and variance During the tracking process, the location and speed There is a correlation between them, meaning that the position of a historical object in the next candidate image frame is related to the speed of the current candidate image frame.

[0065] Location and speed The correlation between them can be used to determine the position. and speed The covariance matrix is ​​used to calculate each element of the covariance matrix. It can represent the first The variable and the first The degree of correlation between variables.

[0066] When initializing the Kalman filter, the position of the historical object is obtained for the first time. And speed If it is unknown, then the initial velocity can be... Set to 0. During initialization, a position can be assumed. and speed The error is incoherent, i.e., 0. An initial covariance matrix is ​​defined to represent this uncertainty.

[0067]

[0068] In the formula, Let be the initial covariance matrix. The variance of the historical object location estimate. The variance of the velocity estimate for historical objects.

[0069] For example, It can be set based on the noise characteristics of the detector (which can be used to detect the detection position of historical objects or target objects in the image frame), such as a large variance related to the width and height of the detection box. It can be set to a large value to represent the uncertainty of the initial velocity estimate.

[0070] The remote sensing video multi-target tracking method based on the embodiments of this application effectively addresses target occlusion and missed detection by initializing a Kalman filter and assigning a tracker to the first historical object, ensuring the continuity of the trajectory, improving the accuracy of multi-target tracking, and reducing the number of ID switching.

[0071] In the embodiments of this application, predicting the predicted position of the historical object corresponding to the historical motion information in the candidate image frame based on the historical motion information includes: inputting the historical motion information of the historical object into the corresponding Kalman filter; and predicting the predicted position of the historical object in the candidate image frame through the state transition equation of the Kalman filter.

[0072] For example, historical motion information can include the position and velocity of historical objects. During the tracking process, for the first... The optimal estimate of the position and velocity of historical objects in each candidate image frame can be calculated using the following formula;

[0073]

[0074]

[0075] In the formula, For historical objects in the first The state vector in each candidate image frame For historical objects in the first The uncertainty of the state vector in each candidate image frame. The variance of the historical object location estimate. The variance of the velocity estimate for historical objects. and All are the cross-variances of the position and velocity of historical objects, and are transposes of each other.

[0076] Suppose the transformation relationship between the state vector of the current candidate image frame and the next candidate image frame can be represented by a matrix. If we express this, then the state of an ideal system can be represented by the following formula based on basic kinematic principles;

[0077]

[0078]

[0079]

[0080] In the formula, For historical objects in the first The ideal position in each candidate image frame For historical objects in the first The ideal position in each candidate image frame The time interval between adjacent candidate image frames. For historical objects in the first Ideal speed in candidate image frames For historical objects in the first Ideal speed in candidate image frames For historical objects in the first The state vector in each candidate image frame Let be the state transition matrix.

[0081] Based on the above formula, and according to the rules of matrix operations, the transformation relationship of the covariance matrix can be obtained;

[0082]

[0083] In the formula, For historical objects in the first The uncertainty of the state vector in each candidate image frame.

[0084] However, the external environment may affect the system state. Therefore, we can assume that there is acceleration, and the ideal system state will change, which can be expressed by the following formula.

[0085]

[0086]

[0087]

[0088] In the formula, For historical objects in the first Predicted location in candidate image frames For the acceleration of historical objects, For historical objects in the first Prediction speed in candidate image frames For the control matrix, For control vectors, For historical objects in the first The predicted state vector in each candidate image frame.

[0089] Based on the above formula, and according to the rules of matrix operations, the transformation relationship of the predicted covariance matrix can be obtained.

[0090]

[0091] In the formula, For historical objects in the first The prediction uncertainty of the state vector in each candidate image frame. It is noise.

[0092] The remote sensing video multi-target tracking method based on the embodiments of this application effectively filters out detection noise by inputting historical motion information into a Kalman filter, making the prediction results closer to the real motion trajectory and reducing identity jumps and trajectory fragmentation.

[0093] In the embodiments of this application, the detection position of the target object or the historical object can be detected in the following way: based on the conversion method of the preset sensor, the distribution of readings in the sensor is obtained; based on the distribution of readings in the sensor, the detection position of the target object or the historical object is obtained.

[0094] For example, the estimation results can be improved based on the sensor's object detection results in the current candidate image frame. By defining the sensor's transformation method as an observation matrix, the distribution of the current sensor readings (i.e., ideal observations) can be obtained;

[0095]

[0096]

[0097] In the formula, For the sensor in the first Ideal observations in candidate image frames For the observation matrix, Let be the covariance matrix (i.e., uncertainty) of the observed values.

[0098] Similarly, sensors also exhibit noise, and the covariance of the noise is defined as... The mean of the distribution can be obtained through observation, and finally the optimal estimate of the ideal observed value can be obtained.

[0099]

[0100]

[0101]

[0102] In the formula, For correction amount, Let be the covariance matrix of the noise. For historical objects in the first The updated predicted state vector in each candidate image frame. For historical objects in the first The prediction uncertainty of the updated state vector in each candidate image frame.

[0103] For example, in a tracking scenario, define As a representation of the state of the target object, The center position of the bounding box. Aspect ratio, For height, These represent their velocities in the image coordinates. A Kalman filter based on uniform motion and a linear model is used as the tracking predictor. As a direct observation of the detection location of the target object in any candidate image frame .

[0104] The remote sensing video multi-target tracking method based on the embodiments of this application analyzes the reading distribution by pre-setting the sensor conversion method, effectively suppressing measurement errors and interference in the original data, and enhancing the adaptability to complex environments such as different lighting and occlusion.

[0105] Figure 2 A flowchart illustrating the calculation of attribute similarity according to an embodiment of this application is shown.

[0106] like Figure 2 As shown, the attribute similarity calculation in this embodiment includes steps S151 to S153.

[0107] In step S151, the current depth feature, current texture feature, and current color feature in the current attribute information are extracted, as are the historical depth feature, historical texture feature, and historical color feature in the historical attribute information.

[0108] For example, Local Binary Patterns (LBP) feature vectors can be used as texture features (such as current texture features or historical texture features).

[0109] It is understandable that Local Binary Mode (LCM) is a texture descriptor that can be used to extract local features of a target. It can serve as a feature representation for measuring the local texture information of an image and remains unchanged under changes such as rotation, grayscale, and illumination.

[0110] The LBP feature vector is calculated in a 3×3 window. The gray value of the center pixel is used as a reference, and the values ​​of the surrounding 8 pixels are compared with it. If the gray value of the center pixel is greater than the gray value of the surrounding pixels, the pixel position is marked as 0; otherwise, it is marked as 1.

[0111] After the above processing, the eight points surrounding the 3×3 window can generate an 8-bit binary number (which can be converted to decimal, i.e., LBP code, of which there are 256 types). This 8-bit binary number is used as the LBP value of the center pixel of the window, and this LBP value can reflect the texture information of the area. If the above operation is performed on each pixel in an image (such as multiple image frames of remote sensing video), the LBP feature map of that image frame can be obtained.

[0112] For example, to obtain the LBP vector, the LBP map can be divided into 2×2 small regions, and the histogram of each small region can be calculated, which represents the frequency of each digit. The histograms of each small region are then concatenated to form a feature vector, and this feature vector is then normalized to obtain a unit feature vector, which is the desired texture feature.

[0113] For example, the color space can be divided into A color range is used to quantize candidate image frames. Connected regions are defined based on the pixel values ​​of the quantized candidate image frames. The requirements for a connected region are that the pixel values ​​of the pixels in the region are equal and there is a path between any two pixels.

[0114] For example, 1% of the total number of pixels in a candidate image frame can be used as a pixel threshold. The number of pixels in each connected region can be counted. If this number is greater than the set pixel threshold, the pixels in that region are considered aggregated pixels; otherwise, they are considered non-aggregated pixels. (The sentence is incomplete and requires further context.) The number of aggregated pixels and non-aggregated pixels in each color interval are defined as aggregated pixels. Non-aggregated pixels Then the color aggregation vector of the candidate image frame can be ,in, This represents the total number of color ranges. , For the first The number of aggregated pixels in each color range For the first The number of non-aggregated pixels in each color range.

[0115] Finally, the color aggregation vector is transformed into the desired color features (such as current color features or historical color features) through a normalization operation.

[0116] In step S152, the depth feature distance between the current depth feature and the historical depth feature, the texture feature distance between the current texture feature and the historical texture feature, and the color feature distance between the current color feature and the historical color feature are calculated respectively.

[0117] For example, the The tracker and the first The depth feature distance between object detection results It can be obtained based on distance information or 3D structural information of the target object or historical objects.

[0118] For example, the The tracker and the first The texture feature distance between the detection results of an object can be expressed by the following formula;

[0119]

[0120] In the formula, For the first The tracker and the first Texture feature distance between object detection results For the first The texture features corresponding to the current attribute information in the object detection results For the first The historical object corresponding to the tracker is in the first... Texture features in candidate image frames For the first The feature set stored in each tracker.

[0121] For example, the The tracker and the first The color feature distance between the detection results of an object can be expressed by the following formula;

[0122]

[0123] In the formula, For the first The tracker and the first Color feature distance between object detection results For the first The color feature corresponding to the current attribute information in the object detection result. For the first The historical object corresponding to the tracker is in the first... Color features in candidate image frames.

[0124] In step S153, the depth feature distance, texture feature distance, and color feature distance are fused to obtain the attribute similarity.

[0125] For example, the depth feature distance, texture feature distance, and color feature distance can be concatenated into a long vector, and the Euclidean distance or cosine distance of this long vector can be directly calculated as the attribute similarity.

[0126] The remote sensing video multi-target tracking method based on the embodiments of this application extracts and calculates the distances of depth, texture, and color features respectively, and then fuses them to obtain a comprehensive attribute similarity. This fully utilizes the complementarity of multimodal features and improves the accuracy of target re-identification.

[0127] Figure 3 A schematic comparison diagram of the original image and the texture feature image according to an embodiment of this application is shown.

[0128] like Figure 3 As shown, Figure 3 (a) is the original image of the target object in a candidate image frame. The original image is a clear top-down view of the aircraft with a solid color background, complete aircraft outline and details, and high contrast. Figure 3 (b) in the image is the texture feature image of the original image. After local binary mode extraction, the edge and structural texture of the aircraft are preserved. The background presents dense texture noise, which highlights the texture features of the target object and enhances the distinguishability of the target object against a complex texture background.

[0129] In the embodiments of this application, the positional similarity between the detected position and the predicted position can be calculated as follows: extract the detection bounding box of the detected position and the predicted bounding box of the predicted position; calculate the positional similarity between the detection bounding box and the predicted bounding box based on the Euclidean distance.

[0130] For example, detecting bounding boxes is =( The predicted bounding box is... =( ), where the size of the bounding box is detected. and predict the size of the bounding box It can be set to be the same, according to and The Euclidean distance between them, the first The tracker and the first The positional similarity between the detection results of multiple objects can be expressed by the following formula;

[0131]

[0132] In the formula, For the first The tracker and the first Positional similarity between object detection results To detect the coordinates of the center point of the bounding box, This is to predict the coordinates of the center point of the bounding box.

[0133] For example, the overall similarity is based on attribute similarity and location similarity, and can be expressed by the following formula;

[0134]

[0135] In the formula, For the first The tracker and the first The overall similarity between the detection results of individual objects These are the weighting coefficients.

[0136] In the embodiments of this application, when the target object successfully matches any historical object, the historical attribute information and historical motion information corresponding to any historical object are updated based on the object detection result of the target object, including: updating the parameters of the Kalman filter corresponding to any historical object based on the detection position of the target object to obtain the updated historical motion information; and updating the historical attribute information stored in the tracker corresponding to any historical object based on the current attribute information of the target object to obtain the updated historical attribute information.

[0137] It is understandable that the Kalman filter maintains the state of the target object or historical objects (i.e., ,Location and speed and uncertainty (i.e., covariance matrix) After a successful match, new observations were obtained. (i.e., the successfully matched detection position), use this detection position to correct the corresponding predicted position, and obtain the updated historical motion information.

[0138] It is understandable that the appearance of the target object (such as lighting, viewpoint, pose, etc.) will change over time, so it is necessary to use the current attribute information of the target object to adjust the historical attribute information stored in the tracker.

[0139] In the embodiments of this application, the target object is cascaded with multiple historical objects based on the comprehensive similarity and the loss count value, including: sorting the multiple historical objects in ascending order of the loss count value; and matching the target object with each historical object in the sorted order based on the comprehensive similarity.

[0140] If a target is occluded for a long time, the uncertainty of the Kalman filter prediction becomes significant. This means the predicted location of the target object in the current candidate image frame becomes less accurate. Therefore, when multiple historical tracking trajectories compete for the same object detection result during the matching process, historical tracking trajectories with shorter disappearance times are more accurate and should be prioritized for matching.

[0141] For example, the Hungarian algorithm can be used to match historical objects with target objects based on similarity.

[0142] The remote sensing video multi-target tracking method based on the embodiments of this application sorts historical objects in ascending order of loss count value and performs cascade matching in sequence, giving priority to matching the most recently frequently occurring historical objects, effectively solving the problem of long-term occlusion targets interfering with short-term targets.

[0143] In embodiments of this application, if the target object fails to match, the method further includes: initializing a new Kalman filter based on the detection position of the target object, and assigning a new tracker to the target object; obtaining the target tracking trajectory of the target object based on the new tracker and the new Kalman filter, and storing the current attribute information corresponding to the target object into the new tracker.

[0144] In embodiments of this application, when the target object fails to match any historical object, the method further includes: for any historical tracking trajectory corresponding to a historical object that has not matched the target object in any candidate image frame, increasing the loss count value of the historical tracking trajectory; if the loss count value is less than a preset counting threshold, retaining the corresponding historical tracking trajectory and continuing to match the corresponding historical tracking trajectory with the target object in the next candidate image frame; before the loss count value reaches the counting threshold, if any historical object is rematched with the object detection result of the target object, resetting the loss count value of the historical tracking trajectory corresponding to any historical object to zero, and updating the historical tracking trajectory corresponding to any historical object based on the object detection result of the rematched target object; if the loss count value is greater than or equal to the counting threshold, deleting the corresponding historical tracking trajectory from the trajectory set.

[0145] For any candidate image frame, the matching process includes the following steps:

[0146] (1) At the input end, obtain a trajectory set containing multiple historical tracking trajectories. and a detection set containing multiple object detection results And a maximum age has been set. (i.e., the counting threshold);

[0147] (2) Calculate the association cost matrix according to the similarity calculation formula. , For comprehensive similarity;

[0148] (3) Calculate the gating matrix , For the first The tracker and the first Gating results between object detection results;

[0149] (4) Initialize the matching set of the matching object detection results Initialize the unmatched set of the unmatched object detection results. ;

[0150] (5) Assumption Age represents the number of image frames in which the historical tracking trajectory disappears. The value of can be 1, ..., .according to The values ​​are selected in ascending order, and a set of historical tracking trajectories is selected one at a time. The tracking trajectory is matched with the detection results according to the principle of minimum distance matching.

[0151] Following the cascaded matching process described above, matching is performed in ascending order of Missing_age. Successfully matched historical tracking trajectories and object detection results are assigned IDs, and the detection results are used to update the Kalman filter parameters.

[0152] If a trajectory fails to match, its Missing_age (missing count) is incremented by one. Missing_age refers to the number of candidate image frames for which a historical tracked trajectory has failed to match. Missing_age = 0 indicates a successful match in the previous frame, while Missing_age = N indicates N frames without a match. The larger the Missing_age, the lower the probability of that candidate image frame appearing. When the value of Missing_age exceeds a set counting threshold... If a target object or historical object is detected as a match, it is considered that the target object or historical object has left the tracking scene, and its corresponding tracking trajectory (i.e., historical tracking trajectory or target tracking trajectory) is deleted. If an object detection result is a match, it is considered that the object detection result corresponds to a newly appeared target object, and it is added to the trajectory set as a new trajectory for tracking in subsequent frames.

[0153] In the embodiments of this application, the unmatched historical tracking trajectory has a trajectory state, which includes an unconfirmed state and a confirmed state. The method further includes: for the unconfirmed historical tracking trajectory, if the historical tracking trajectory is successfully matched in a consecutive preset number of candidate image frames, updating the trajectory state of the historical tracking trajectory to a confirmed state; if the historical tracking trajectory is not successfully matched in a number of candidate image frames, deleting the historical tracking trajectory from the trajectory set.

[0154] The following is combined with Figure 4 The process of matching target objects and historical objects in the embodiments of this application will be described in detail.

[0155] Figure 4 A flowchart illustrating the matching of target objects and historical objects according to an embodiment of this application is shown.

[0156] like Figure 4 As shown, the process of matching the target object with the historical object in this embodiment includes steps S411 to S490.

[0157] In step S411, the object detection result of the target object is obtained.

[0158] In step S412, a pre-established set of trajectories containing multiple historical tracking trajectories is obtained.

[0159] In step S421, the Kalman filter predicts the position of the historical tracking trajectory of the confirmed state in a candidate image frame.

[0160] In step S422, the Kalman filter predicts the position of the historical tracking trajectory in an unconfirmed state within a candidate image frame.

[0161] In step S430, the target object and historical objects are cascaded and matched.

[0162] In step S441, the historical tracking trajectories that were not matched after cascaded matching are output.

[0163] In step S442, the results of unmatched object detection after cascading matching are output.

[0164] In step S443, the detection results of the matched objects after cascading matching are output.

[0165] After cascaded matching, the detected object results can be used to update the parameters of the Kalman filter to obtain the target tracking trajectory of the target object corresponding to the detected object result, and then add the target tracking trajectory to the trajectory set.

[0166] In step S450, cross-comparison (CUC) matching is performed on the unmatched historical tracking trajectories and object detection results.

[0167] In step S461, the historical tracking trajectories that did not match after crossover and union matching are output.

[0168] In step S462, the results of the detection of unmatched objects after cross-union matching are output.

[0169] In step S463, the results of the matched object detection after cross-union matching are output.

[0170] After the cross-union-match (CUC) is performed, the detected object results can be used to initialize a new Kalman filter to obtain a new target tracking trajectory. Based on the new target tracking trajectory, the parameters of the corresponding Kalman filter are updated, and the new target tracking trajectory is added to the trajectory set.

[0171] In step S471, historical tracking trajectories that did not match after cross-comparison and union matching but whose trajectory status is confirmed are processed.

[0172] Historical tracking trajectories that did not match after intersection-union comparison but whose trajectory status is confirmed are deleted from the trajectory set.

[0173] In step S472, historical tracking trajectories that did not match after cross-comparison and union matching but whose trajectory status is unconfirmed are processed.

[0174] Determine the loss count value of the historical tracking trajectory; if the loss count value is greater than or equal to the counting threshold... In the case of a missing track, the historical tracking track is deleted from the track set; if the missing count is less than the counting threshold... In the case of unconfirmed historical tracking trajectory, retain the historical tracking trajectory and continue with the subsequent steps S411 to S490.

[0175] If three consecutive candidate image frames are hit, the trajectory status of the historical tracking trajectory is changed from unconfirmed to confirmed.

[0176] In embodiments of this application, the method further includes: inputting multiple candidate image frames into a pre-trained unified neural network model, wherein the unified neural network model simultaneously outputs object detection results and matching results; and generating target tracking trajectories for each target object based on the object detection results and matching results.

[0177] For example, a unified neural network model includes: a shared backbone network for extracting shared feature representations of multiple candidate image frames; a detector connected to the shared backbone network for outputting bounding boxes and class confidence scores for each target object based on the shared feature representations; a feature embedder connected to the shared backbone network for extracting discriminative feature vectors for each target object based on the shared feature representations; and a matcher connected to the feature embedder for predicting matching relationships between each target object and between historical objects and target objects based on feature vectors of target objects in adjacent candidate image frames.

[0178] For example, the matcher can use a graph neural network to construct a cross-frame bipartite graph by treating the target object in each candidate image frame as a node, and determine the matching relationship between the target object and historical objects or between the target object in the previous candidate image frame and the target object in the next candidate image frame by predicting the existence of edges.

[0179] The remote sensing video multi-target tracking method based on the embodiments of this application uses an end-to-end unified neural network model to simultaneously output object detection results and matching results, avoiding the accumulation of detection and matching errors. By jointly optimizing detection, feature embedding, and association loss, the network can autonomously learn more discriminative features, simplifying the tracking process, eliminating the need for complex post-processing, and improving the accuracy and efficiency of multi-target tracking.

[0180] Based on the above-mentioned remote sensing video multi-target tracking method, this application also provides a remote sensing video multi-target tracking device, which will be described below in conjunction with... Figure 5 The device is described in detail.

[0181] Figure 5 A schematic block diagram of a remote sensing video multi-target tracking device according to an embodiment of this application is shown.

[0182] like Figure 5 As shown, the remote sensing video multi-target tracking device 300 of this embodiment includes an image acquisition module 510, an image detection module 520, a set acquisition module 530, a position prediction module 540, a similarity calculation module 550, an object matching module 560, and a trajectory tracking module 570.

[0183] The image acquisition module 510 is used to acquire multiple image frames from the remote sensing video and extract multiple candidate image frames containing the target object. In one embodiment, the image acquisition module 510 can be used to perform step S110 described above, which will not be repeated here.

[0184] The image detection module 520 is used to detect the object detection result of the target object contained in each candidate image frame. The object detection result includes the detection location and current attribute information. In one embodiment, the image detection module 520 can be used to perform step S120 described above, which will not be repeated here.

[0185] The set acquisition module 530 is used to acquire a pre-established set of trajectories containing multiple historical tracking trajectories. Each historical tracking trajectory represents the motion trajectory of a tracked historical object during a historical time period. The trajectory set stores the loss count value of each historical tracking trajectory, the historical motion information of the corresponding historical object, and historical attribute information. The loss count value represents the number of consecutive image frames in which a historical tracking trajectory has not been successfully matched before any candidate image frame. In one embodiment, the set acquisition module 530 can be used to execute step S130 described above, which will not be repeated here.

[0186] The position prediction module 540 is used to predict the predicted position of the historical object corresponding to the historical motion information in the candidate image frame based on the historical motion information. In one embodiment, the position prediction module 540 can be used to perform step S140 described above, which will not be repeated here.

[0187] The similarity calculation module 550 is used to calculate the positional similarity between the detected position and the predicted position, and to calculate the attribute similarity between the current attribute information and the historical attribute information. The positional similarity and attribute similarity are then fused to obtain a comprehensive similarity. In one embodiment, the similarity calculation module 550 can be used to perform step S150 as described above, which will not be repeated here.

[0188] The object matching module 560 is used to perform cascade matching between the target object and multiple historical objects based on comprehensive similarity and loss count values. In one embodiment, the object matching module 560 can be used to perform step S160 described above, which will not be repeated here.

[0189] The trajectory tracking module 570 is used to update the historical attribute information and historical motion information corresponding to any historical object based on the object detection result of the target object when the target object successfully matches any historical object, and obtain the target tracking trajectory of the target object in the remote sensing video according to the updated historical tracking trajectory of any historical object. In one embodiment, the trajectory tracking module 570 can be used to perform step S170 described above, which will not be repeated here.

[0190] According to embodiments of this application, any multiple modules among the image acquisition module 510, image detection module 520, set acquisition module 530, position prediction module 540, similarity calculation module 550, object matching module 560, and trajectory tracking module 570 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the image acquisition module 510, image detection module 520, set acquisition module 530, position prediction module 540, similarity calculation module 550, object matching module 560, and trajectory tracking module 570 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPMA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the image acquisition module 510, image detection module 520, set acquisition module 530, position prediction module 540, similarity calculation module 550, object matching module 560, and trajectory tracking module 570 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0191] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined or combined in various ways without departing from the spirit and teachings of this application. All such combinations or combinations fall within the scope of this application.

[0192] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A method for multi-target tracking in remote sensing video, characterized in that, include: Acquire multiple image frames from remote sensing video, and extract multiple candidate image frames containing the target object from the multiple image frames; Detect the object detection result of the target object contained in each of the candidate image frames, wherein the object detection result includes the detection location and current attribute information; Obtain a pre-established trajectory set containing multiple historical tracking trajectories. Each historical tracking trajectory represents the motion trajectory of a tracked historical object during a historical period. The trajectory set stores the loss count value of each historical tracking trajectory, the historical motion information and historical attribute information of the corresponding historical object. The loss count value represents the number of consecutive candidate image frames that a historical tracking trajectory has failed to match before any candidate image frame. Based on the historical motion information, predict the predicted position of the historical object corresponding to the historical motion information in the candidate image frame; Calculate the positional similarity between the detected position and the predicted position, and calculate the attribute similarity between the current attribute information and the historical attribute information. Then, fuse the positional similarity and the attribute similarity to obtain a comprehensive similarity. Based on the comprehensive similarity and the loss count, the target object is matched in a concatenated manner with multiple historical objects; If the target object is successfully matched with any historical object, the historical attribute information and historical motion information corresponding to any historical object are updated based on the object detection result of the target object, and the target tracking trajectory of the target object in the remote sensing video is obtained according to the updated historical tracking trajectory of any historical object.

2. The method according to claim 1, characterized in that, The trajectory set is established in the following way: For a candidate image frame in which any historical object appears for the first time among the plurality of candidate image frames, a Kalman filter is initialized based on the detection position corresponding to the candidate image frame, and a tracker is assigned to the historical object. The tracker is used to store the historical attribute information of the historical object. Based on the tracker and the Kalman filter, the historical tracking trajectory of the historical object is obtained; The trajectory set is established based on the historical tracking trajectory of each historical object.

3. The method according to claim 2, characterized in that, Based on the historical motion information, predicting the predicted position of the historical object corresponding to the historical motion information in the candidate image frame includes: The historical motion information of the historical object is input into the corresponding Kalman filter; The predicted position of the historical object in the candidate image frame is predicted using the state transition equation of the Kalman filter.

4. The method according to claim 1, characterized in that, Calculating the attribute similarity between the current attribute information and the historical attribute information includes: Extract the current depth feature, current texture feature, and current color feature from the current attribute information, and extract the historical depth feature, historical texture feature, and historical color feature from the historical attribute information; Calculate the depth feature distance between the current depth feature and the historical depth feature, the texture feature distance between the current texture feature and the historical texture feature, and the color feature distance between the current color feature and the historical color feature, respectively. The depth feature distance, texture feature distance, and color feature distance are fused to obtain the attribute similarity.

5. The method according to claim 2, characterized in that, If the target object successfully matches any historical object, based on the object detection result of the target object, the historical attribute information and historical motion information corresponding to any historical object are updated, including: Based on the detection position of the target object, the parameters of the Kalman filter corresponding to any historical object are updated to obtain the updated historical motion information; Based on the current attribute information of the target object, the historical attribute information stored in the tracker corresponding to any historical object is updated to obtain the updated historical attribute information.

6. The method according to claim 1, characterized in that, Based on the comprehensive similarity and the loss count, the target object is cascaded with multiple historical objects, including: The historical objects are sorted in ascending order of their lost count values; Following the sorted order, the target object is matched with each of the historical objects sequentially based on the comprehensive similarity.

7. The method according to claim 1, characterized in that, If the target object is not successfully matched, the method further includes: Based on the detection position of the target object, initialize a new Kalman filter and assign a new tracker to the target object; Based on the new tracker and the new Kalman filter, the target tracking trajectory of the target object is obtained, and the current attribute information corresponding to the target object is stored in the new tracker.

8. The method according to claim 7, characterized in that, If the target object fails to match any historical object, the method further includes: For any historical tracking trajectory corresponding to any historical object that does not match the target object in any candidate image frame, increase the loss count value of the historical tracking trajectory; If the lost count value is less than a preset count threshold, the corresponding historical tracking trajectory is retained, and the corresponding historical tracking trajectory is continued to be matched with the target object in the next candidate image frame; Before the loss count value reaches the count threshold, if any historical object is re-matched with the object detection result of the target object, the loss count value of the historical tracking trajectory corresponding to any historical object is reset to zero, and the historical tracking trajectory corresponding to any historical object is updated based on the object detection result of the re-matched target object. If the lost count value is greater than or equal to the count threshold, the corresponding historical tracking trajectory will be deleted from the trajectory set.

9. The method according to claim 8, characterized in that, The unmatched historical tracking trajectories have trajectory states, including unconfirmed and confirmed states; the method further includes: For the historical tracking trajectory in an unconfirmed state, if the historical tracking trajectory is successfully matched in a consecutive preset number of candidate image frames, the trajectory status of the historical tracking trajectory will be updated to a confirmed state. If the historical tracking trajectory fails to match any of the candidate image frames in the specified number, the historical tracking trajectory is deleted from the trajectory set.

10. A remote sensing video multi-target tracking device, characterized in that, include: The image acquisition module is used to acquire multiple image frames in remote sensing video and extract multiple candidate image frames containing the target object from the multiple image frames; An image detection module is used to detect the object detection result of the target object contained in each candidate image frame, wherein the object detection result includes the detection location and current attribute information; The set acquisition module is used to acquire a pre-established set of trajectories containing multiple historical tracking trajectories. Each historical tracking trajectory represents the motion trajectory of a tracked historical object during a historical period. The set of trajectories stores the loss count value of each historical tracking trajectory, the historical motion information of the corresponding historical object, and the historical attribute information. The loss count value represents the number of consecutive candidate image frames that a historical tracking trajectory has failed to match before any candidate image frame. The position prediction module is used to predict the predicted position of the historical object corresponding to the historical motion information in the candidate image frame based on the historical motion information. The similarity calculation module is used to calculate the positional similarity between the detected position and the predicted position, and to calculate the attribute similarity between the current attribute information and the historical attribute information, and to fuse the positional similarity and attribute similarity to obtain a comprehensive similarity. An object matching module is used to perform cascade matching between the target object and multiple historical objects based on the comprehensive similarity and the loss count value; The trajectory tracking module is used to update the historical attribute information and historical motion information corresponding to any historical object based on the object detection result of the target object when the target object is successfully matched with any historical object, and to obtain the target tracking trajectory of the target object in the remote sensing video based on the updated historical tracking trajectory of any historical object.

Images