Target tracking method and device, computer device and storage medium

Abstract

The application provides a target tracking method and device, computer equipment and a storage medium. The method comprises: acquiring a plurality of infrared images; for the i-th infrared image, determining a first detection box of a target object contained in the i-th infrared image based on a first detection algorithm in a target tracking algorithm, and determining a second prediction box of the target object contained in the i-th infrared image based on a second prediction algorithm in the target tracking algorithm; wherein the second prediction algorithm comprises a dynamic adjustment parameter, and the dynamic adjustment parameter corresponding to the i-th infrared image is a parameter for adjusting the aspect ratio of the second prediction box of the i-th infrared image, which is determined based on the first detection box and the second prediction box corresponding to the (i-1)-th infrared image; i is a positive integer; and a tracking result of the target object in the i-th infrared image is determined based on the first detection box and the second prediction box. In this way, the accuracy of infrared target detection and tracking can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically to a target tracking method, apparatus, computer device, and storage medium. Background Technology

[0002] Infrared detection and tracking technology boasts advantages such as long detection range and strong anti-interference capabilities, leading to its widespread application in military and civilian fields, including missile defense, aerospace, and security. However, its application in infrared small target detection and tracking faces numerous complex constraints. Targets are imaged at long distances, occupying a small proportion of the image and lacking texture features, making them susceptible to clutter and noise interference. Therefore, detecting and tracking multiple weak infrared targets with low signal-to-noise ratios in complex cloud environments remains a challenging task. Summary of the Invention

[0003] In view of this, this application provides a target tracking method, apparatus, computer device, and storage medium to improve the accuracy of infrared target detection and tracking.

[0004] Firstly, this application provides a target tracking method, including:

[0005] Acquire multiple frames of infrared images;

[0006] For the i-th frame of infrared image, a first detection box of the target object contained in the i-th frame of infrared image is determined based on a first detection algorithm in the target tracking algorithm, and a second prediction box of the target object contained in the i-th frame of infrared image is determined based on a second prediction algorithm in the target tracking algorithm; wherein, the first detection algorithm is an algorithm for detection based on image features of the infrared image; the second prediction algorithm is an algorithm for prediction based on motion features of the target object in the infrared image; the second prediction algorithm includes dynamic adjustment parameters, and the dynamic adjustment parameters corresponding to the i-th frame of infrared image are parameters used to adjust the aspect ratio of the second prediction box of the i-th frame of infrared image, determined based on the first detection box and the second prediction box corresponding to the (i-1)-th frame of infrared image; i is a positive integer;

[0007] Based on the first detection box and the second prediction box, the tracking result of the target object in the i-th frame infrared image is determined.

[0008] In one possible implementation, the target tracking algorithm is the DeepSORT algorithm, which is a deep simple online real-time tracking algorithm, and the second prediction algorithm is a Kalman filter.

[0009] In one possible implementation, the method further includes determining the dynamic adjustment parameters corresponding to the i-th frame infrared image according to the following method:

[0010] Determine the intersection-union ratio (I / O) between the first detection box and the second prediction box corresponding to the (i-1)th frame of the infrared image;

[0011] Based on the intersection-union ratio, the error function between the first detection box and the second prediction box corresponding to the (i-1)th frame infrared image is determined;

[0012] Based on the error function and the controller, the dynamic adjustment parameters corresponding to the i-th frame infrared image are determined; wherein, the controller is used to control the rate of change of the shape of the second prediction box.

[0013] In one possible implementation, the Kalman filter is used to predict the position of the target object in the next frame based on the motion characteristics of the target object in the previous frame of the infrared image, wherein the motion characteristics are characterized by a direction vector;

[0014] The method further includes:

[0015] Obtain the first direction vector of the target object in the (i-1)th frame of the infrared image, and predict the second direction vector of the target object in the i-th frame of the infrared image based on the Kalman filter and the first direction vector;

[0016] Based on the first detection box of the target object in the i-th frame infrared image, the second direction vector is adjusted to obtain the third direction vector;

[0017] Based on the orientation vector of the target object in the N frames of infrared images preceding the i-th frame, anomaly detection is performed on the third orientation vector; N is a positive integer greater than 1.

[0018] If an anomaly is detected, the third-party directional vector is corrected.

[0019] In one possible implementation, the anomaly detection of the third direction vector based on the direction vector of the target object in the N frames of infrared images preceding the i-th frame includes:

[0020] Based on the orientation vector of the target object in the N frames of infrared images before the i-th frame, determine the mean vector and standard deviation vector;

[0021] Anomaly detection is performed on the third-party vector based on the mean vector and the standard deviation vector.

[0022] In one possible implementation, the correction of the third-party vector includes correcting the third-party vector according to any one of the following methods:

[0023] The median of the orientation vector of the target object in the first K frames of infrared images is used as the corrected third orientation vector; K is a positive integer greater than 1;

[0024] Increase the value of the measurement noise covariance matrix in the Kalman filter; wherein, the measurement noise covariance matrix is ​​used to characterize the degree of trust in the location information of the target object detected by the first detection algorithm;

[0025] Reinitialize the Kalman filter.

[0026] In one possible implementation, determining the tracking result of the target object in the i-th frame infrared image based on the first detection box and the second prediction box includes:

[0027] The first detection box and the second prediction box are matched to determine the trajectory box of the target object; the trajectory box of the target object is the tracking result of the target object in the i-th frame infrared image.

[0028] Secondly, this application also provides a target tracking device, the device comprising:

[0029] The acquisition module is used to acquire multiple frames of infrared images;

[0030] A determination module is configured to, for an i-th frame of infrared image, determine a first detection box of a target object contained in the i-th frame of infrared image based on a first detection algorithm in the target tracking algorithm, and determine a second prediction box of the target object contained in the i-th frame of infrared image based on a second prediction algorithm in the target tracking algorithm; wherein, the first detection algorithm is an algorithm for detection based on image features of the infrared image; the second prediction algorithm is an algorithm for prediction based on motion features of the target object in the infrared image; the second prediction algorithm includes dynamic adjustment parameters, the dynamic adjustment parameters corresponding to the i-th frame of infrared image being parameters determined based on the first detection box and the second prediction box corresponding to the (i-1)-th frame of infrared image, used to adjust the aspect ratio of the second prediction box of the i-th frame of infrared image; i is a positive integer;

[0031] The tracking module is used to determine the tracking result of the target object in the i-th frame infrared image based on the first detection box and the second prediction box.

[0032] Thirdly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the target tracking method as described in the first aspect or any possible implementation of the first aspect.

[0033] Fourthly, this application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the target tracking method as described in the first aspect or any possible implementation of the first aspect.

[0034] The target tracking method, apparatus, computer device, and storage medium provided in this application, after acquiring multiple frames of infrared images, can determine a first detection box containing the target object based on a first detection algorithm of the target tracking algorithm, and determine a second prediction box containing the target object based on a second prediction algorithm of the target tracking algorithm. Then, the tracking result of the target object is determined based on the first detection box and the second prediction box. Since the dynamic adjustment parameters in the second prediction algorithm are determined based on the first detection box and the second prediction box of the previous frame of infrared image, when the second prediction algorithm predicts the second prediction box based on the dynamic adjustment parameters, the aspect ratio of the second prediction box can be closer to the current shape of the target object, thereby improving the accuracy of infrared target detection and tracking in subsequent target object detection and tracking. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating a target tracking method in an exemplary embodiment of this application;

[0036] Figure 2 This is a flowchart illustrating an anomaly detection method for direction vectors according to an exemplary embodiment of this application;

[0037] Figure 3 This is an overall block diagram illustrating a target tracking method in an exemplary embodiment of this application;

[0038] Figure 4 This is a schematic diagram of the architecture of a target tracking device shown in an exemplary embodiment of this application;

[0039] Figure 5 This is a schematic diagram of the structure of a computer device shown in an exemplary embodiment of this application. Detailed Implementation

[0040] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0041] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0042] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0043] Infrared detection and tracking technology boasts advantages such as long detection range and strong anti-interference capabilities, leading to its widespread application in military and civilian fields, including missile defense, aerospace, and security. However, its application in infrared small target detection and tracking faces numerous complex constraints. Targets are imaged at long distances, occupying a small proportion of the image and lacking texture features, making them susceptible to clutter and noise interference. Therefore, detecting and tracking multiple weak infrared targets with low signal-to-noise ratios in complex cloud environments remains a challenging task.

[0044] In related technologies, infrared target detection is generally performed using the Deep Simple Online and Realtime Tracking (DeepSORT) algorithm.

[0045] The overall process of the DeepSORT algorithm can be divided into the following steps:

[0046] 1. Target detection: Use target detectors (such as YOLO, Faster R-CNN, etc.) to detect infrared image frames and obtain the detection boxes of the tracked targets.

[0047] 2. Feature Extraction: Use deep learning models to extract the appearance features of the target, which are used in the subsequent matching process.

[0048] 3. Kalman Filter Prediction: The Kalman filter is used to predict the position of the tracked target in the next frame, and the predicted bounding box of the tracked target is obtained.

[0049] 4. Hungarian Algorithm Matching: The Hungarian algorithm is used to calculate the optimal match between the detection box and the prediction box.

[0050] 5. Track Management: Maintain the track of each tracked target, including track initialization, updating and deletion.

[0051] During infrared target detection and tracking, the size / shape of the target may change. For example, when an aircraft flies in different attitudes, it will appear in infrared images of different sizes.

[0052] As can be seen from the above process, one of the key steps in the DeepSORT algorithm lies in the matching process between the detection box and the predicted box. In the target detection process described above, detection is based on image features of infrared image frames. Therefore, the aspect ratio of the detection box is closer to the shape of the target being tracked. However, the Kalman filter determines the predicted box of the target based on its motion characteristics. The size of the predicted box may deviate significantly from the current target size. Thus, when matching the detection box and the predicted box, a match may not be possible, thereby affecting the tracking process.

[0053] Based on this, this application provides a target tracking method, apparatus, computer device, and storage medium. After acquiring multiple frames of infrared images, a first detection box containing the target object can be determined based on a first detection algorithm of the target tracking algorithm, and a second prediction box containing the target object can be determined based on a second prediction algorithm of the target tracking algorithm. Then, the tracking result of the target object is determined based on the first detection box and the second prediction box. Since the dynamic adjustment parameters in the second prediction algorithm are determined based on the first detection box and the second prediction box of the previous frame of infrared image, when the second prediction algorithm predicts the second prediction box based on the dynamic adjustment parameters, the aspect ratio of the second prediction box can be closer to the current shape of the target object, thereby improving the accuracy of infrared target detection and tracking in subsequent target object detection and tracking.

[0054] The target tracking method provided in this application will be described in detail below with reference to specific embodiments. See also Figure 1 The flowchart shown is a target tracking method provided in this application, which includes the following steps:

[0055] S101. Acquire multiple frames of infrared images.

[0056] Here, the acquired multi-frame infrared images can be multiple consecutive infrared images.

[0057] Optionally, after acquiring multiple frames of infrared images, the infrared images can be preprocessed. For example, a Gaussian curvature filter can be used to enhance the infrared images. The Gaussian curvature filter is more flexible in processing local edge features and can highlight the curvature difference between the target object and the background, thereby significantly enhancing the discernibility of small targets (i.e., the target object) in the infrared image, making them stand out more and laying the foundation for subsequent detection and tracking steps.

[0058] Optionally, before preprocessing the infrared image, the parameters of the Gaussian curvature filter can be determined based on the size of the target object in the infrared image and the noise level of the image. Then, the infrared image can be preprocessed using the Gaussian curvature filter according to these parameters to ensure that the features of small targets can be highlighted in different scenarios.

[0059] S102. For the i-th frame infrared image, a first detection box of the target object contained in the i-th frame infrared image is determined based on the first detection algorithm in the target tracking algorithm, and a second prediction box of the target object contained in the i-th frame infrared image is determined based on the second prediction algorithm in the target tracking algorithm.

[0060] Wherein, the first detection algorithm is an algorithm for detection based on the image features of the infrared image; the second prediction algorithm is an algorithm for prediction based on the motion features of the target object in the infrared image; the second prediction algorithm includes dynamic adjustment parameters, the dynamic adjustment parameters corresponding to the i-th frame infrared image are parameters for adjusting the shape of the second prediction box of the i-th frame infrared image, determined based on the first detection box and the second prediction box corresponding to the (i-1)-th frame infrared image; i is a positive integer.

[0061] Optionally, the target tracking algorithm described above can be the DeepSORT algorithm, the first detection algorithm can be a human vision-based detection algorithm, such as the Double-Neighborhood Gradient (DGRAD) algorithm, or a neural network-based detection algorithm; the second prediction algorithm can be a Kalman filter. The following embodiments will use the DGRAD algorithm as an example for the first detection algorithm, but the first detection algorithm in this application is not limited to the DGRAD algorithm.

[0062] The first detection box can be a real observation value, used to reflect the actual detection position of the target object in the current frame, but the actual detection position has a certain amount of noise and error; the second prediction box is a prediction value, which is the inference result obtained by the Kalman filter through the motion model (such as the uniform motion model), used to reflect the expected position of the target object, and is an estimate of the future position of the target object.

[0063] S103. Based on the first detection box and the second prediction box, determine the tracking result of the target object in the i-th frame infrared image.

[0064] In one possible implementation, when determining the dynamic adjustment parameters corresponding to the i-th frame infrared image, the intersection-union ratio (CIU) between the first detection box and the second prediction box corresponding to the (i-1)-th frame infrared image can be determined first; then, based on the CIU, the error function between the first detection box and the second prediction box corresponding to the (i-1)-th frame infrared image can be determined; and then, based on the error function and the controller, the dynamic adjustment parameters corresponding to the i-th frame infrared image can be determined; wherein, the controller is used to control the rate of change of the shape of the second prediction box.

[0065] For example, the error function can be calculated using the following formula:

[0066] (1)

[0067] in, Represents the error function. Indicates the second prediction box. This indicates the first detection box. This represents the intersection-union ratio (IU) of the first detection box and the second predicted box.

[0068] In the above formula, when the error between the first detection box and the second prediction box is large, that is, when the intersection-union ratio between the first detection box and the second prediction box is low, the value of the error function increases exponentially, which facilitates a rapid response when the size difference between the first detection box and the second prediction box is too large; when the error between the first detection box and the second prediction box is small, that is, when the intersection-union ratio between the first detection box and the second prediction box is high, the change of the error function tends to be stable, which improves the tolerance for minor errors.

[0069] The controller can be represented by the following formula:

[0070] (2)

[0071] in, Indicates controller, This is a preset proportionality coefficient.

[0072] The controller can be used to characterize the rate of change of the shape of the second prediction box, that is, the amount of change of the dynamic adjustment parameters of the second prediction box per unit time. This parameter can reflect the motion characteristics of the target object's shape during the motion process.

[0073] The dynamically adjustable parameters can be calculated using the following formula:

[0074] (3)

[0075] in, This represents the dynamic adjustment parameters of the i-th frame of the infrared image. This represents the dynamic adjustment parameters of the (i-1)th frame of the infrared image. This represents the time interval between the i-th infrared image and the i-th infrared image.

[0076] The dynamic adjustment parameter can be used to characterize the shape of the detection box, that is, the dynamic adjustment parameter can be the ratio of the width of the detection box to its height, and this parameter can reflect the appearance characteristics of the target object.

[0077] The dynamic adjustment parameters in this application are determined based on the matching of the first detection box and the second prediction box in the previous frame. Therefore, the dynamic adjustment parameters in this application can be understood as feedback control based on the tracking results. The feedback-controlled dynamic adjustment parameters can adjust the shape of the detection box in real time according to different target objects and scenes, thereby enabling the detection box to better adapt to changes in the size / shape of the target object and improving the accuracy and robustness of tracking the target object in deformable scenes. On the other hand, the feedback-controlled dynamic adjustment parameters can utilize an error function and a controller to optimize the rate of change of the dynamic adjustment parameters, allowing the dynamic adjustment parameters to converge to the optimal value more quickly, improving the stability and continuity of the algorithm.

[0078] In one possible implementation, when determining the tracking result of the target object in the i-th frame infrared image based on the first detection box and the second prediction box, the first detection box and the second prediction box can be matched to determine the trajectory box of the target object; the trajectory box of the target object is the tracking result of the target object in the i-th frame infrared image.

[0079] In practical applications, infrared images may contain multiple target objects. Thus, when performing detection based on the first detection algorithm and the second prediction algorithm, multiple first detection boxes and multiple second prediction boxes can be detected. When matching the first detection boxes and the second prediction boxes, for example, the intersection-union ratio of the first detection boxes and the second prediction boxes, as well as the image feature distance between the first detection boxes and the second prediction boxes, can be combined to determine the first detection boxes and the second prediction boxes that contain the same target object. Then, the trajectory box of the target object is determined based on the first detection boxes and the second prediction boxes that contain the same target object.

[0080] Optionally, the step of determining the trajectory box of the target object can be performed by a Kalman filter. After the first detection algorithm determines the first detection box, it can input the first detection box into the Kalman filter. After determining the second prediction box, the Kalman filter can determine the trajectory box of the target object based on the first detection box and the second prediction box.

[0081] For example, when determining the trajectory box of a target object, a first detection box and a second predicted box containing the same target object can be weighted and fused. The weighted and fused trajectory box filters out random jitter in the detection and is smoother and more stable than the first detection box. Since it has been corrected by the true measurement value of the current frame (i.e., the first detection box), the weighted and fused trajectory box is more accurate than the second predicted box.

[0082] After the bounding box is determined in each frame of the infrared image, the Kalman filter can be updated. Specifically, the Kalman filter records information such as the aspect ratio (i.e., dynamically adjusted parameters), the position of the target object, and the current velocity of the target object. After the bounding box is determined, the aspect ratio, the position of the target object, and the current velocity of the target object can be updated. The information recorded by the updated Kalman filter can be used to predict the second prediction box in the next frame of the infrared image.

[0083] For example, the aircraft's attitude changes during flight, and the aircraft appears flattened. Taking time t as the current frame, time t-1 as the previous frame, and time t+1 as the next frame, the above target tracking method is introduced as an example.

[0084] 1. Time t-1: The aircraft is flying horizontally, appearing wider in the image. The aspect ratio (i.e., the dynamically adjusted parameter) of the Kalman filter in the DeepSORT algorithm is... =3.0 (width / height).

[0085] 2. Time t: Position prediction is performed using a Kalman filter to determine the aircraft's current second prediction box.

[0086] Before processing the current frame image, the DeepSORT algorithm makes predictions based on the state at time t-1. At this time, the Kalman filter still considers the aircraft to be wide, so the aspect ratio of the output second prediction box is also approximately 3.0.

[0087] 3. Time t: The DGRAD detection algorithm detects an aircraft in the t-th frame of the image, and the aspect ratio of the first detection box output is 5.0.

[0088] At time t, the aircraft may have changed its flight attitude, and the aircraft has become visually flatter. Therefore, the aspect ratio will change compared to the previous frame, as detected by the DGRAD algorithm.

[0089] 4. Time t: Based on the first detection box and the second prediction box at time t, calculate the intersection-over-union ratio, error function, controller, etc., and update the dynamically adjusted parameters, such as the updated... =4.0.

[0090] 5. Time t+1: At the current moment, the dynamic adjustment parameter of the Kalman filter has been adjusted from 3.0 to 4.0. Therefore, when detecting the second prediction box for the image of frame t+1, the aspect ratio of the second prediction box is "flatter" than before, and is closer to the shape of the aircraft currently displayed in the image.

[0091] If the aircraft's attitude continues to change, the process of updating and dynamically adjusting the parameters will continue. The parameters will be gradually adjusted to closely match the shape of the aircraft, thereby achieving stable tracking of small targets with changing shapes.

[0092] When determining the second prediction box, the Kalman filter can predict the position of the target object in the next frame based on the motion characteristics of the target object in the previous frame of the infrared image. The motion characteristics can be characterized by direction vectors. For example, the motion characteristics can include the direction and speed of the target object in the image coordinate system. Then, the position of the second prediction box is corrected (i.e., the trajectory box is determined) by the detection results of DGRAD.

[0093] Specifically, the Kalman filter can predict the target object's position in the next frame based on the target object's position information in the previous frame (represented by the trajectory bounding box of the previous frame), the target object's velocity and direction (i.e., direction vector) in the previous frame, and then correct the position of the second predicted box using the detection result of DGRAD in the next frame (i.e., the first detection box). However, since DGRAD detects the target object's position information in each frame of the image independently, if the position of the target object detected by DGRAD changes drastically due to interference, the Kalman filter's detection will be indirectly affected, leading to tracking failure.

[0094] Therefore, in order to address the problem that Kalman filters are susceptible to single-frame detection errors, leading to inaccurate trajectory prediction and tracking failure, this application also proposes a method for anomaly detection of the direction vector of the Kalman filter.

[0095] See Figure 2 The diagram shown is a flowchart of an anomaly detection method for direction vectors provided in this application, including the following steps:

[0096] S201. Obtain the first direction vector of the target object in the (i-1)th frame infrared image, and predict the second direction vector of the target object in the i-th frame infrared image based on the Kalman filter and the first direction vector.

[0097] Specifically, the state vector can contain the target object's position information and orientation vector, and the state vector can be represented as: ,in, This indicates the location information of the target object. This represents the velocity of the target object in the x and y directions, respectively; subsequent prediction and adjustment of the direction vector can be performed based on the state vector.

[0098] For example, it can be calculated using the following formula:

[0099] (4)

[0100] in, The second state vector represents the target object in the i-th frame of the infrared image, and this state vector includes a second direction vector. Let F represent the first state vector of the target object in the (i-1)th frame of the infrared image. This state vector includes a first direction vector. F represents the state transition matrix, which describes the physical laws governing the motion of the target object. For example, for a uniform motion model, if the time interval between two frames of infrared images is dt, then F can be expressed as:

[0101]

[0102] S202. Based on the first detection box of the target object in the i-th frame infrared image, adjust the second direction vector to obtain the third direction vector.

[0103] For example, the error covariance matrix of the i-th frame can be calculated first. This error covariance matrix represents the uncertainty measure for the predicted second state vector and can be calculated using the following formula:

[0104] (5)

[0105] in, Let i represent the error covariance matrix of the i-th frame. Let i represent the error covariance matrix of the (i-1)th frame. The process noise covariance matrix is ​​a fixed matrix pre-set based on experience. It is used to characterize the random deviation between the motion model (such as a uniform motion model) and the actual motion of the target object, ensuring that the uncertainty will increase appropriately during the prediction process.

[0106] Then, based on the error covariance matrix, the Kalman gain can be calculated. For example, it can be calculated using the following formula:

[0107] (6)

[0108] in, Let represent the Kalman gain of the i-th frame of the infrared image, which, according to the above formula, incorporates the uncertainty of the prediction (from...). (embodied) and measurement uncertainty (by) (This is reflected in) calculating an optimal weight. The measurement matrix is ​​determined by the position information detected by the DGRAD detector. The measurement noise covariance matrix is ​​a fixed matrix preset based on experience, representing the degree of confidence in the position information output by the DGRAD detector.

[0109] Then, based on the Kalman gain, the updated third state vector can be determined. The third state vector includes a third direction vector, which can be calculated, for example, using the following formula:

[0110] (7)

[0111] in, Represents the third state vector. This represents the location information of the target object in the i-th frame of the infrared image output by the DGRAD detector.

[0112] From the updated third state vector Extract the third-direction vector from the middle. .

[0113] The updated third-party vector can be obtained using the above formula. Furthermore, the error covariance matrix can be updated, and the updated error covariance matrix is ​​used to calculate the error covariance matrix for the next frame. For example, it can be calculated using the following formula:

[0114] (8)

[0115] in, 4 A 4-dimensional identity matrix.

[0116] It should be noted that for newly emerging targets, their initial state vector is determined by the first detection result of the DGRAD detector, with the velocity component being 0. The initial error covariance matrix P is set as a diagonal matrix, with larger values ​​on the diagonal indicating high uncertainty in the initial state, requiring rapid convergence based on subsequent measurements.

[0117] S203. Based on the direction vector of the target object in the N frames of infrared images before the i-th frame, perform anomaly detection on the third direction vector; N is a positive integer greater than 1.

[0118] In one possible implementation, when performing anomaly detection on the third-party direction vector based on the direction vector of the target object in the N frames of infrared images before the i-th frame, the mean vector and standard deviation vector can be determined based on the direction vector of the target object in the N frames of infrared images before the i-th frame; and then anomaly detection is performed on the third-party direction vector based on the mean vector and the standard deviation vector.

[0119] For example, we can first maintain a historical queue of direction vectors from the past N frames that have been identified as normal and ultimately adopted. .

[0120] Then, a motion model baseline is established, which includes the mean vector. and standard deviation vector For example, it can be calculated using the following formula:

[0121] (9)

[0122] (10)

[0123] When performing anomaly detection on the third-party vector based on the mean vector and the standard deviation vector, the error score of the third-party vector relative to the mean vector and the standard deviation vector can be calculated. If the error score exceeds a preset threshold, the third-party vector of the i-th frame is determined to be an anomaly.

[0124] For example, the error fraction can be calculated using the following formula:

[0125] (11)

[0126] in, This represents the error fraction.

[0127] S204. If an anomaly is detected, the third-party directional vector is corrected.

[0128] In one possible implementation, when correcting a third-direction vector, the third-direction vector can be corrected using any of the following methods:

[0129] (1) The median of the direction vector of the target object in the first K frames of infrared images is used as the corrected third direction vector; K is a positive integer greater than 1;

[0130] (2) Increase the value of the measurement noise covariance matrix in the Kalman filter; wherein the measurement noise covariance matrix is ​​used to characterize the degree of trust in the location information of the target object detected by the first detection algorithm;

[0131] Here, the measurement noise covariance matrix is ​​the same as that in formula (6) above. By increasing the measurement noise covariance matrix The value of can reduce the weight of subsequent abnormal measurements.

[0132] (3) Reinitialize the Kalman filter.

[0133] Finally, the third-order vector, after the above detection and correction, can be used to update the final state of the Kalman filter (i.e., update the information recorded by the Kalman filter) and can serve as the basis for predicting the position of the second prediction box in the next frame. In this way, the irreversible damage to the entire tracking trajectory caused by a single erroneous detection result can be effectively avoided, thus ensuring the continuity and accuracy of tracking small moving targets in complex cloud scenarios.

[0134] The following section provides a brief overview of the target tracking method described above, using the overall block diagram as an example. (See also...) Figure 3 The diagram shown is an overall block diagram of a target tracking method provided in this application. First, an infrared image is acquired from an infrared image data source. Then, the infrared image is processed by Gaussian curvature filtering. Specifically, the filtering parameters (including standard deviation, filtering radius, etc.) can be determined through an adaptive parameter adjustment scheme based on noise level and contrast. Then, the filtered image is detected by DGRAD. Based on the detection results of DGRAD, DeepSORT tracking is performed, including anomaly detection processing of direction vectors and aspect ratio parameter adjustment based on feedback control. Finally, the tracking result is output. The tracking result can include the trajectory information and motion information of the target object. The motion information includes the position of the target object (represented by the target object's trajectory box), as well as its velocity and direction (represented by a third-party direction vector).

[0135] For a detailed explanation of the above steps, please refer to the above embodiment, which will not be repeated here.

[0136] Corresponding to the embodiments of the aforementioned target tracking method, this application also provides embodiments of a target tracking device.

[0137] See Figure 4 The diagram shown is a schematic representation of the architecture of a target tracking device provided in this application, comprising:

[0138] Acquisition module 401 is used to acquire multiple frames of infrared images;

[0139] The determining module 402 is configured to, for the i-th frame of infrared image, determine a first detection box of the target object contained in the i-th frame of infrared image based on a first detection algorithm in the target tracking algorithm, and determine a second prediction box of the target object contained in the i-th frame of infrared image based on a second prediction algorithm in the target tracking algorithm; wherein, the first detection algorithm is an algorithm for detection based on image features of the infrared image; the second prediction algorithm is an algorithm for prediction based on motion features of the target object in the infrared image; the second prediction algorithm includes dynamic adjustment parameters, the dynamic adjustment parameters corresponding to the i-th frame of infrared image are parameters for adjusting the aspect ratio of the second prediction box of the i-th frame of infrared image determined based on the first detection box and the second prediction box corresponding to the (i-1)-th frame of infrared image; i is a positive integer;

[0140] The tracking module 403 is used to determine the tracking result of the target object in the i-th frame infrared image based on the first detection box and the second prediction box.

[0141] In one possible implementation, the target tracking algorithm is the DeepSORT algorithm, which is a deep simple online real-time tracking algorithm, and the second prediction algorithm is a Kalman filter.

[0142] In one possible implementation, the determining module 402 is further configured to determine the dynamic adjustment parameters corresponding to the i-th frame infrared image according to the following method:

[0143] Determine the intersection-union ratio (I / O) between the first detection box and the second prediction box corresponding to the (i-1)th frame of the infrared image;

[0144] Based on the intersection-union ratio, the error function between the first detection box and the second prediction box corresponding to the (i-1)th frame infrared image is determined;

[0145] Based on the error function and the controller, the dynamic adjustment parameters corresponding to the i-th frame infrared image are determined; wherein, the controller is used to control the rate of change of the shape of the second prediction box.

[0146] In one possible implementation, the Kalman filter is used to predict the position of the target object in the next frame based on the motion characteristics of the target object in the previous frame of the infrared image, wherein the motion characteristics are characterized by a direction vector;

[0147] The device also includes a correction module 404:

[0148] Obtain the first direction vector of the target object in the (i-1)th frame of the infrared image, and predict the second direction vector of the target object in the i-th frame of the infrared image based on the Kalman filter and the first direction vector;

[0149] Based on the first detection box of the target object in the i-th frame infrared image, the second direction vector is adjusted to obtain the third direction vector;

[0150] Based on the orientation vector of the target object in the N frames of infrared images preceding the i-th frame, anomaly detection is performed on the third orientation vector; N is a positive integer greater than 1.

[0151] If an anomaly is detected, the third-party directional vector is corrected.

[0152] In one possible implementation, the correction module 404, when performing anomaly detection on the third direction vector based on the direction vector of the target object in the N frames of infrared images preceding the i-th frame, is used to:

[0153] Based on the orientation vector of the target object in the N frames of infrared images before the i-th frame, determine the mean vector and standard deviation vector;

[0154] Anomaly detection is performed on the third-party vector based on the mean vector and the standard deviation vector.

[0155] In one possible implementation, the correction of the third-party vector includes correcting the third-party vector according to any one of the following methods:

[0156] The median of the orientation vector of the target object in the first K frames of infrared images is used as the corrected third orientation vector; K is a positive integer greater than 1;

[0157] Increase the value of the measurement noise covariance matrix in the Kalman filter; wherein, the measurement noise covariance matrix is ​​used to characterize the degree of trust in the location information of the target object detected by the first detection algorithm;

[0158] Reinitialize the Kalman filter.

[0159] In one possible implementation, the tracking module 403, when determining the tracking result of the target object in the i-th frame infrared image based on the first detection box and the second prediction box, is used to:

[0160] The first detection box and the second prediction box are matched to determine the trajectory box of the target object; the trajectory box of the target object is the tracking result of the target object in the i-th frame infrared image.

[0161] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0162] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0163] This application also provides a computer-readable storage medium storing a computer program that can be used to execute the target tracking method described in the above embodiments.

[0164] This application also provides a computer device, see [link to relevant documentation] Figure 5 The diagram shown is a structural schematic of the computer device provided in this application. At the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the target tracking method described in the above embodiments. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0165] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0166] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0167] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0168] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0169] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0170] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0171] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0172] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A target tracking method, characterized in that, The method includes: Acquire multiple frames of infrared images; For the i-th frame of infrared image, a first detection box of the target object contained in the i-th frame of infrared image is determined based on a first detection algorithm in the target tracking algorithm, and a second prediction box of the target object contained in the i-th frame of infrared image is determined based on a second prediction algorithm in the target tracking algorithm; wherein, the first detection algorithm is an algorithm for detection based on image features of the infrared image; the second prediction algorithm is an algorithm for prediction based on motion features of the target object in the infrared image; the second prediction algorithm includes dynamic adjustment parameters, and the dynamic adjustment parameters corresponding to the i-th frame of infrared image are parameters used to adjust the aspect ratio of the second prediction box of the i-th frame of infrared image, determined based on the first detection box and the second prediction box corresponding to the (i-1)-th frame of infrared image; i is a positive integer; Based on the first detection box and the second prediction box, the tracking result of the target object in the i-th frame infrared image is determined; The dynamic adjustment parameters corresponding to the i-th frame of the infrared image are determined according to the following method: Determine the intersection-union ratio (I / O) between the first detection box and the second prediction box corresponding to the (i-1)th frame of the infrared image; Based on the intersection-union ratio, the error function between the first detection box and the second prediction box corresponding to the (i-1)th frame infrared image is determined; Based on the error function and the controller, the dynamic adjustment parameters corresponding to the i-th frame infrared image are determined; wherein, the controller is used to control the rate of change of the shape of the second prediction box.

2. The method according to claim 1, characterized in that, The target tracking algorithm is the DeepSORT algorithm, which is a simple online real-time tracking algorithm, and the second prediction algorithm is a Kalman filter.

3. The method according to claim 2, characterized in that, The Kalman filter is used to predict the position of the target object in the next frame based on the motion characteristics of the target object in the previous frame of the infrared image. The motion characteristics are represented by a direction vector. The method further includes: Obtain the first direction vector of the target object in the (i-1)th frame of the infrared image, and predict the second direction vector of the target object in the i-th frame of the infrared image based on the Kalman filter and the first direction vector; Based on the first detection box of the target object in the i-th frame infrared image, the second direction vector is adjusted to obtain the third direction vector; Based on the orientation vector of the target object in the N frames of infrared images preceding the i-th frame, anomaly detection is performed on the third orientation vector; N is a positive integer greater than 1. If an anomaly is detected, the third-party directional vector is corrected.

4. The method according to claim 3, characterized in that, The anomaly detection of the third direction vector based on the direction vector of the target object in the N frames of infrared images preceding the i-th frame includes: Based on the orientation vector of the target object in the N frames of infrared images before the i-th frame, determine the mean vector and standard deviation vector; Anomaly detection is performed on the third-party vector based on the mean vector and the standard deviation vector.

5. The method according to claim 3, characterized in that, The correction of the third-party vector includes correcting the third-party vector according to any one of the following methods: The median of the orientation vector of the target object in the first K frames of infrared images is used as the corrected third orientation vector; K is a positive integer greater than 1; Increase the value of the measurement noise covariance matrix in the Kalman filter; wherein, the measurement noise covariance matrix is ​​used to characterize the degree of trust in the location information of the target object detected by the first detection algorithm; Reinitialize the Kalman filter.

6. The method according to claim 1, characterized in that, The step of determining the tracking result of the target object in the i-th frame infrared image based on the first detection box and the second prediction box includes: The first detection box and the second prediction box are matched to determine the trajectory box of the target object; the trajectory box of the target object is the tracking result of the target object in the i-th frame infrared image.

7. A target tracking device, characterized in that, The device includes: The acquisition module is used to acquire multiple frames of infrared images; A determination module is configured to, for an i-th frame of infrared image, determine a first detection box of a target object contained in the i-th frame of infrared image based on a first detection algorithm in the target tracking algorithm, and determine a second prediction box of the target object contained in the i-th frame of infrared image based on a second prediction algorithm in the target tracking algorithm; wherein, the first detection algorithm is an algorithm for detection based on image features of the infrared image; the second prediction algorithm is an algorithm for prediction based on motion features of the target object in the infrared image; the second prediction algorithm includes dynamic adjustment parameters, the dynamic adjustment parameters corresponding to the i-th frame of infrared image being parameters determined based on the first detection box and the second prediction box corresponding to the (i-1)-th frame of infrared image, used to adjust the aspect ratio of the second prediction box of the i-th frame of infrared image; i is a positive integer; The tracking module is used to determine the tracking result of the target object in the i-th frame infrared image based on the first detection box and the second prediction box; The determining module is further configured to determine the dynamic adjustment parameters corresponding to the i-th frame infrared image according to the following method: Determine the intersection-union ratio (I / O) between the first detection box and the second prediction box corresponding to the (i-1)th frame of the infrared image; Based on the intersection-union ratio, the error function between the first detection box and the second prediction box corresponding to the (i-1)th frame infrared image is determined; Based on the error function and the controller, the dynamic adjustment parameters corresponding to the i-th frame infrared image are determined; wherein, the controller is used to control the rate of change of the shape of the second prediction box.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the target tracking method as described in any one of claims 1 to 6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the target tracking method as described in any one of claims 1 to 6.

Images