Target tracking method and device, computer equipment and storage medium

Abstract

The invention provides a target tracking method and device, computer equipment and a storage medium. The method comprises the steps that multiple frames of infrared images are acquired; for the ith frame of infrared image, determining a first detection frame of a target object contained in the ith frame of infrared image based on a first detection algorithm in target tracking algorithms, and determining a second prediction frame of the target object contained in the ith frame of infrared image based on a second prediction algorithm in the target tracking algorithms; wherein the second prediction algorithm comprises a dynamic adjustment parameter, and the dynamic adjustment parameter corresponding to the ith frame of infrared image is a parameter which is determined based on a first detection frame and a second prediction frame corresponding to the (i-1) th frame of infrared image and is used for adjusting the length-width ratio of the second prediction frame of the ith frame of infrared image; i is a positive integer; and determining a tracking result of the target object in the ith frame of infrared image based on the first detection frame and the second prediction frame. In this way, the accuracy of infrared target detection and tracking can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and more specifically to a target tracking method, apparatus, computer equipment, and storage medium. Background Art

[0002] Infrared detection and tracking technology, with its advantages of long detection range and strong anti-interference capabilities, is widely used in military and civilian fields, including missile defense, aerospace, and security. However, the application of infrared small target detection and tracking technology faces many complex constraints. Targets are imaged at long distances, occupy a small portion of the image, lack texture features, and are susceptible to clutter and noise. Therefore, detecting and tracking multiple weak infrared targets with low signal-to-noise ratio in complex cloud scenes is a challenging task. Summary of the Invention

[0003] In view of this, the present application provides a target tracking method, apparatus, computer equipment and storage medium to improve the accuracy of infrared target detection and tracking.

[0004] In a first aspect, the present application provides a target tracking method, comprising: Acquire multiple frames of infrared images; For the i-th infrared image frame, a first detection frame of the target object contained in the i-th infrared image frame is determined based on a first detection algorithm in a target tracking algorithm, and a second prediction frame of the target object contained in the i-th infrared image frame is determined based on a second prediction algorithm in the target tracking algorithm; wherein the first detection algorithm is an algorithm for detection based on image features of the infrared image; the second prediction algorithm is an algorithm for prediction based on motion features of the target object in the infrared image; the second prediction algorithm includes a dynamic adjustment parameter, and the dynamic adjustment parameter corresponding to the i-th infrared image frame is a parameter determined based on the first detection frame and the second prediction frame corresponding to the i-1-th infrared image frame, and is used to adjust the aspect ratio of the second prediction frame of the i-th infrared image frame; i is a positive integer; Based on the first detection frame and the second prediction frame, a tracking result of the target object in the i-th infrared image frame is determined.

[0005] In a possible implementation, the target tracking algorithm is a Deep Simple Online Real-time Tracking (DeepSORT) algorithm, and the second prediction algorithm is a Kalman filter.

[0006] In a possible implementation, the method further includes determining the dynamic adjustment parameter corresponding to the i-th infrared image frame according to the following method, including: Determine an intersection-over-union ratio between a first detection frame and the second prediction frame corresponding to the (i-1)th infrared image frame; Determining an error function between a first detection frame and the second prediction frame corresponding to the (i-1)th infrared image frame based on the intersection-over-union ratio; Based on the error function and the controller, a dynamic adjustment parameter corresponding to the i-th frame infrared image is determined; wherein the controller is used to control the changing speed of the second prediction box shape.

[0007] In one possible implementation, the Kalman filter is used to predict the position of the target object in the next frame of infrared image based on the motion characteristics of the target object in the previous frame, where the motion characteristics are represented by a direction vector; The method further comprises: Obtaining a first direction vector of the target object in the (i-1)th infrared image frame, and predicting a second direction vector of the target object in the (i)th infrared image frame based on the Kalman filter and the first direction vector; Adjusting the second direction vector based on a first detection frame of the target object in the i-th infrared image frame to obtain a third direction vector; Based on the direction vector of the target object in the N infrared image frames before the i-th frame, anomaly detection is performed on the third direction vector; N is a positive integer greater than 1; When an abnormality is detected, the third direction vector is corrected.

[0008] In one possible implementation, the performing abnormality detection on the third direction vector based on the direction vector of the target object in the infrared image frames N frames before the i-th frame includes: Determine a mean vector and a standard deviation vector based on the direction vector of the target object in the N infrared image frames before the i-th frame; Anomaly detection is performed on the third direction vector based on the mean vector and the standard deviation vector.

[0009] In a possible implementation, the correcting the third direction vector includes correcting the third direction vector according to any one of the following methods: The median of the direction vectors of the target object in the first K frames of infrared images is used as the corrected third direction vector; K is a positive integer greater than 1; Increasing a value of a measurement noise covariance matrix in the Kalman filter; wherein the measurement noise covariance matrix is ​​used to represent a degree of trust in the position information of the target object detected by the first detection algorithm; Reinitialize the Kalman filter.

[0010] In a possible implementation, determining a tracking result of the target object in the i-th infrared image frame based on the first detection frame and the second prediction frame includes: The first detection frame and the second prediction frame are matched to determine a trajectory frame of the target object; the trajectory frame of the target object is a tracking result of the target object in the i-th frame infrared image.

[0011] In a second aspect, the present application further provides a target tracking device, comprising: An acquisition module, used for acquiring multiple frames of infrared images; A determination module is configured to determine, for an i-th infrared image frame, a first detection frame of a target object contained in the i-th infrared image frame based on a first detection algorithm in a target tracking algorithm, and to determine a second prediction frame of the target object contained in the i-th infrared image frame based on a second prediction algorithm in the target tracking algorithm; wherein the first detection algorithm is an algorithm for detection based on image features of the infrared image; the second prediction algorithm is an algorithm for prediction based on motion features of the target object in the infrared image; the second prediction algorithm includes a dynamic adjustment parameter, and the dynamic adjustment parameter corresponding to the i-th infrared image frame is a parameter determined based on the first detection frame and the second prediction frame corresponding to the i-1-th infrared image frame, and is used to adjust the aspect ratio of the second prediction frame of the i-th infrared image frame; and i is a positive integer; A tracking module is used to determine a tracking result of the target object in the i-th infrared image frame based on the first detection frame and the second prediction frame.

[0012] In a third aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the target tracking method as described in the first aspect or any possible implementation method of the first aspect.

[0013] In a fourth aspect, the present application also provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the target tracking method as described in the first aspect or any possible implementation scheme of the first aspect are implemented.

[0014] In the target tracking method, apparatus, computer device, and storage medium provided in the present application, after acquiring multiple frames of infrared images, a first detection frame containing the target object can be determined based on a first detection algorithm of the target tracking algorithm, and a second prediction frame containing the target object can be determined based on a second prediction algorithm of the target tracking algorithm. The tracking result of the target object is then determined based on the first detection frame and the second prediction frame. Because the dynamic adjustment parameters in the second prediction algorithm are determined based on the first detection frame and the second prediction frame of the previous infrared image frame, when the second prediction algorithm predicts the second prediction frame based on the dynamic adjustment parameters, the aspect ratio of the second prediction frame can be closer to the current shape of the target object, thereby improving the accuracy of infrared target detection and tracking during subsequent target object detection and tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of a target tracking method shown in an exemplary embodiment of the present application; Figure 2 This is a flow chart of a method for detecting anomalies in a direction vector according to an exemplary embodiment of the present application; Figure 3 This is an overall block diagram of a target tracking method shown in an exemplary embodiment of the present application; Figure 4 is a schematic diagram of the architecture of a target tracking device shown in an exemplary embodiment of the present application; Figure 5 It is a structural diagram of a computer device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0016] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0017] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0018] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0019] Infrared detection and tracking technology, with its advantages of long detection range and strong anti-interference capabilities, is widely used in military and civilian fields, including missile defense, aerospace, and security. However, the application of infrared small target detection and tracking technology faces many complex constraints. Targets are imaged at long distances, occupy a small portion of the image, lack texture features, and are susceptible to clutter and noise. Therefore, detecting and tracking multiple weak infrared targets with low signal-to-noise ratio in complex cloud scenes is a challenging task.

[0020] In related technologies, infrared target detection is generally performed using the Deep Simple Online and Realtime Tracking (DeepSORT) algorithm.

[0021] The overall process of the DeepSORT algorithm can be divided into the following steps: 1. Object detection: Use an object detector (such as YOLO, Faster R-CNN, etc.) to detect the infrared image frame and obtain the detection frame of the tracked object.

[0022] ‌2. Feature extraction‌: Use a deep learning model to extract the appearance features of the target, which are used in the subsequent matching process.

[0023] ‌3. Kalman filter prediction‌: Use the Kalman filter to predict the position of the tracked target in the next frame and obtain the predicted frame of the tracked target.

[0024] ‌4. Hungarian algorithm matching‌: Use the Hungarian algorithm to calculate the optimal match between the detection box and the prediction box.

[0025] ‌5. Track management‌: Maintain the track of each tracked target, including track initialization, update, and deletion.

[0026] During infrared target detection and tracking, the size / shape of the target may change. For example, when an aircraft flies in different postures, it will appear in different sizes on the infrared image.

[0027] According to the above process, one of the key steps of the DeepSORT algorithm lies in the matching process between the detection frame and the prediction frame. In the above target detection process, detection is based on the image features of the infrared image frame. Therefore, the aspect ratio of the detection frame is closer to the shape of the tracking target. However, the Kalman filter determines the tracking target's prediction frame based on the tracking target's motion characteristics. The size of the prediction frame may deviate significantly from the current tracking target. As a result, when matching the detection frame and the prediction frame, they may not match, which in turn affects the tracking process.

[0028] Based on this, the present application provides a target tracking method, apparatus, computer device, and storage medium. After acquiring multiple frames of infrared images, a first detection frame containing a target object can be determined based on a first detection algorithm of the target tracking algorithm, and a second prediction frame containing the target object can be determined based on a second prediction algorithm of the target tracking algorithm. Then, the tracking result of the target object is determined based on the first detection frame and the second prediction frame. Since the dynamic adjustment parameters in the second prediction algorithm are determined based on the first detection frame and the second prediction frame of the previous frame of infrared image, when the second prediction algorithm predicts the second prediction frame based on the dynamic adjustment parameters, the aspect ratio of the second prediction frame can be closer to the current shape of the target object, thereby improving the accuracy of infrared target detection and tracking when performing subsequent target object detection and tracking.

[0029] The target tracking method provided by this application will be described in detail below with reference to specific embodiments. Figure 1 FIG. 1 is a flow chart of a target tracking method provided by this application, which includes the following steps: S101: Acquire multiple frames of infrared images.

[0030] Here, the acquired multiple frames of infrared images may be multiple frames of continuous infrared images.

[0031] Optionally, after acquiring multiple infrared image frames, the infrared images can be preprocessed. For example, the infrared images can be enhanced using a Gaussian curvature filter. This filter is more flexible in processing local edge features and can highlight the curvature difference between the target object and the background, significantly enhancing the discernibility of small targets (i.e., target objects) in infrared images. This makes the small targets stand out more clearly in the infrared image, laying the foundation for subsequent detection and tracking steps.

[0032] Optionally, before preprocessing the infrared image, the parameters of the Gaussian curvature filter can be determined based on the size of the target object in the infrared image and the noise level of the image, and then the infrared image can be preprocessed through the Gaussian curvature filter according to the parameters to determine the features that can highlight the small target in different scenes.

[0033] S102. For the i-th infrared image frame, determine a first detection frame of the target object contained in the i-th infrared image frame based on a first detection algorithm in the target tracking algorithm, and determine a second prediction frame of the target object contained in the i-th infrared image frame based on a second prediction algorithm in the target tracking algorithm.

[0034] Among them, the first detection algorithm is an algorithm for detection based on the image features of the infrared image; the second prediction algorithm is an algorithm for prediction based on the motion features of the target object in the infrared image; the second prediction algorithm includes dynamic adjustment parameters, and the dynamic adjustment parameters corresponding to the i-th frame infrared image are parameters determined based on the first detection frame and the second prediction frame corresponding to the i-1-th frame infrared image, and are used to adjust the shape of the second prediction frame of the i-th frame infrared image; i is a positive integer.

[0035] Optionally, the target tracking algorithm may be a DeepSORT algorithm; the first detection algorithm may be a detection algorithm based on human vision, such as a Double-Neighborhood Gradient (DGRAD) algorithm or a neural network-based detection algorithm; and the second prediction algorithm may be a Kalman filter. The following embodiments will use the DGRAD algorithm as an example for the first detection algorithm, but the first detection algorithm in this application is not limited to the DGRAD algorithm.

[0036] The first detection frame can be a real observation value, which is used to reflect the actual detection position of the target object in the current frame, but the actual detection position has certain noise and errors; the second prediction frame is a prediction value, which is the inference result obtained by the Kalman filter through a motion model (such as a uniform motion model), which is used to reflect the expected position of the target object and is an estimate of the future position of the target object.

[0037] S103 : Determine a tracking result of the target object in the i-th infrared image frame based on the first detection frame and the second prediction frame.

[0038] In one possible implementation, when determining the dynamic adjustment parameters corresponding to the i-th frame infrared image, the intersection-and-union ratio between the first detection frame corresponding to the i-1-th frame infrared image and the second prediction frame can be determined first; then, based on the intersection-and-union ratio, the error function between the first detection frame corresponding to the i-1-th frame infrared image and the second prediction frame can be determined; then, based on the error function and a controller, the dynamic adjustment parameters corresponding to the i-th frame infrared image can be determined; wherein, the controller is used to control the speed of change of the shape of the second prediction frame.

[0039] For example, the error function can be calculated by the following formula: (1) in, represents the error function, represents the second prediction box, represents the first detection frame, represents the intersection-over-union (IoU) of the first detection box and the second prediction box.

[0040] In the above formula, when the error between the first detection frame and the second prediction frame is large, that is, when the intersection-over-union ratio between the first detection frame and the second prediction frame is low, the value of the error function increases exponentially, which facilitates a rapid response to the situation when the size difference between the first detection frame and the second prediction frame is too large; when the error between the first detection frame and the second prediction frame is small, that is, when the intersection-over-union ratio between the first detection frame and the second prediction frame is high, the change of the error function tends to be stable, which improves the moderate tolerance for slight errors.

[0041] The controller can be expressed by the following formula: (2) in, Represents the controller, It is the preset positive proportional coefficient.

[0042] The controller can be used to characterize the change speed of the shape of the second prediction frame, that is, the change amount of the dynamic adjustment parameter of the second prediction frame per unit time. This parameter can reflect the motion characteristics of the shape of the target object during the movement process.

[0043] The dynamic adjustment parameters can be calculated using the following formula: (3) in, represents the dynamic adjustment parameters of the i-th frame infrared image, represents the dynamic adjustment parameters of the i-1th frame infrared image, Represents the time interval between the i-th infrared image frame and the i-th infrared image frame.

[0044] The dynamic adjustment parameter may be used to characterize the shape of the detection frame, that is, the dynamic adjustment parameter may be a ratio of the width of the detection frame divided by the height, and the parameter may reflect the appearance characteristics of the target object.

[0045] The dynamic adjustment parameters in this application are determined based on the matching of the first detection frame and the second prediction frame in the previous frame. Therefore, the dynamic adjustment parameters in this application can be understood as feedback control based on the tracking results. The dynamic adjustment parameters based on feedback control can adjust the shape of the detection frame in real time according to the target object and scene, so that the detection frame can better adapt to the size / shape changes of the target object, and improve the accuracy and robustness of tracking the target object in deformation scenes. On the other hand, the dynamic adjustment parameters based on feedback control can use the error function and the controller to optimize the rate of change of the dynamic adjustment parameters, so that the dynamic adjustment parameters can converge to the optimal value faster, thereby improving the stability and continuity of the algorithm.

[0046] In one possible implementation, when determining the tracking result of the target object in the i-th frame infrared image based on the first detection frame and the second prediction frame, the first detection frame and the second prediction frame can be matched to determine the trajectory frame of the target object; the trajectory frame of the target object is the tracking result of the target object in the i-th frame infrared image.

[0047] In practical applications, an infrared image may contain multiple target objects. In this way, when detecting based on the first detection algorithm and the second prediction algorithm, multiple first detection frames and multiple second prediction frames can be detected. When matching the first detection frame and the second prediction frame, the intersection-over-union ratio of the first detection frame and the second prediction frame, as well as the image feature distance between the first detection frame and the second prediction frame, can be combined to determine the first detection frame and the second prediction frame containing the same target object, and then the trajectory frame of the target object can be determined based on the first detection frame and the second prediction frame containing the same target object.

[0048] Optionally, the step of determining the trajectory box of the target object can be performed by a Kalman filter. After determining the first detection box, the first detection algorithm can input the first detection box into the Kalman filter. After determining the second prediction box, the Kalman filter can determine the trajectory box of the target object based on the first detection box and the second prediction box.

[0049] For example, when determining the trajectory frame of a target object, a first detection frame and a second prediction frame containing the same target object can be weightedly fused. The weighted fused trajectory frame filters out random jitter from detection and is smoother and more stable than the first detection frame. Because it is corrected by the actual measurement value of the current frame (i.e., the first detection frame), the weighted fused trajectory frame is more accurate than the second prediction frame.

[0050] After determining the trajectory frame for each infrared image frame, the Kalman filter can be updated. Specifically, the Kalman filter records information such as the aspect ratio (a dynamically adjusted parameter) as well as the target object's position and current speed. After determining the trajectory frame, these information can be updated. The updated Kalman filter records this information and can be used to predict the second prediction frame for the next infrared image frame.

[0051] For example, the posture of the aircraft will change during flight, and the aircraft will become flatter visually. The above target tracking method is exemplarily introduced with time t as the current frame, time t-1 as the previous frame, and time t+1 as the next frame.

[0052] 1. Time t-1: The aircraft is flying horizontally, and the image is wider. The aspect ratio (i.e., the dynamic adjustment parameter) of the Kalman filter of the DeepSORT algorithm is =3.0 (width / height).

[0053] 2. Time t: Use the Kalman filter to predict the position and determine the current second prediction frame of the aircraft.

[0054] Before processing the current frame image, the DeepSORT algorithm makes a prediction based on the state at time t-1. At this moment, the Kalman filter still believes that the aircraft is wider, so the aspect ratio of the output second prediction box is also approximately equal to 3.0.

[0055] 3. Time t: The DGRAD detection algorithm detects the aircraft in the t-th frame image, and the aspect ratio of the first detection frame output is 5.0.

[0056] At the tth moment, the aircraft may have changed its flight attitude and become visually flattened. Therefore, through the DGRAD algorithm detection, the aspect ratio will change compared with the previous frame.

[0057] 4. Time t: Based on the first detection frame and the second prediction frame at time t, calculate the intersection-over-union ratio, error function, controller, etc., and update the dynamic adjustment parameters, such as the updated =4.0.

[0058] 5. Time t+1: At this moment, the dynamic adjustment parameters of the Kalman filter have been adjusted from 3.0 to 4.0. Therefore, when detecting the second prediction box for the t+1 frame image, the aspect ratio of the second prediction box is flatter than before, closer to the shape of the aircraft currently displayed in the image.

[0059] If the aircraft's attitude continues to change, the above process of updating the dynamic adjustment parameters will continue, and the dynamic adjustment parameters will gradually adjust to be close to the shape of the aircraft, thereby achieving stable tracking of small targets with changing shapes.

[0060] When determining the second prediction frame, the Kalman filter can predict the position of the target object in the next frame based on the motion characteristics of the target object in the previous frame of infrared image. The motion characteristics can be represented by a direction vector. The motion characteristics can exemplarily include the motion direction and speed of the target object in the image coordinate system. The position of the second prediction frame is then corrected using the detection results of DGRAD (i.e., the trajectory frame is determined).

[0061] Specifically, the Kalman filter can predict the position of the target object in the next frame based on the target object's position information in the previous frame (which can be represented by the trajectory box of the previous frame), the target object's speed and direction (i.e., the direction vector) in the previous frame, and then correct the position of the second predicted frame based on the detection results of the DGRAD in the next frame (i.e., the first detection frame). However, since DGRAD detects the position information of the target object in each frame of the image independently, if the position of the target object detected by DGRAD changes drastically due to interference, the Kalman filter's detection will be indirectly affected, resulting in tracking failure.

[0062] Therefore, in order to solve the problem that the Kalman filter is susceptible to single-frame detection errors, resulting in inaccurate trajectory prediction and tracking failure, the present application also proposes a method for detecting abnormalities in the direction vector of the Kalman filter.

[0063] See also Figure 2 FIG. 1 is a flow chart of a method for detecting anomalies in a direction vector provided by the present application, comprising the following steps: S201 , obtaining a first direction vector of the target object in the (i-1)th infrared image frame, and predicting a second direction vector of the target object in the (i)th infrared image frame based on the Kalman filter and the first direction vector.

[0064] Specifically, the state vector may include the location information and direction vector of the target object. The state vector may be represented as: ,in, Indicates the location information of the target object. Indicates the speed of the target object in the x-direction and y-direction respectively; subsequent prediction and adjustment of the direction vector can be performed based on the state vector.

[0065] For example, it can be calculated by the following formula: (4) in, represents a second state vector of the target object in the i-th infrared image frame, wherein the state vector includes a second direction vector, represents the first state vector of the target object in the infrared image frame i-1, which includes the first direction vector. F represents the state transfer matrix, which is used to describe the physical laws of the target object's motion. For example, for a uniform motion model, if the time interval between two infrared image frames is dt, then F can be expressed as:

[0066] S202 : Based on the first detection frame of the target object in the i-th infrared image frame, adjust the second direction vector to obtain a third direction vector.

[0067] For example, the error covariance matrix of the i-th frame may be calculated first. The error covariance matrix is ​​used to represent the uncertainty measure for the predicted second state vector and may be calculated using the following formula: (5) in, represents the error covariance matrix of the i-th frame, represents the error covariance matrix of the i-1th frame, It represents the process noise covariance matrix, which is a fixed matrix preset based on experience. It is used to characterize the random deviation between the motion model (such as the uniform motion model) and the actual motion of the target object, ensuring that the uncertainty will increase moderately during the prediction process.

[0068] Then, the Kalman gain can be calculated based on the error covariance matrix. For example, it can be calculated using the following formula: (6) in, represents the Kalman gain of the infrared image of the i-th frame, which integrates the uncertainty of the prediction (given by and measurement uncertainty (reflected by embodied), is to calculate an optimal weight, represents the measurement matrix, which is determined by the position information detected by the DGRAD detector, It represents the measurement noise covariance matrix, which is a fixed matrix preset based on experience and represents the degree of trust in the position information output by the DGRAD detector.

[0069] Then, an updated third state vector can be determined according to the Kalman gain. The third state vector includes a third direction vector. Exemplarily, it can be calculated by the following formula: (7) in, represents the third state vector, It represents the location information of the target object in the i-th infrared image output by the DGRAD detector.

[0070] From the updated third state vector Extract the third direction vector .

[0071] The updated third direction vector can be obtained through the above formula. In addition, the above error covariance matrix can be updated. The updated error covariance matrix is ​​used to calculate the error covariance matrix of the next frame. Exemplarily, it can be calculated by the following formula: (8) in, 4 4's identity matrix.

[0072] It should be noted that for a newly appeared target, its initial state vector is determined by the first detection result of the DGRAD detector, and the velocity component is 0. The initial error covariance matrix P is set to a diagonal matrix, and the values ​​on the diagonal are set large, indicating that the initial state has high uncertainty and needs to rely on subsequent measurements for rapid convergence.

[0073] S203 . Based on the direction vector of the target object in the infrared image frames N frames before the i-th frame, perform anomaly detection on the third direction vector; N is a positive integer greater than 1.

[0074] In one possible implementation, when performing anomaly detection on the third direction vector based on the direction vector of the target object in the N infrared image frames before the i-th frame, the mean vector and the standard deviation vector can be determined based on the direction vector of the target object in the N infrared image frames before the i-th frame; and then, the third direction vector can be detected for anomaly based on the mean vector and the standard deviation vector.

[0075] For example, a historical queue of direction vectors that have been identified as normal and finally adopted in the past N frames can be maintained. .

[0076] Then establish the motion model baseline, which includes the mean vector and standard deviation vector , can be calculated by the following formula: (9) (10) When performing anomaly detection on the third direction vector based on the mean vector and the standard deviation vector, an error score of the third direction vector relative to the mean vector and the mean square error vector can be calculated. When the error score exceeds a preset threshold, the third direction vector of the i-th frame is determined to be abnormal.

[0077] For example, the error score can be calculated by the following formula: (11) in, Represents the error score.

[0078] S204: When an abnormality is detected, correct the third direction vector.

[0079] In a possible implementation, when correcting the third direction vector, the third direction vector may be corrected by any of the following methods: (1) The median of the direction vector of the target object in the first K frames of infrared image is used as the corrected third direction vector; K is a positive integer greater than 1; (2) increasing the value of the measurement noise covariance matrix in the Kalman filter; wherein the measurement noise covariance matrix is ​​used to represent the degree of trust in the position information of the target object detected by the first detection algorithm; Here, the measurement noise covariance matrix is ​​the , by increasing the measurement noise covariance matrix The value of can reduce the weight of subsequent abnormal measurements.

[0080] (3) Reinitialize the Kalman filter.

[0081] Ultimately, the third direction vector, after detection and correction, is used to update the final state of the Kalman filter (that is, to update the information recorded by the Kalman filter) and serves as the basis for predicting the position of the second prediction frame in the next frame. This effectively prevents a single erroneous detection result from irreversibly damaging the entire tracking trajectory, thereby ensuring the continuity and accuracy of tracking small maneuvering targets in complex cloud scenarios.

[0082] The following is a brief introduction to the above target tracking method in conjunction with the overall block diagram. Figure 3As shown in the figure, it is an overall block diagram of a target tracking method provided by the present application. First, an infrared image is obtained from an infrared image data source, and then the infrared image is subjected to Gaussian curvature filtering. Specifically, the filtering parameters (including standard deviation, filtering radius, etc.) can be determined according to the noise level and contrast through a parameter adaptive adjustment scheme. Then, the filtered image is detected by DGRAD, and DeepSORT tracking is performed based on the detection result of DGRAD, including an abnormality detection process of the direction vector and an aspect ratio parameter adjustment based on feedback control. Finally, the tracking result is output. The tracking result may include trajectory information and motion information of the target object. The motion information includes the position of the target object (represented by the trajectory box of the target object), as well as the speed and direction (represented by a third direction vector), etc.

[0083] For a detailed description of the above steps, please refer to the above embodiment and will not be repeated here.

[0084] Corresponding to the aforementioned embodiments of the target tracking method, the present application also provides embodiments of a target tracking device.

[0085] See also Figure 4 FIG. 1 is a schematic diagram of the architecture of a target tracking device provided by this application, including: An acquisition module 401 is used to acquire multiple frames of infrared images; Determination module 402 is configured to determine, for an i-th infrared image frame, a first detection frame of a target object contained in the i-th infrared image frame based on a first detection algorithm in a target tracking algorithm, and to determine a second prediction frame of the target object contained in the i-th infrared image frame based on a second prediction algorithm in the target tracking algorithm; wherein the first detection algorithm is an algorithm for detection based on image features of the infrared image; the second prediction algorithm is an algorithm for prediction based on motion features of the target object in the infrared image; the second prediction algorithm includes a dynamic adjustment parameter, the dynamic adjustment parameter corresponding to the i-th infrared image frame being a parameter determined based on the first detection frame and the second prediction frame corresponding to the i-1-th infrared image frame, and used to adjust the aspect ratio of the second prediction frame of the i-th infrared image frame; and i is a positive integer; The tracking module 403 is configured to determine a tracking result of the target object in the i-th infrared image frame based on the first detection frame and the second prediction frame.

[0086] In a possible implementation, the target tracking algorithm is a Deep Simple Online Real-time Tracking (DeepSORT) algorithm, and the second prediction algorithm is a Kalman filter.

[0087] In a possible implementation, the determining module 402 is further configured to determine the dynamic adjustment parameter corresponding to the i-th infrared image frame according to the following method, including: Determine an intersection-over-union ratio between a first detection frame and the second prediction frame corresponding to the (i-1)th infrared image frame; Determining an error function between a first detection frame and the second prediction frame corresponding to the (i-1)th infrared image frame based on the intersection-over-union ratio; Based on the error function and the controller, a dynamic adjustment parameter corresponding to the i-th frame infrared image is determined; wherein the controller is used to control the changing speed of the second prediction box shape.

[0088] In one possible implementation, the Kalman filter is used to predict the position of the target object in the next frame of infrared image based on the motion characteristics of the target object in the previous frame, where the motion characteristics are represented by a direction vector; The apparatus further includes a correction module 404: Obtaining a first direction vector of the target object in the (i-1)th infrared image frame, and predicting a second direction vector of the target object in the (i)th infrared image frame based on the Kalman filter and the first direction vector; Adjusting the second direction vector based on a first detection frame of the target object in the i-th infrared image frame to obtain a third direction vector; Based on the direction vector of the target object in the N infrared image frames before the i-th frame, anomaly detection is performed on the third direction vector; N is a positive integer greater than 1; When an abnormality is detected, the third direction vector is corrected.

[0089] In one possible implementation, the correction module 404, when performing anomaly detection on the third direction vector based on the direction vector of the target object in the infrared image frames N frames before the i-th frame, is configured to: Determine a mean vector and a standard deviation vector based on the direction vector of the target object in the N infrared image frames before the i-th frame; Anomaly detection is performed on the third direction vector based on the mean vector and the standard deviation vector.

[0090] In a possible implementation, the correcting the third direction vector includes correcting the third direction vector according to any one of the following methods: The median of the direction vectors of the target object in the first K frames of infrared images is used as the corrected third direction vector; K is a positive integer greater than 1; Increasing a value of a measurement noise covariance matrix in the Kalman filter; wherein the measurement noise covariance matrix is ​​used to represent a degree of trust in the position information of the target object detected by the first detection algorithm; Reinitialize the Kalman filter.

[0091] In one possible implementation, when determining the tracking result of the target object in the i-th infrared image frame based on the first detection frame and the second prediction frame, the tracking module 403 is configured to: The first detection frame and the second prediction frame are matched to determine a trajectory frame of the target object; the trajectory frame of the target object is a tracking result of the target object in the i-th frame infrared image.

[0092] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0093] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0094] The present application also provides a computer-readable storage medium, which stores a computer program. The computer program can be used to execute the target tracking method described in the above embodiment.

[0095] This application also provides a computer device, see Figure 5 As shown, it is a structural diagram of the computer device provided in this application. At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the target tracking method described in the above embodiment. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0096] Embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0097] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform the corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can be implemented as, special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0098] Computers suitable for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to such mass storage devices to receive data from them or to transmit data to them, or both. However, a computer does not necessarily have such devices. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0099] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0100] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of specific embodiments of specific inventions. Certain features described in multiple embodiments within this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claimed as such, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of the sub-combination.

[0101] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0102] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential sequence to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0103] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A target tracking method, characterized in that: The method comprises: Acquire multiple frames of infrared images; For the i-th infrared image frame, a first detection frame of the target object contained in the i-th infrared image frame is determined based on a first detection algorithm in a target tracking algorithm, and a second prediction frame of the target object contained in the i-th infrared image frame is determined based on a second prediction algorithm in the target tracking algorithm; wherein the first detection algorithm is an algorithm for detection based on image features of the infrared image; the second prediction algorithm is an algorithm for prediction based on motion features of the target object in the infrared image; the second prediction algorithm includes a dynamic adjustment parameter, and the dynamic adjustment parameter corresponding to the i-th infrared image frame is a parameter determined based on the first detection frame and the second prediction frame corresponding to the i-1-th infrared image frame, and is used to adjust the aspect ratio of the second prediction frame of the i-th infrared image frame; i is a positive integer; Based on the first detection frame and the second prediction frame, a tracking result of the target object in the i-th infrared image frame is determined.

2. The method according to claim 1, characterized in that The target tracking algorithm is the Deep Simple Online Real-time Tracking (DeepSORT) algorithm, and the second prediction algorithm is the Kalman filter.

3. The method according to claim 1, characterized in that The method further includes determining a dynamic adjustment parameter corresponding to the i-th infrared image frame according to the following method, including: Determine an intersection-over-union ratio between a first detection frame and the second prediction frame corresponding to the (i-1)th infrared image frame; Determining an error function between a first detection frame and the second prediction frame corresponding to the (i-1)th infrared image frame based on the intersection-over-union ratio; Based on the error function and the controller, a dynamic adjustment parameter corresponding to the i-th frame infrared image is determined; wherein the controller is used to control the changing speed of the second prediction box shape.

4. The method according to claim 2, characterized in that The Kalman filter is used to predict the position of the target object in the next frame according to the motion characteristics of the target object in the previous frame of infrared image, wherein the motion characteristics are represented by a direction vector; The method further comprises: Obtaining a first direction vector of the target object in the (i-1)th infrared image frame, and predicting a second direction vector of the target object in the (i)th infrared image frame based on the Kalman filter and the first direction vector; Adjusting the second direction vector based on a first detection frame of the target object in the i-th infrared image frame to obtain a third direction vector; Based on the direction vector of the target object in the N infrared image frames before the i-th frame, anomaly detection is performed on the third direction vector; N is a positive integer greater than 1; When an abnormality is detected, the third direction vector is corrected.

5. The method according to claim 4, characterized in that The performing abnormality detection on the third direction vector based on the direction vector of the target object in the N infrared image frames before the i-th frame includes: Determine a mean vector and a standard deviation vector based on the direction vector of the target object in the N infrared image frames before the i-th frame; Anomaly detection is performed on the third direction vector based on the mean vector and the standard deviation vector.

6. The method according to claim 4, characterized in that The correcting the third direction vector includes correcting the third direction vector according to any one of the following methods: The median of the direction vectors of the target object in the first K frames of infrared images is used as the corrected third direction vector; K is a positive integer greater than 1; Increasing a value of a measurement noise covariance matrix in the Kalman filter; wherein the measurement noise covariance matrix is ​​used to represent a degree of trust in the position information of the target object detected by the first detection algorithm; Reinitialize the Kalman filter.

7. The method according to claim 1, characterized in that The determining, based on the first detection frame and the second prediction frame, a tracking result of the target object in the i-th infrared image frame includes: The first detection frame and the second prediction frame are matched to determine a trajectory frame of the target object; the trajectory frame of the target object is a tracking result of the target object in the i-th frame infrared image.

8. A target tracking device, characterized in that: The device comprises: An acquisition module, used for acquiring multiple frames of infrared images; A determination module is configured to determine, for an i-th infrared image frame, a first detection frame of a target object contained in the i-th infrared image frame based on a first detection algorithm in a target tracking algorithm, and to determine a second prediction frame of the target object contained in the i-th infrared image frame based on a second prediction algorithm in the target tracking algorithm; wherein the first detection algorithm is an algorithm for detection based on image features of the infrared image; the second prediction algorithm is an algorithm for prediction based on motion features of the target object in the infrared image; the second prediction algorithm includes a dynamic adjustment parameter, and the dynamic adjustment parameter corresponding to the i-th infrared image frame is a parameter determined based on the first detection frame and the second prediction frame corresponding to the i-1-th infrared image frame, and is used to adjust the aspect ratio of the second prediction frame of the i-th infrared image frame; and i is a positive integer; A tracking module is used to determine a tracking result of the target object in the i-th infrared image frame based on the first detection frame and the second prediction frame.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the target tracking method according to any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the target tracking method according to any one of claims 1 to 7 are implemented.

Images