Target tracking method and electronic device

Abstract

The application relates to the technical field of target tracking, and discloses a target tracking method and an electronic device, the method comprising the following steps: acquiring a video stream containing a target, a state vector, a covariance matrix and a tracking quality evaluation value of the target in an n-th frame image; determining a motion prediction factor of the target in an (n+1)-th frame image according to the speed and the covariance matrix; determining a quality prediction factor of the target in the (n+1)-th frame image according to the tracking quality evaluation value and a position bounding box; determining an interframe interval compensation factor of the target in the (n+1)-th frame image according to an interframe interval between the n-th frame image and the (n+1)-th frame image; determining a tracking bounding box coefficient of the target in the (n+1)-th frame image according to the motion prediction factor, the quality prediction factor and the interframe interval compensation factor; obtaining an (n+1)-th frame tracking image of the target by clipping according to the position bounding box and the tracking bounding box coefficient; and tracking the target according to the (n+1)-th frame tracking image. Through the above method, the reliability of target tracking is improved, and the stability of a target tracking result is improved.

Description

Technical Field

[0001] This application relates to the field of target tracking technology, specifically to a target tracking method and electronic device. Background Technology

[0002] Single Object Tracking (SOT) is a computer vision task that, given an initial bounding box of an object in the first frame of a video (such as manual annotation or detector output), continuously locates the position of the same object in all subsequent frames.

[0003] Current single-target tracking algorithms typically use a fixed search region factor, cropping the search image around the target's position in the current frame and the previous frame for tracking. When computational resources are limited, the video stream frame rate is low. When the target's speed is high, the pixel displacement of the target between adjacent frames is large. If the fixed search region factor is too small, the target may be located outside the search image. If the fixed search region factor is too large, the search image introduces a large amount of background information, leading to drone target tracking failures or errors, and reducing the reliability of drone target tracking. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a target tracking method and electronic device, which not only improves the reliability of target tracking, but also improves the stability of target tracking results.

[0005] According to one aspect of the embodiments of this application, a target tracking method is provided. The method includes: acquiring a video stream containing a target and a state vector, covariance matrix, and tracking quality evaluation value of the target in the nth frame of the video stream, wherein the state vector includes the target's position bounding box and velocity, and n≥1; determining a motion prediction factor of the target in the (n+1)th frame of the video stream based on the velocity and covariance matrix; determining a quality prediction factor of the target in the (n+1)th frame based on the tracking quality evaluation value and the position bounding box; determining a frame interval compensation factor of the target in the (n+1)th frame based on the frame interval between the nth frame and the (n+1)th frame; determining a tracking bounding box coefficient of the target in the (n+1)th frame based on the motion prediction factor, the quality prediction factor, and the frame interval compensation factor; cropping the (n+1)th frame based on the position bounding box and the tracking bounding box coefficient to obtain the (n+1)th frame tracking image of the target; and tracking the target based on the (n+1)th frame tracking image.

[0006] In one alternative approach, determining the motion prediction factor of the target in the (n+1)th frame image based on the velocity and the covariance matrix further includes: predicting the predicted velocity of the target in the (n+1)th frame image based on the velocity; determining the motion prediction value of the target in the (n+1)th frame image based on the predicted velocity; predicting the prediction covariance matrix of the target in the (n+1)th frame image based on the covariance matrix; determining the prediction uncertainty value of the target in the (n+1)th frame image based on the prediction covariance matrix; and multiplying the motion prediction value by the prediction uncertainty value to obtain the motion prediction factor of the target in the (n+1)th frame image.

[0007] In one alternative approach, determining the motion prediction value of the target in the (n+1)th frame image based on the predicted velocity further includes: obtaining the average velocity of the target in the previous n frames; determining the motion prediction value of the target in the (n+1)th frame image based on the predicted velocity and the average velocity; and determining the prediction uncertainty value of the target in the (n+1)th frame image based on the prediction covariance matrix, further including: determining the variance corresponding to the center coordinates of the target in the (n+1)th frame image in the prediction covariance matrix; and determining the prediction uncertainty value of the target in the (n+1)th frame image based on the variance corresponding to the center coordinates.

[0008] In one alternative approach, the tracking quality assessment value is the peak value of the heatmap in the nth frame image; determining the quality prediction factor of the target in the (n+1)th frame image based on the tracking quality assessment value and the location bounding box further includes: determining the tracking confidence of the target in the (n+1)th frame image based on the heatmap peak value; determining the scale change rate of the target in the (n+1)th frame image based on the location bounding box and the location bounding box of the target in the (n-1)th frame image of the video stream; and determining the quality prediction factor of the target in the (n+1)th frame image based on the tracking confidence and the scale change rate.

[0009] In one alternative approach, the position bounding box includes a bounding box width and a bounding box height; determining the scale change rate of the target in the (n+1)th frame image based on the position bounding box and the target's position bounding box in the (n-1)th frame image further includes: determining a first bounding box area of ​​the target in the nth frame image based on the bounding box width and the bounding box height of the target; determining a second bounding box area of ​​the target in the (n-1)th frame image based on the bounding box width and the bounding box height of the target; and determining the scale change rate of the target in the (n+1)th frame image based on the first bounding box area and the second bounding box area.

[0010] In one alternative approach, determining the frame interval compensation factor of the target in the (n+1)th frame image based on the frame interval between the nth frame image and the (n+1)th frame image further includes: dividing the frame interval between the nth frame image and the (n+1)th frame image by the frame interval of the video stream to obtain the frame interval compensation factor of the target in the (n+1)th frame image.

[0011] In one alternative approach, the tracking bounding box coefficients of the target in the (n+1)th frame image are determined based on the motion prediction factor, the quality prediction factor, and the frame interval compensation factor, further comprising: calculating the product of the motion prediction factor, the quality prediction factor, and the frame interval compensation factor to obtain the tracking bounding box coefficients of the target in the (n+1)th frame image.

[0012] In one optional approach, determining the tracking bounding box coefficients of the target in the (n+1)th frame image based on a motion prediction factor, a quality prediction factor, and a frame interval compensation factor further includes: determining initial tracking bounding box coefficients of the target in the (n+1)th frame image based on the motion prediction factor, the quality prediction factor, and the frame interval compensation factor; if the initial tracking bounding box coefficients are greater than or equal to a first preset coefficient and less than or equal to a second preset coefficient, then determining the initial tracking bounding box coefficients as the tracking bounding box coefficients of the target in the (n+1)th frame image, wherein the second preset coefficient is greater than the first preset coefficient; if the initial tracking bounding box coefficients are less than the first preset coefficient, then determining the first preset coefficient as the tracking bounding box coefficients of the target in the (n+1)th frame image; if the initial tracking bounding box coefficients are greater than the second preset coefficient, then determining the second preset coefficient as the tracking bounding box coefficients of the target in the (n+1)th frame image.

[0013] In one alternative approach, the target's position bounding box includes center coordinates, bounding box width, and bounding box height; the (n+1)th frame image is cropped based on the position bounding box and tracking bounding box coefficients to obtain the (n+1)th frame tracking image of the target, further comprising: calculating the target's scale feature value in the nth frame image based on the bounding box width and bounding box height; multiplying the scale feature value by the tracking bounding box coefficients to obtain the target's predicted bounding box width and predicted bounding box height in the (n+1)th frame image; and cropping the (n+1)th frame image based on the center coordinates, predicted bounding box width, and predicted bounding box height to obtain the target's (n+1)th frame tracking image.

[0014] According to another aspect of the embodiments of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the target tracking method of any of the above claims.

[0015] The target tracking method provided in this application is designed for low frame rate video streams. Considering the uncertainty of target motion and the impact of video frame rate on target tracking, it introduces a motion prediction factor for the target in the (n+1)th frame image, determined by the target's velocity and covariance matrix in the nth frame image; a quality prediction factor for the target in the (n+1)th frame image, determined by the target's position bounding box and tracking quality evaluation value in the nth frame image; and a frame interval compensation factor for the target in the (n+1)th frame image, determined by the frame interval between the nth and (n+1)th frames. The motion prediction factor characterizes the uncertainty of the target's motion in the (n+1)th frame image, and the quality prediction factor... The tracking bounding box factor can characterize the quality of target tracking in the (n+1)th frame, while the frame interval compensation factor can characterize the frame rate of the video stream. When the uncertainty of target motion increases, the target tracking quality in the nth frame is low, and the frame rate is low, the tracking bounding box factor can be increased to enlarge the tracking image, increasing the probability that the target is located in the tracking image and avoiding target tracking failure. When the uncertainty of target motion decreases, the target tracking quality in the nth frame is high, and the frame rate is high, the tracking bounding box factor can be decreased to shrink the tracking image. This ensures that the target is located in the tracking image while avoiding the introduction of more background information from the (n+1)th frame, thus avoiding target tracking errors. In summary, by adaptively adjusting the tracking bounding box factor through the motion prediction factor, quality prediction factor, and frame interval compensation factor, the size of the tracking image can be adaptively adjusted, improving the stability of target tracking in low frame rate scenarios.

[0016] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of this application are described below. Attached Figure Description

[0017] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating the target tracking method provided in an embodiment of this application is shown; Figure 2 A schematic diagram of the target tracking device provided in an embodiment of this application is shown; Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0018] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein.

[0019] In the field of drones, during the real-time acquisition of video streams by optoelectronic devices, after the drone target is framed in the first frame of the video stream, the single-target tracking algorithm can continuously and accurately predict the position and bounding box of the drone target in all subsequent frames.

[0020] Current single-target tracking algorithms, such as SiamFC, SiamRPN++, and OSTrack, typically multiply the border of the previous frame image by a fixed search region factor after determining the position and bounding box of the UAV target in the previous frame image to obtain an enlarged border. Then, the position of the UAV target in the previous frame image is used as its position in the current frame image. The search image is cropped from the current frame image according to the enlarged border, and finally, the search image is input into the single-target tracking network for target tracking to obtain the position and bounding box of the target in the current frame image.

[0021] With sufficient computing power, video stream frame rates typically reach over 30fps, and the time interval between two adjacent frames is less than 33 milliseconds. Within this extremely short time, the pixel displacement of the drone target is very small. Therefore, after determining the bounding box of the drone target in the previous frame, multiplying the bounding box by a fixed search region factor allows for the determination of a larger new bounding box in the current frame. The drone target is usually located within this new bounding box in the current frame. Subsequent single-target tracking networks can then accurately track the target by cropping the search image based on this new bounding box.

[0022] However, embedded low-power platforms have limited computing power, resulting in low frame rates for video streams, typically 10-15 fps, with time intervals between adjacent frames ranging from 66-100 milliseconds. The high speed of drone targets causes significant pixel displacement between adjacent frames, resulting in the drone target appearing to jump rather than move smoothly and continuously within the video stream.

[0023] In this scenario, when the fixed search region factor is set too small, the newly determined bounding box is small, and the drone target is likely located outside the search image. The single-target tracking network cannot find the drone target in this search image, leading to tracking failure. When the fixed search region factor is set too large, the newly determined bounding box is too large, increasing the probability that the drone target is within the search image. However, the search image also introduces a large amount of background information, causing a surge in interference when the single-target tracking network identifies the drone target, resulting in incorrect target locking and tracking errors. Therefore, for low frame rate video streams containing drone targets, using a fixed search region factor for drone target tracking will reduce the reliability of drone target tracking.

[0024] To address the aforementioned issues, existing single-target tracking algorithms can adjust the search region factor by predicting the speed of the drone target. When the predicted speed of the drone target is high, the pixel displacement of the drone target increases, and increasing the search region factor ensures that the drone target remains within the new bounding box of the current frame image. When the predicted speed of the drone target is low, the pixel displacement of the drone target decreases, and the search region factor shrinks. While ensuring that the drone target remains within the new bounding box of the current frame, this reduces background information in the search image and improves the reliability of drone target tracking.

[0025] When the drone target's motion is linear, adjusting the search area factor by predicting the drone target's speed can improve the reliability of drone target tracking. However, when the drone target's motion is non-linear, such as sudden stops or sharp turns, the probability of the drone target existing within the new bounding box determined in the current frame image using the above method decreases. This can lead to drone target tracking failures or errors, reducing the stability of the drone target tracking results.

[0026] To address this issue, this application provides a target tracking method for low frame rate video streams. Considering the uncertainty of target motion and the impact of video frame rate on target tracking, this method introduces a motion prediction factor, a quality prediction factor, and a frame interval compensation factor. The motion prediction factor characterizes the uncertainty of target motion in the current frame, the quality prediction factor characterizes the quality of target tracking in the previous frame, and the frame interval compensation factor characterizes the frame rate of the video stream. When the uncertainty of target motion increases, the target tracking quality in the previous frame is low, and the frame rate is low, the search region factor can be increased, thereby increasing the search image and improving the probability that the target is located in the search image, thus avoiding target tracking failure. When the uncertainty of target motion decreases, the target tracking quality in the previous frame is high, and the frame rate is high, the search region factor can be decreased, thereby decreasing the search image. This ensures that the target is located in the search image while avoiding the search image introducing more background information from the current frame, thus avoiding target tracking errors. In summary, by adaptively adjusting the search region factor through the motion prediction factor, quality prediction factor, and frame interval compensation factor, the range of the search image can be adaptively adjusted, improving the stability of target tracking in low frame rate scenarios.

[0027] Figure 1 A flowchart of a target tracking method provided in an embodiment of this application is shown. This method is executed by electronic devices such as computers, servers, drone detection equipment, and drone jamming equipment. Figure 1 As shown, the method includes the following steps: Step S100: Obtain the video stream containing the target and the state vector, covariance matrix, and tracking quality evaluation value of the target in the nth frame of the video stream.

[0028] The state vector includes the target's position bounding box and velocity, where n ≥ 1. The frame rate of the video stream in this embodiment may be lower than 30 fps, for example, 10-15 fps, and the target may be a drone, vehicle, pedestrian, animal, or other object to be tracked. This application uses a drone as an example for illustration and does not constitute a limitation.

[0029] The target's position bounding box can include the target's center coordinates (cx, cy), the bounding box width w, and the bounding box height h. The target's velocity can include the target's horizontal velocity vx and vertical velocity vy.

[0030] The covariance matrix is ​​used to characterize the uncertainty of the state vector; the larger the value, the greater the inaccuracy of the state vector of the target.

[0031] Tracking quality evaluation is an indicator that measures the reliability of the target tracking result in the previous frame image. For example, tracking quality evaluation can be the peak value of the heatmap of the nth frame image output by the single target tracking network, the peak-side-lobe ratio (PSR) of the heatmap, the feature similarity between the tracked images of the (n-1)th frame image and the nth frame image, and the intersection-overunion ratio (IoU) between the bounding box of the nth frame image and the predicted bounding box of the nth frame image obtained by Kalman prediction.

[0032] The target tracking method in this application can start from the second frame, meaning the target can be located in the second frame based on information from the first frame of the video stream. The bounding box of the target's position in the first frame of the video stream can be detected by a detection device. The target's velocity in the first frame is 0. The covariance matrix of the target in the first frame can be set based on the uncertainty of the state vector. The tracking quality evaluation value of the target in the first frame can be set to 1.

[0033] Step S200: Determine the motion prediction factor of the target in the (n+1)th frame of the video stream based on the velocity and covariance matrix.

[0034] The motion predictor can characterize the position of a target in the (n+1)th frame of the image, and its magnitude is positively correlated with the uncertainty of the target's motion. The greater the velocity or the larger the covariance matrix, the greater the displacement of the target in the (n+1)th frame of the image, the higher the uncertainty of the state vector in the nth frame of the image, and the larger the motion predictor.

[0035] Specifically, step S200 may include the following steps: Step S210: Predict the target's velocity in the (n+1)th frame image based on the velocity.

[0036] When the target's motion is linear, a Kalman filter can be used to predict the target's velocity in the (n+1)th frame. When the target's motion is nonlinear, such as when the target switches between constant speed, constant acceleration, and emergency maneuvering, an Extended Kalman Filter (EKF), an Unscented Kalman Filter (UKF), or an Interacting Multiple Model (IMM) can be used to predict the target's velocity in the (n+1)th frame.

[0037] Taking the Kalman filter as an example, the prediction speed of the target in the (n+1)th frame of the image. It can be determined using formula (1): (1) Step S220: Determine the motion prediction value of the target in the (n+1)th frame image based on the predicted velocity.

[0038] Specifically, the average velocity of the target in the first n frames of the image can be obtained first. Then based on the predicted speed and average speed Determine the motion prediction value of the target in the (n+1)th frame of the image.

[0039] motion prediction value It can be determined using formula (2): (2) in, This represents the baseline tracking bounding box coefficients when the target is a drone. It can be set to 4; The motion sensitivity coefficient of the target can range from 0.5 to 1.0.

[0040] The faster the target moves in the nth frame, the faster its predicted speed will be in the (n+1)th frame. As a result, the predicted motion value of the target in the (n+1)th frame will be larger. Within the same frame interval, the pixel displacement of the target in the image will be larger. Therefore, a larger tracking range is needed to cover the target in the (n+1)th frame.

[0041] Step S230: Predict the predicted covariance matrix of the target in the (n+1)th frame image based on the covariance matrix.

[0042] Specifically, the process noise covariance matrix of the Kalman filter can be set, and the state transition matrix of the Kalman filter can be determined according to the definition of the state vector. The process noise covariance matrix represents the difference between the target motion model and the actual motion, while the state transition matrix describes how the target's state vector evolves from the nth frame to the (n+1)th frame. Then, the Kalman filter can determine the prediction covariance matrix of the target in the (n+1)th frame based on the process noise covariance matrix, the state transition matrix, and the target's covariance matrix in the nth frame.

[0043] Specifically, the prediction covariance matrix of the target in the (n+1)th frame image can be determined using formula (3). : (3) in, Represents the state transition matrix. This represents the covariance matrix of the target in the nth frame of the image. This represents the process noise covariance matrix.

[0044] Step S240: Determine the prediction uncertainty of the target in the (n+1)th frame image based on the prediction covariance matrix.

[0045] Specifically, the variance corresponding to the center coordinates of the target in the (n+1)th frame of the image can be determined in the prediction covariance matrix, and then the prediction uncertainty of the target in the (n+1)th frame of the image can be determined based on the variance corresponding to the center coordinates.

[0046] In the prediction covariance matrix, the variance of the center coordinate of the target in the (n+1)th frame image is the variance of the center coordinate cx. variance of cy , represent the uncertainty of the center coordinates cx and cy of the target in the (n+1)th frame of the image, respectively.

[0047] Prediction uncertainty can represent the uncertainty about the target's position in the (n+1)th frame of the image. It can be determined using formula (4): (4) in, Indicates the reference uncertainty standard deviation. This represents the uncertainty sensitivity coefficient, which can range from 0.3 to 0.8.

[0048] Step S250: Multiply the motion prediction value by the prediction uncertainty value to obtain the motion prediction factor of the target in the (n+1)th frame image.

[0049] When a target undergoes nonlinear maneuvers such as sudden stops or sharp turns, the predicted velocity in the (n+1)th frame is significantly biased based on the target's velocity prediction in the nth frame. Through steps S210-S250, the uncertainty component related to position is extracted from the prediction covariance matrix to calculate the prediction uncertainty value. This allows the prediction uncertainty value to increase with the target's position uncertainty, thereby increasing the motion prediction factor. Finally, when the target's motion prediction is unreliable, the tracking bounding box coefficient can be increased to expand the tracking image in the (n+1)th frame, increasing the probability that the target is within the tracking image and ensuring the accuracy of target tracking.

[0050] Step S300: Determine the quality prediction factor of the target in the (n+1)th frame image based on the tracking quality evaluation value and the position bounding box.

[0051] Specifically, step S300 may include the following steps: Step S310: Determine the tracking confidence of the target in the (n+1)th frame image based on the heatmap peak.

[0052] Tracking confidence is used to evaluate and correct the tracking quality assessment value of the nth frame image.

[0053] When the target tracking quality of the nth frame image is not high, the quality prediction factor should be increased to increase the tracking bounding box coefficient, thereby increasing the range of the tracked target in the (n+1)th frame image and preventing target loss.

[0054] Taking the heatmap peak as an example, when the heatmap peak of the nth frame image output by the single-target tracking network is low, it indicates that the target tracking result may be biased or the target may be partially occluded, meaning the tracking quality is not high. In this case, it is necessary to increase the tracking confidence to increase the quality prediction factor, and then increase the tracking bounding box coefficient. This allows the target to be tracked within a larger tracking image in the (n+1)th frame image, thus relocating the target.

[0055] Specifically, tracking confidence levels It can be determined using formula (5): (5) in, This represents the confidence level sensitivity coefficient, which can range from 1 to 2.

[0056] Step S320: Determine the scale change rate of the target in the (n+1)th frame image based on the position bounding box and the position bounding box of the target in the (n-1)th frame image of the video stream.

[0057] The rate of scale change reflects the speed at which a target approaches or moves away. A larger rate of scale change occurs when a target approaches or moves away rapidly, while a smaller rate of scale change occurs when a target approaches or moves away slowly.

[0058] Specifically, the scale change rate of the target in the (n+1)th frame can be determined by the ratio between the border area of ​​the target in the (n-1)th frame and the border area of ​​the target in the nth frame. Alternatively, the scale change rate of the target in the (n+1)th frame can be determined by the ratio between the border width and border height of the target in the (n-1)th frame and the border width and border height of the target in the nth frame.

[0059] Taking the ratio between border areas as an example, step S320 may include the following steps: Step S321: Determine the area of ​​the first border of the target in the nth frame image based on the border width and border height of the target in the nth frame image.

[0060] Step S322: Determine the area of ​​the second border of the target in the (n-1)th frame image based on the border width and border height of the target in the (n-1)th frame image.

[0061] Step S323: Determine the scale change rate of the target in the (n+1)th frame image based on the area of ​​the first border and the area of ​​the second border.

[0062] Multiplying the border width w of the target in the nth frame by the border height h of the target in the nth frame yields the area of ​​the target's first border in the nth frame. Multiplying the border width w of the target in the (n-1)th frame by the border height h of the target in the (n-1)th frame yields the area of ​​the target's second border in the (n-1)th frame. .

[0063] Then, the scale change rate of the target in the (n+1)th frame can be calculated using formula (6). : (6) in, This represents the scale sensitivity coefficient, which can range from 0.5 to 1.5.

[0064] It should be noted that when n=1, the scale change rate of the target in the (n+1)th frame is 1.

[0065] When the area of ​​the first border Area of ​​the second border When the ratio is greater than 1 and increases, it indicates that the target is rapidly approaching, and the area of ​​the first border... Area of ​​the second border When the ratio is less than 1 and decreases, it indicates that the target is moving away rapidly. Both rapid movement away and rapid approach indicate a high target speed; in this case, increasing the rate of scale change... This increases the quality prediction factor, which in turn increases the tracking bounding box coefficient, thereby increasing the size of the tracking image used to track the target. A larger tracking image increases the probability that the target is located within the tracking image. Therefore, tracking the target within a larger tracking image in the (n+1)th frame can increase the probability of tracking the target and ultimately improve the reliability of target tracking.

[0066] Alternatively, in step S323, the area of ​​the first border can also be used as a reference. Area of ​​the second border The second derivative of the ratio determines the rate of scale change of the target in the (n+1)th frame of the image.

[0067] Step S330: Determine the quality prediction factor of the target in the (n+1)th frame image based on the tracking confidence and scale change rate.

[0068] The quality prediction factor of the target in the (n+1)th frame can be determined according to formula (7): Quality predictor = max(1, (7) The above formula ensures that the quality prediction factor is greater than or equal to 1, guaranteeing that when the tracking quality is high and the scale change rate is small, or when the tracking quality is low and the scale change rate is large, the bounding box in the (n+1)th frame image can be larger than the bounding box of the target output by the single target tracking network in the nth frame image, thus ensuring the reliability of target tracking.

[0069] After steps S310-S330, when the scale change rate is large, even if target tracking is successful in the nth frame, the target tracking quality in the (n+1)th frame will still be reduced. Conversely, when the target tracking quality in the nth frame is small, even if the scale change rate is small, the target tracking quality in the (n+1)th frame will still be reduced. Therefore, determining the target quality prediction factor in the (n+1)th frame based on the product of tracking confidence and scale change rate allows for a comprehensive consideration of the impact of both tracking confidence and scale change rate on tracking quality. In cases of low tracking confidence or high scale change rate, the quality prediction factor can be increased to increase the target's tracking bounding box coefficient in the (n+1)th frame, enabling target tracking within a larger tracking area and ensuring reliable target tracking.

[0070] Step S400: Determine the frame interval compensation factor of the target in the (n+1)th frame image based on the frame interval between the nth frame image and the (n+1)th frame image.

[0071] The frame interval compensation factor can characterize the frame rate fluctuation of a video stream. When the video stream experiences frame drops or a decrease in frame rate, the frame interval compensation factor increases.

[0072] Specifically, the frame interval between the nth frame and the (n+1)th frame can be divided by the frame interval of the video stream to obtain the frame interval compensation factor for the target in the (n+1)th frame. Here, the frame interval of the video stream is the nominal frame interval of the video stream, which is the standard frame interval of the device acquiring the video stream.

[0073] Under normal circumstances, the frame rate of a video stream is constant. However, in scenarios with limited computing power, variations in the load of the device capturing the video stream can lead to frame loss or a frame interval between adjacent frames exceeding the nominal frame interval, resulting in a reduced frame rate. In such cases, the pixel displacement of the target in adjacent frames increases. Therefore, it is necessary to increase the target's tracking range to prevent the actual movement distance from exceeding the tracking range. When frame loss or a reduced frame rate occurs in the video stream, the frame interval between the nth and (n+1)th frames increases. When the frame interval between the nth and (n+1)th frames exceeds the video stream's frame interval, the frame interval compensation factor is greater than 1, increasing the target's tracking bounding box coefficient. This, in turn, increases the target's tracking range, preventing the target from exceeding the tracking range and improving the reliability of target tracking.

[0074] When the video frame rate increases, and the frame interval between the nth frame and the (n+1)th frame decreases to less than or equal to the frame interval of the video stream, the frame interval compensation factor is less than or equal to 1. The target tracking bounding box coefficient decreases or remains unchanged, thereby reducing or keeping the target tracking range unchanged. This reduces unnecessary background information, avoids target tracking errors, and improves the reliability of target tracking.

[0075] Step S500: Determine the tracking bounding box coefficients of the target in the (n+1)th frame image based on the motion prediction factor, quality prediction factor, and frame interval compensation factor.

[0076] The tracking bounding box coefficient is the search region factor, which is used to calculate the size of the tracking image.

[0077] Specifically, the tracking bounding box coefficients of the target in the (n+1)th frame can be obtained by multiplying the motion prediction factor, the quality prediction factor, and the frame interval compensation factor. For example, the tracking bounding box coefficients of the target in the (n+1)th frame = motion prediction factor × quality prediction factor × frame interval compensation factor.

[0078] In this embodiment, the motion prediction value is multiplied by the prediction uncertainty to obtain the motion prediction factor, and the tracking confidence is multiplied by the scale change rate to obtain the quality prediction factor. Then, the motion prediction factor is multiplied by the quality prediction factor, which can avoid mutual interference between the motion prediction value, the prediction uncertainty, the tracking confidence, and the scale change rate, and can also achieve complementarity between motion prediction and quality prediction, thereby improving the reliability of target tracking.

[0079] In some embodiments, weights can be assigned to the motion prediction factor, quality prediction factor, and frame interval compensation factor, respectively. Then, the motion prediction factor, quality prediction factor, and frame interval compensation factor are multiplied by their respective weights and finally summed to obtain the tracking bounding box coefficients of the target in the (n+1)th frame image. Alternatively, the maximum value among the motion prediction factor, quality prediction factor, and frame interval compensation factor can be determined as the tracking bounding box coefficients of the target in the (n+1)th frame image.

[0080] In other embodiments, after determining the motion prediction value, prediction uncertainty value, tracking confidence, scale change rate, and frame interval compensation factor of the target in the (n+1)th frame image, the motion prediction value, prediction uncertainty value, tracking confidence, scale change rate, and frame interval compensation factor can be input into a small fully connected network, and the optimal factor value obtained by regression of the fully connected network can be determined as the tracking bounding box coefficient of the target in the (n+1)th frame image.

[0081] When the tracking bounding box coefficient is too small, the target may be lost due to the small tracking range. When the tracking bounding box coefficient is too large, the tracking range is too large, and too much background information is introduced, which may cause the target tracking to err due to excessive interference.

[0082] Therefore, to solve the above problem, the initial tracking bounding box coefficients of the target in the (n+1)th frame image can be determined based on the motion prediction factor, quality prediction factor, and frame interval compensation factor. Then, it is determined whether the initial tracking bounding box coefficients are greater than or equal to a first preset coefficient and less than or equal to a second preset coefficient, wherein the second preset coefficient is greater than the first preset coefficient. If the initial tracking bounding box coefficients are greater than or equal to the first preset coefficient and less than or equal to the second preset coefficient, then the initial tracking bounding box coefficients are determined as the tracking bounding box coefficients of the target in the (n+1)th frame image; if the initial tracking bounding box coefficients are less than the first preset coefficient, then the first preset coefficient is determined as the tracking bounding box coefficients of the target in the (n+1)th frame image; if the initial tracking bounding box coefficients are greater than the second preset coefficient, then the second preset coefficient is determined as the tracking bounding box coefficients of the target in the (n+1)th frame image.

[0083] Assuming the first preset coefficient is set to 3 and the second preset coefficient is set to 8, when the initial tracking bounding box coefficient of the target in the (n+1)th frame is 5, the tracking bounding box coefficient of the target in the (n+1)th frame is 5; when the initial tracking bounding box coefficient of the target in the (n+1)th frame is 2.5, the tracking bounding box coefficient of the target in the (n+1)th frame is 3; and when the initial tracking bounding box coefficient of the target in the (n+1)th frame is 9, the tracking bounding box coefficient of the target in the (n+1)th frame is 8.

[0084] In this way, the final tracking bounding box coefficients can be avoided from being too small or too large, preventing the target from being lost due to an insufficient tracking range, or preventing the target from being tracked incorrectly due to the introduction of too much background information, thus improving the reliability of target tracking.

[0085] Step S600: Crop the (n+1)th frame image based on the position bounding box and the tracking bounding box coefficients to obtain the (n+1)th frame tracking image of the target.

[0086] Specifically, the scale feature value of the target in the nth frame image can be calculated based on the border width and border height of the target in the nth frame image. Then, the scale feature value is multiplied by the tracking border coefficient to obtain the predicted border width and predicted border height of the target in the (n+1)th frame image. Finally, the (n+1)th frame image is cropped based on the center coordinates, predicted border width, and predicted border height to obtain the (n+1)th frame tracking image of the target.

[0087] In this embodiment of the application, the value obtained by multiplying the scale feature value by the tracking bounding box coefficient is simultaneously determined as the predicted bounding box width and predicted bounding box height of the target in the (n+1)th frame image, that is, the predicted bounding box width and predicted bounding box height of the target in the (n+1)th frame image are equal.

[0088] For example, the predicted bounding box width and predicted bounding box height of the target in the (n+1)th frame image can be calculated according to formula (8): (8) in, This represents the predicted bounding box width and predicted bounding box height of the target in the (n+1)th frame of the image. This represents the floor function. This represents the border width of the target in the nth frame of the image. This represents the height of the target's border in the nth frame of the image. This represents the tracking bounding box coefficient of the target in the (n+1)th frame of the image.

[0089] After determining the predicted bounding box width and predicted bounding box height of the target in the (n+1)th frame image, a square region can be cropped from the (n+1)th frame image as the tracking image, centered on the center coordinates of the target in the (n)th frame image or the center coordinates of the target predicted by the Kalman filter in the (n+1)th frame image, according to the predicted bounding box width and predicted bounding box height.

[0090] Step S700: Track the target based on the tracking image of frame n+1.

[0091] After cropping the (n+1)th frame of the tracking image, the (n+1)th frame can be scaled to a fixed size for input into the single-target tracking network. The single-target tracking network can be an algorithm based on a Siamese network architecture, such as SiamFC or SiamRPN++, or an algorithm based on Transformer, such as OSTrack or MixFormer.

[0092] When the (n+1)th frame of the tracking image is input into the single-target tracking network, the network can output the bounding box position and velocity of the target in the (n+1)th frame through operations such as feature extraction, matching, or regression. The single-target tracking network can also output the tracking quality evaluation value of the target in the (n+1)th frame (e.g., heatmap peaks).

[0093] After the single-target tracking network tracks the target based on the (n+1)th frame image, the target's position bounding box, velocity, and covariance matrix (i.e., the predicted covariance matrix of the (n+1)th frame image) are used as observations and input into the Kalman filter for updating. This yields the predicted velocity and predicted covariance matrix of the target in the (n+2)th frame image. Then, based on the predicted velocity and predicted covariance matrix of the target in the (n+2)th frame image, the motion prediction factor of the target in the (n+2)th frame image can be determined. Subsequently, the quality prediction factor of the target in the (n+2)th frame image is determined based on the tracking quality evaluation value of the target in the (n+1)th frame image. Based on the... The frame interval between the (n+1)th and (n+2)th frames determines the frame interval compensation factor for the target in the (n+2)th frame. Further, based on the target's quality prediction factor, quality prediction factor, and frame interval compensation factor in the (n+2)th frame, the tracking bounding box coefficients for the target in the (n+2)th frame can be determined. Finally, the (n+2)th frame is cropped based on the target's position bounding box in the (n+1)th frame and the tracking bounding box coefficients in the (n+2)th frame, yielding the target's (n+2)th frame tracking image. This (n+2)th frame tracking image is then input into a single-target tracking network, which outputs the tracking results, thus achieving target tracking. After this, the target's position bounding box, velocity, and covariance matrix (i.e., the prediction covariance matrix of the (n+2)th frame) output by the single-target tracking network are used as observations… until the target tracking is completed based on the video stream acquired by the optoelectronic device.

[0094] The target tracking method provided in this application is designed for low frame rate video streams. Considering the uncertainty of target motion and the impact of video frame rate on target tracking, it introduces a motion prediction factor for the target in the (n+1)th frame image, determined by the target's velocity and covariance matrix in the nth frame image; a quality prediction factor for the target in the (n+1)th frame image, determined by the target's position bounding box and tracking quality evaluation value in the nth frame image; and a frame interval compensation factor for the target in the (n+1)th frame image, determined by the frame interval between the nth and (n+1)th frames. The motion prediction factor characterizes the uncertainty of the target's motion in the (n+1)th frame image, and the quality prediction factor... The tracking bounding box factor can characterize the quality of target tracking in the (n+1)th frame, while the frame interval compensation factor can characterize the frame rate of the video stream. When the uncertainty of target motion increases, the target tracking quality in the nth frame is low, and the frame rate is low, the tracking bounding box factor can be increased to enlarge the tracking image, increasing the probability that the target is located in the tracking image and avoiding target tracking failure. When the uncertainty of target motion decreases, the target tracking quality in the nth frame is high, and the frame rate is high, the tracking bounding box factor can be decreased to shrink the tracking image. This ensures that the target is located in the tracking image while avoiding the introduction of more background information from the (n+1)th frame, thus avoiding target tracking errors. In summary, by adaptively adjusting the tracking bounding box factor through the motion prediction factor, quality prediction factor, and frame interval compensation factor, the size of the tracking image can be adaptively adjusted, improving the stability of target tracking in low frame rate scenarios.

[0095] Figure 2 A schematic diagram of the target tracking device provided in an embodiment of this application is shown, as follows: Figure 2As shown, the target tracking device 100 includes: an acquisition module 110, a first determination module 120, a second determination module 130, a third determination module 140, a fourth determination module 150, a cropping module 160, and a tracking module 170. The acquisition module 110 acquires a video stream containing the target and the target's state vector, covariance matrix, and tracking quality evaluation value in the nth frame of the video stream. The state vector includes the target's position bounding box and velocity, where n ≥ 1. The first determination module 120 determines the motion prediction factor of the target in the (n+1)th frame of the video stream based on the velocity and covariance matrix. The second determination module 130 determines the quality prediction factor of the target in the (n+1)th frame based on the tracking quality evaluation value and the position bounding box. The third determination module 140 determines the frame interval compensation factor of the target in the (n+1)th frame based on the frame interval between the nth and (n+1)th frames. The fourth determination module 150 determines the tracking bounding box coefficient of the target in the (n+1)th frame based on the motion prediction factor, the quality prediction factor, and the frame interval compensation factor. The cropping module 160 is used to crop the (n+1)th frame image based on the position bounding box and the tracking bounding box coefficients to obtain the (n+1)th frame tracking image of the target. The tracking module 170 is used to track the target based on the (n+1)th frame tracking image.

[0096] The specific implementation process and beneficial effects of each module in the target tracking device 100 of the above embodiments to achieve the above functions can be referred to the foregoing. Figure 1 The embodiment of the target tracking method shown will not be described in detail here.

[0097] The target tracking device 100 of this application embodiment also includes other modules for performing the steps of the above-described target tracking method embodiments, which will not be described in detail here.

[0098] Figure 3 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0099] like Figure 3 As shown, the electronic device 200 may include a processor 202 and a memory 204.

[0100] The memory 204 is used to store the computer program 206. The memory 204 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive. The computer program 206 may include computer-executable instructions.

[0101] The processor 202 is used to execute the computer program 206 to implement the above-described target tracking method embodiment.

[0102] Processor 202 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The electronic device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0103] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described target tracking method embodiment.

[0104] This application provides a computer program that can be executed by a processor to implement the target tracking method described above.

[0105] This application provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described target tracking method embodiment.

[0106] In the several embodiments provided in this application, any function, if implemented as a software functional module / unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, part or all of the technical solutions of this application can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or other electronic device) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing computer program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of this application are not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0108] It should be noted that the above embodiments are illustrative of this application and not restrictive, and those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In claims enumerating several means, several units or modules of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

[0109] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A target tracking method, characterized in that, The method includes: Obtain a video stream containing the target and the state vector, covariance matrix, and tracking quality evaluation value of the target in the nth frame of the video stream, wherein the state vector includes the target's position bounding box and velocity, and n≥1; The motion prediction factor of the target in the (n+1)th frame of the video stream is determined based on the velocity and the covariance matrix. The quality prediction factor of the target in the (n+1)th frame image is determined based on the tracking quality evaluation value and the location bounding box. Based on the frame interval between the nth frame image and the (n+1)th frame image, determine the frame interval compensation factor of the target in the (n+1)th frame image; The tracking bounding box coefficients of the target in the (n+1)th frame image are determined based on the motion prediction factor, the quality prediction factor, and the frame interval compensation factor. The (n+1)th frame image is cropped according to the position bounding box and the tracking bounding box coefficients to obtain the (n+1)th frame tracking image of the target; The target is tracked based on the (n+1)th frame tracking image.

2. The method according to claim 1, characterized in that, Determining the motion prediction factor of the target in the (n+1)th frame image based on the velocity and the covariance matrix further includes: The predicted velocity of the target in the (n+1)th frame of the image is predicted based on the velocity. The motion prediction value of the target in the (n+1)th frame image is determined based on the predicted velocity. Predict the prediction covariance matrix of the target in the (n+1)th frame image based on the covariance matrix; The prediction uncertainty value of the target in the (n+1)th frame image is determined based on the prediction covariance matrix; Multiply the motion prediction value by the prediction uncertainty value to obtain the motion prediction factor of the target in the (n+1)th frame image.

3. The method according to claim 2, characterized in that, Determining the motion prediction value of the target in the (n+1)th frame image based on the predicted velocity further includes: Obtain the average velocity of the target in the first n frames of the image; The motion prediction value of the target in the (n+1)th frame image is determined based on the predicted velocity and the average velocity. Determining the prediction uncertainty of the target in the (n+1)th frame image based on the prediction covariance matrix further includes: Determine the variance of the target in the prediction covariance matrix corresponding to the center coordinates of the target in the (n+1)th frame of the image; The prediction uncertainty of the target in the (n+1)th frame image is determined based on the variance corresponding to the center coordinates.

4. The method according to claim 1, characterized in that, The tracking quality evaluation value is the peak value of the heatmap of the nth frame image; Determining the quality prediction factor of the target in the (n+1)th frame image based on the tracking quality evaluation value and the position bounding box further includes: The tracking confidence of the target in the (n+1)th frame image is determined based on the peak value of the heat map; The scale change rate of the target in the (n+1)th frame is determined based on the position bounding box and the position bounding box of the target in the (n-1)th frame of the video stream; The quality prediction factor of the target in the (n+1)th frame image is determined based on the tracking confidence and the scale change rate.

5. The method according to claim 4, characterized in that, The position border includes the border width and the border height; Determining the scale change rate of the target in the (n+1)th frame image based on the position bounding box and the position bounding box of the target in the (n-1)th frame image further includes: The area of ​​the target in the first border of the nth frame image is determined based on the border width and border height of the target in the nth frame image; The second border area of ​​the target in the (n-1)th frame image is determined based on the border width and border height of the target in the (n-1)th frame image; The scale change rate of the target in the (n+1)th frame image is determined based on the area of ​​the first border and the area of ​​the second border.

6. The method according to claim 1, characterized in that, The step of determining the frame interval compensation factor of the target in the (n+1)th frame image based on the frame interval between the nth frame image and the (n+1)th frame image further includes: Divide the frame interval between the nth frame and the (n+1)th frame by the frame interval of the video stream to obtain the frame interval compensation factor of the target in the (n+1)th frame.

7. The method according to claim 1, characterized in that, Determining the tracking bounding box coefficients of the target in the (n+1)th frame image based on the motion prediction factor, the quality prediction factor, and the frame interval compensation factor further includes: The tracking bounding box coefficients of the target in the (n+1)th frame image are obtained by multiplying the motion prediction factor, the quality prediction factor, and the frame interval compensation factor.

8. The method according to claim 1, characterized in that, Determining the tracking bounding box coefficients of the target in the (n+1)th frame image based on the motion prediction factor, the quality prediction factor, and the frame interval compensation factor further includes: The initial tracking bounding box coefficients of the target in the (n+1)th frame image are determined based on the motion prediction factor, the quality prediction factor, and the frame interval compensation factor. If the initial tracking bounding box coefficient is greater than or equal to the first preset coefficient and less than or equal to the second preset coefficient, then the initial tracking bounding box coefficient is determined as the tracking bounding box coefficient of the target in the (n+1)th frame image, wherein the second preset coefficient is greater than the first preset coefficient; If the initial tracking bounding box coefficient is less than the first preset coefficient, then the first preset coefficient is determined as the tracking bounding box coefficient of the target in the (n+1)th frame image; If the initial tracking bounding box coefficient is greater than the second preset coefficient, then the second preset coefficient is determined as the tracking bounding box coefficient of the target in the (n+1)th frame image.

9. The method according to claim 1, characterized in that, The target's location bounding box includes center coordinates, bounding width, and bounding height; The step of cropping the (n+1)th frame image based on the position bounding box and the tracking bounding box coefficient to obtain the (n+1)th frame tracking image of the target further includes: Calculate the scale feature value of the target in the nth frame image based on the border width and border height; Multiplying the scale feature value by the tracking bounding box coefficient yields the predicted bounding box width and predicted bounding box height of the target in the (n+1)th frame image. The (n+1)th frame image of the target is obtained by cropping the image based on the center coordinates, the predicted bounding box width, and the predicted bounding box height.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the target tracking method according to any one of claims 1 to 9.

Images