Target tracking method and device, computer readable storage medium and terminal equipment

Abstract

The invention belongs to the technical field of target tracking, and particularly relates to a target tracking method and device, a computer readable storage medium and terminal equipment. The method comprises the following steps: acquiring a target video frame; performing target detection on the target video frame to obtain a target detection result; based on a distributed robust optimization Kalman filter, determining a target prediction result corresponding to the target video frame; matching the target detection result with the target prediction result to obtain a target matching result; and determining a target tracking result based on the target matching result. According to the method, the target prediction result can be obtained through prediction by using the distributed robust optimization Kalman filter; therefore, various uncertainties can be fully considered, more accurate target prediction can be provided in the face of unknown noise, and the success rate and robustness of the tracking process can be improved.

Description

Technical Field

[0001] The present application belongs to the field of target tracking technology, and in particular, relates to a target tracking method, device, computer-readable storage medium and terminal device. Background Art

[0002] Multi-Object Tracking (MOT) is an important research topic in the field of computer vision and navigation, which involves the problem of simultaneously tracking multiple moving objects in a video sequence. Multi-object tracking is crucial for many applications such as video surveillance, self-driving cars, drone monitoring, motion analysis, and military reconnaissance.

[0003] However, the actual tracking environment is in a dynamic change and full of uncertainty. For example, the actual movement of the target often does not conform to the uniform linear motion assumed in the prior art. In addition, in actual applications, it may be affected by more noise, such as noise caused by light changes and sensor observation noise. Existing target tracking methods are difficult to cope with uncertain and dynamically changing environments, and the tracking success rate and robustness are not high. Summary of the invention

[0004] In view of this, embodiments of the present application provide a target tracking method, apparatus, computer-readable storage medium and terminal device to solve the problem that existing target tracking methods are difficult to cope with uncertain and dynamically changing environments, resulting in low tracking success rate and robustness.

[0005] A first aspect of an embodiment of the present application provides a target tracking method, which may include:

[0006] Get the target video frame;

[0007] Performing target detection on the target video frame to obtain a target detection result;

[0008] Determining a target prediction result corresponding to the target video frame based on a distributed robust optimized Kalman filter;

[0009] Matching the target detection result with the target prediction result to obtain a target matching result;

[0010] Based on the target matching result, a target tracking result is determined.

[0011] In a specific implementation of the first aspect, determining a target prediction result corresponding to the target video frame based on a distributed robust optimized Kalman filter includes:

[0012] Obtaining a previous update result; wherein the previous update result is an update result of the tracking target in a previous video frame, and the previous video frame is a video frame before the target video frame;

[0013] Based on the previously updated result, the target prediction result is predicted using the distributed robust optimized Kalman filter.

[0014] In a specific implementation of the first aspect, obtaining the previous update result includes:

[0015] Acquire a previous detection result and a previous prediction result; wherein the previous detection result is a detection result of the tracked target in the previous video frame, and the previous prediction result is a prediction result of the tracked target in the previous video frame;

[0016] Matching the previous detection result with the previous prediction result to obtain a previous matching result;

[0017] The previous matching result is updated by using the distributed robust optimized Kalman filter to obtain the previous updated result.

[0018] In a specific implementation of the first aspect, the previous matching result includes a previous matching success result and a previous matching failure result;

[0019] The method of updating the previous matching result by using the distributed robust optimized Kalman filter to obtain the previous updated result includes:

[0020] Using the distributed robust optimized Kalman filter, the previous successful matching result is updated to obtain a successful matching update result;

[0021] Based on the previous match failure result, determine a previous reserved result; wherein the previous reserved result is the previous match failure result reserved in the previous video frame;

[0022] The previous update result is determined based on the matching successful update result and the previous reserved result.

[0023] In a specific implementation of the first aspect, the using the distributed robust optimized Kalman filter to update the previous successful matching result to obtain a successful matching update result includes:

[0024] Based on the previous successful matching results, a fuzzy set is established; wherein the fuzzy set is used to describe distribution uncertainty;

[0025] Based on the fuzzy set, determining the target minimization maximization problem;

[0026] Solve the target minimization and maximization problem to obtain the matching success update result.

[0027] In a specific implementation of the first aspect, matching the target detection result with the target prediction result to obtain a target matching result includes:

[0028] Using a preset appearance feature extraction model, extracting appearance features from the target detection result to obtain a first appearance feature; wherein the appearance feature extraction model is a pre-trained artificial intelligence model for performing appearance feature extraction;

[0029] Using the appearance feature extraction model, extracting the appearance feature of the target prediction result to obtain a second appearance feature;

[0030] calculating a similarity between the first appearance feature and the second appearance feature;

[0031] Based on the degree of similarity, the target detection result is matched with the target prediction result to obtain the target matching result.

[0032] In a specific implementation of the first aspect, after matching the target detection result with the target prediction result to obtain a target matching result, the method further includes:

[0033] Based on the target matching result, the target prediction result is updated using the distributed robust optimized Kalman filter to obtain a target update result;

[0034] Based on the target update result, the distributed robust optimized Kalman filter is used to predict subsequent prediction results; wherein the subsequent prediction results are prediction results of the tracked target in subsequent video frames, and the subsequent video frame is a video frame after the target video frame.

[0035] A second aspect of an embodiment of the present application provides a target tracking device, which may include:

[0036] A target video frame acquisition module, used to acquire a target video frame;

[0037] A target detection module is used to perform target detection on the target video frame to obtain a target detection result;

[0038] A prediction result determination module, used to determine a target prediction result corresponding to the target video frame based on a distributed robust optimized Kalman filter;

[0039] A result matching module, used to match the target detection result with the target prediction result to obtain a target matching result;

[0040] The tracking result determination module is used to determine the target tracking result based on the target matching result.

[0041] In a specific implementation of the second aspect, the prediction result determination module includes:

[0042] A previous update result acquisition submodule, used to acquire a previous update result; wherein the previous update result is an update result of the tracking target in a previous video frame, and the previous video frame is a video frame before the target video frame;

[0043] The target prediction result prediction submodule is used to predict the target prediction result based on the previous update result using the distributed robust optimized Kalman filter.

[0044] In a specific implementation of the second aspect, the previous update result acquisition submodule includes:

[0045] A previous result acquisition unit, used to acquire a previous detection result and a previous prediction result; wherein the previous detection result is a detection result of a tracked target in the previous video frame, and the previous prediction result is a prediction result of a tracked target in the previous video frame;

[0046] A previous result matching unit, used for matching the previous detection result with the previous prediction result to obtain a previous matching result;

[0047] The previous matching result updating unit is used to update the previous matching result by using the distributed robust optimized Kalman filter to obtain the previous updated result.

[0048] In a specific implementation of the second aspect, the previous matching result includes a previous matching success result and a previous matching failure result;

[0049] The previous matching result updating unit comprises:

[0050] A previous matching result updating subunit is used to update the previous successful matching result by using the distributed robust optimized Kalman filter to obtain a successful matching update result;

[0051] A previously retained result determination subunit, configured to determine a previously retained result based on the previously failed matching result; wherein the previously retained result is the previously failed matching result retained in the previously video frame;

[0052] The previous update result determination subunit is used to determine the previous update result based on the matching successful update result and the previous reserved result.

[0053] In a specific implementation of the second aspect, the previous matching result updating subunit includes:

[0054] A fuzzy set establishment subunit, used to establish a fuzzy set based on the previous successful matching result; wherein the fuzzy set is used to describe distribution uncertainty;

[0055] A target problem determination subunit, used for determining a target minimization maximization problem based on the fuzzy set;

[0056] The target problem solving subunit is used to solve the target minimization and maximization problem to obtain the matching success update result.

[0057] In a specific implementation of the second aspect, the result matching module includes:

[0058] A first appearance feature extraction submodule, configured to extract appearance features from the target detection result using a preset appearance feature extraction model to obtain a first appearance feature; wherein the appearance feature extraction model is a pre-trained artificial intelligence model for performing appearance feature extraction;

[0059] A second appearance feature extraction submodule, configured to extract appearance features from the target prediction result using the appearance feature extraction model to obtain a second appearance feature;

[0060] a similarity calculation submodule, configured to calculate the similarity between the first appearance feature and the second appearance feature;

[0061] The result matching submodule is used to match the target detection result with the target prediction result based on the similarity level to obtain the target matching result.

[0062] In a specific implementation of the second aspect, the target tracking device further includes:

[0063] A target prediction result updating module is used to update the target prediction result based on the target matching result by using the distributed robust optimized Kalman filter to obtain a target update result;

[0064] A subsequent prediction result prediction module is used to predict subsequent prediction results based on the target update result using the distributed robust optimized Kalman filter; wherein the subsequent prediction result is the prediction result of the tracked target in the subsequent video frame, and the subsequent video frame is the next video frame after the target video frame.

[0065] A third aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned target tracking methods are implemented.

[0066] The fourth aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned target tracking methods when executing the computer program.

[0067] A fifth aspect of an embodiment of the present application provides a computer program product. When the computer program product is executed on a terminal device, the terminal device executes the steps of any one of the above-mentioned target tracking methods.

[0068] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application obtain a target video frame; perform target detection on the target video frame to obtain a target detection result; determine a target prediction result corresponding to the target video frame based on a distributionally robust optimized Kalman filter; match the target detection result with the target prediction result to obtain a target matching result; determine a target tracking result based on the target matching result. In the embodiments of the present application, the distributionally robust optimized Kalman filter can be used to predict the target prediction result; accordingly, various uncertainties can be fully considered, so that more accurate target prediction can be provided in the face of unknown noise, which helps to improve the success rate and robustness of the tracking process. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0070] Figure 1 This is a flow chart of an embodiment of a target tracking method in an embodiment of the present application;

[0071] Figure 2 It is a schematic diagram of the structure of YOLOv10;

[0072] Figure 3 A schematic diagram of the process of obtaining the previous update result;

[0073] Figure 4 A schematic diagram of the process of obtaining previous matching results;

[0074] Figure 5 Another schematic diagram of the process of obtaining the previous update result;

[0075] Figure 6 A schematic diagram of the updating process of the motion trajectory of the tracking target;

[0076] Figure 7 This is a structural diagram of an embodiment of a target tracking device in an embodiment of the present application;

[0077] Figure 8 This is a schematic block diagram of a terminal device in an embodiment of the present application. DETAILED DESCRIPTION

[0078] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0079] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0080] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0081] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0082] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0083] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0084] Multi-Object Tracking (MOT) is an important research topic in the field of computer vision and navigation, which involves the problem of simultaneously tracking multiple moving objects in a video sequence. Multi-object tracking is crucial for many applications such as video surveillance, self-driving cars, drone monitoring, motion analysis, and military reconnaissance.

[0085] In the field of multi-target tracking, Deep Simple Online and Realtime Tracking (DeepSORT) is a commonly used algorithm that combines target detection and data association technology to perform efficient target tracking. Specifically, DeepSort can use the classic Kalman Filter (KF) to predict the state of the tracked target in the kth frame based on the posterior state of the tracked target in the k-1th frame to obtain the prior state of the tracked target in the kth frame. After that, the prior state of the tracked target in the kth frame can be data-associated with the state of the tracked target detected in the kth frame to determine the position of the tracked target in the kth frame, and the prior state of the tracked target in the kth frame can be updated according to the data association result of the kth frame to obtain the posterior state of the tracked target in the kth frame. According to the position of the tracked target in each frame, the motion trajectory corresponding to the tracked target can be formed. Through the prediction and update of the Kalman filter, DeepSort can perform efficient tracking.

[0086] However, the actual tracking environment is in a dynamic change and full of uncertainty. For example, the actual movement of the target is complex and irregular, and often does not conform to the uniform linear motion assumed in the classic Kalman filter. In addition, in actual applications, it may be affected by more noise, such as noise caused by light changes and sensor observation noise. Existing target tracking methods are difficult to cope with uncertain and dynamically changing environments, and the tracking success rate and robustness are not high.

[0087] In view of this, embodiments of the present application provide a target tracking method, apparatus, computer-readable storage medium and terminal device to solve the problem that existing target tracking methods are difficult to cope with uncertain and dynamically changing environments, resulting in low tracking success rate and robustness.

[0088] It should be noted that the executor of the method of the present application is a terminal device, specifically, it can be a monitoring device, a car, a robot, an aircraft or other computing device, or it can be other computing devices.

[0089] In an embodiment of the present application, a moment-based distributionally robust Kalman filter (MKF) can be used to predict and update the state of the tracked target, so as to more robustly handle uncertainties in a dynamic environment and improve the tracking success rate.

[0090] Specifically, see Figure 1 In an embodiment of the present application, an embodiment of a target tracking method may include the following steps S101 to S105:

[0091] Step S101: Acquire a target video frame.

[0092] In the embodiment of the present application, the motion trajectory of each target in the video can be tracked.

[0093] For example, in the application scenario of traffic control, monitoring equipment can be used to track the movement trajectory of vehicles in the road area to better determine whether the vehicle has violated regulations such as speeding; in the application scenario of criminal inspection, monitoring equipment can be used to track the movement trajectory of suspects in the arrest area to better arrest the suspects; in the application scenario of sports events, drones can be used to track the movement trajectory of athletes to provide spectators with a more dynamic viewing experience.

[0094] Specifically, each video frame in the video may be detected frame by frame, and data of the tracking targets in different video frames may be associated to obtain the positions of the tracking targets in different video frames, thereby determining the motion trajectory of the tracking targets.

[0095] For ease of description, the target tracking method of the embodiment of the present application will be described below by taking any video frame in a video that requires target tracking (referred to as a target video frame) as an example.

[0096] Step S102: Perform target detection on the target video frame to obtain a target detection result.

[0097] In an embodiment of the present application, a preset target detection model can be used to detect the position of the tracking target in the target video frame to obtain a target detection result.

[0098] Among them, the target detection model can be any common artificial intelligence model for target detection in the prior art, specifically including but not limited to YOLO series, Convolutional Neural Networks (CNN), Region-based Convolutional Neural Network (R-CNN), Faster Region-based Convolutional Neural Network (Faster R-CNN) and other common artificial intelligence models for target detection, and this application does not make specific limitations on this.

[0099] In the embodiment of the present application, YOLOv10 can be preferably used to perform target detection on the target video frame. Since YOLOv10 can detect targets of different scales more stably, the robustness of target detection can be improved.

[0100] Figure 2 A schematic diagram of the structure of YOLOv10 is shown; specifically, the structure of YOLOv10 can include a dual label assignment module and a consistent matching metric module, and the dual label assignment module can include a backbone network (Backbone), a neck network, a one-to-many head module and a one-to-one head module; wherein the backbone network specifically adopts an enhanced version of the cross stage partial network (CSPNet) for feature extraction to improve gradient flow and reduce computational redundancy; the neck network specifically includes a path aggregation network (PAN) layer for realizing multi-scale feature fusion; the one-to-many head label assignment module can generate multiple predictions for each object, while the one-to-one head label assignment module can generate an optimal prediction for each object; by combining these two label assignment strategies, rich supervision in training and efficient end-to-end deployment can be guaranteed, thereby realizing accurate detection of small targets in complex backgrounds; and the consistent matching metric (Consistent Matching Metric) can be used to evaluate the consistency between the predicted results and the true results.

[0101] It should be noted that the results (including prediction results, detection results and update results) referred to in the embodiments of the present application are all the positions of the tracked targets, which can be specifically expressed in the form of detection boxes.

[0102] Step S103: Determine a target prediction result corresponding to the target video frame based on the distributed robust optimized Kalman filter.

[0103] In an embodiment of the present application, a distributed robust optimized Kalman filter can be used to predict the position of each tracking target in the target video frame to obtain a target prediction result.

[0104] Specifically, the position of each tracking target in the target video frame can be predicted based on the previous update result, wherein the previous update result is the update result of the tracking target in the previous video frame, and the previous video frame is the video frame before the target video frame.

[0105] For ease of understanding, the process of obtaining the previous update result will be described first below.

[0106] See also Figure 3 In the embodiment of the present application, a previous detection result and a previous prediction result can be obtained; wherein the previous detection result is the detection result of the tracked target in the previous video frame, and the previous prediction result is the prediction result of the tracked target in the previous video frame. Afterwards, the previous detection result can be matched with the previous prediction result to obtain a previous matching result; after obtaining the previous matching result, the previous matching result can be updated using a distributed robust Kalman filter to obtain a previous updated result.

[0107] By utilizing the distributed robust Kalman filter, various uncertainties in the detection and prediction process can be fully considered, thereby providing more accurate target prediction in the face of unknown noise, laying the foundation for accurate and stable target tracking.

[0108] Specifically, the target detection model can be used to detect the position of the tracked target in the previous video frame to obtain the previous detection result; the distributed robust Kalman filter can also be used to predict the position of the tracked target in the previous video frame to obtain the previous prediction result. The above two processes are independent of each other. The embodiment of the present application does not limit the execution order of the above two processes. In practical applications, the two processes can be executed in parallel, or one of the processes can be executed first and then the other process.

[0109] In an embodiment of the present application, after obtaining the previous detection result and the previous prediction result, cascade matching can be performed on the two. Specifically, the appearance features corresponding to the previous detection result and the previous prediction result can be extracted first, and then cascade matching can be performed based on the extracted appearance features. Accordingly, when the target's motion pattern is more complex, matching can be performed based on more reliable and stable appearance features to better adapt to complex scenes and improve the usability of the target tracking method.

[0110] Here, a preset appearance feature extraction model can be used to extract appearance features of the image in the detection box corresponding to the previous detection result, and appearance features can be extracted from the image in the detection box corresponding to the previous prediction result, to obtain appearance features corresponding to the two results. The appearance feature extraction model is a pre-trained artificial intelligence model for extracting appearance features.

[0111] In an embodiment of the present application, a training data set may be obtained to train an initial artificial intelligence model to obtain an appearance feature extraction model of an embodiment of the present application. Here, the initial artificial intelligence model may preferably be AlexNet; the training data set may include a preset number of sample images including the tracking target and the corresponding marked appearance features.

[0112] During training, the sample images in the training sample set can be used as the input of the initial artificial intelligence model, and the corresponding marked appearance features can be used as the expected output of the initial artificial intelligence model; after obtaining the actual appearance features of the model output, the training loss value in the training process can be calculated based on the marked appearance features, the actual appearance features and the preset loss function; based on the training loss value, the initial artificial intelligence model can be fine-tuned; when the preset stop training conditions are met, the training of the model can be terminated, and the fine-tuned artificial intelligence model can be used as the appearance feature extraction model of the embodiment of the present application.

[0113] After the appearance feature extraction model is trained, it can be applied to actual appearance feature extraction applications; here, the trained appearance feature extraction model can be used to extract appearance features of the image in the detection box corresponding to the previous detection result and the image in the detection box corresponding to the previous prediction result to obtain the corresponding appearance features.

[0114] Afterwards, the similarity between the appearance features of the previous detection result and the appearance features of the previous prediction result may be calculated. Here, the cosine similarity between the two appearance features may be used as the similarity between the two appearance features.

[0115] When performing cascade matching on the two, the target that has not been lost can be matched first, and the target that has been lost the longest can be matched last to improve the matching accuracy. In the specific matching process, the Hungarian matching algorithm can be used to perform more efficient cascade matching on the similarity of appearance features, such as Figure 4 shown.

[0116] For the previous detection results and the previous prediction results that failed to match successfully during the cascade matching process, a secondary matching can be performed.

[0117] When performing secondary matching in the prior art, a matching strategy based on Intersection over Union (IoU) is usually used for secondary matching. Specifically, if the IoU between the detection box of the previous detection result and the detection box of the previous prediction result for secondary matching is greater than the preset IoU threshold, it can be considered that the two are matched successfully; and if the IoU between the detection box of the previous detection result and the detection box of the previous prediction result is less than or equal to the IoU threshold, it can be considered that the two are matched unsuccessfully. However, when the scale of the tracked target changes, there may be a certain difference between the size of the previous detection result and the size of the previous prediction result, and the value of the IoU of the two will be greatly affected; and if there is a slight position difference between the detection box of the previous detection result and the detection box of the previous prediction result, the IoU of the two will present a significantly smaller value; that is, the value of IoU is easily affected by the size and position of the comparison object, and it is difficult to stably measure the similarity between objects.

[0118] Considering the great instability of the secondary matching based on IoU, the embodiment of the present application may preferably adopt a matching strategy based on the normalized Wasserstein distance (NWD) for secondary matching; specifically, the NWD between the detection frame of the previous detection result and the detection frame of the previous prediction result may be calculated; if the NWD is less than a preset distance threshold, it may be considered that the previous detection result and the previous prediction result have failed to match; if the NWD is greater than or equal to the distance threshold, it may be considered that the previous detection result and the previous prediction result have successfully matched. Among them, the distance threshold may be set according to actual needs, and the present application does not limit this.

[0119] Compared with IoU, NWD is insensitive to the size and position of the compared objects, so it can more stably measure the similarity between the compared objects and improve the matching accuracy and credibility.

[0120] In the embodiment of the present application, the results of successful matching in the cascading process and successful matching in the secondary matching can be used as the previous successful matching results; and the results of unsuccessful matching in the secondary matching can be used as the previous failed matching results.

[0121] For previous failed matching results, the previous failed matching results that need to be retained (referred to as previously retained results) can be determined; specifically, among the previous failed matching results, if the number of consecutive frames in which a certain result fails to match is greater than a preset frame number threshold (TLost), the tracking of the result can be terminated; if the number of consecutive frames in which the result fails to match is less than or equal to the frame number threshold, the result can be determined as the previously retained result; wherein, the frame number threshold can be set according to actual needs, and this application does not limit this.

[0122] For example, TLost is 20; if a result fails to match within 20 consecutive frames, the tracking of the result can be terminated; if the result fails to match within 3 consecutive frames, since the number of consecutive frames in which the result fails to match is less than 20, the result can be determined as the previously retained result.

[0123] By filtering the previously retained results, it is possible to stop tracking a target that has been lost for a long time, while continuing to track a target that has been lost for a short time, thereby optimizing the allocation of computing resources during the tracking process.

[0124] For the previous successful matching results, the distributed robust optimized Kalman filter can be used to update them to obtain the successful matching update results.

[0125] Specifically, for linear systems, we have:

[0126] x t =F t x t-1 +G t w t-1

[0127] y t =H t x t +v t

[0128] Among them, x t is the state variable at time t, x t-1 is the state variable at time t-1, y t is the observed variable at time t, w t-1 is the process noise at time t-1, v t is the observation noise at time t; F t is the nominal system matrix, G t is the covariance matrix of process noise, H t is the state observation matrix. t , G t and H t All are set according to the uniform linear motion model.

[0129] In the embodiment of the present application, the state variables include the center point of the detection box (c x ,c y ) and the corresponding rate of change v x and v y , the width w of the detection box and its rate of change v w , the height h of the detection box and its rate of change v h ; The state variable can be recorded as x t :=(c x,v x ,c y ,v y ,w,v w ,h,v h ).

[0130] For the state variable x at time t t and the observed variable y t , we can determine the joint variable z of the two t , specifically, z t :=[x t T ,y t T ] T .

[0131] Here, it can be considered that z t It conforms to the Gaussian distribution, that is, z t =N(c z ,S z ); N(c z ,S z ) is a possible distribution that takes into account distribution uncertainty, which is specifically the nominal distribution N(μ z ,Σ z ), the nominal distribution is the distribution under the assumption of the classical Kalman filter without considering the distribution uncertainty; where z t The mean vector of Covariance matrix S xx is the variance of the state variable, S yy is the variance of the observed variable, S xy and S yx are the covariances of the state variables and the observed variables.

[0132] Similarly, the nominal distribution N(μ z ,Σ z ) has a mean vector μ x is the mean of the state variable, μ y is the mean of the observed variables, the covariance matrix Σ xx is the variance of the state variable, Σ yy is the variance of the observed variable, Σ xy and Σ yx are the covariances of the state variables and the observed variables.

[0133] For z t , we can establish the following fuzzy set P M :

[0134]

[0135] in, is the fuzzy set P M members of Yes t possible distribution, γ1, γ2, and γ3 are the first adjustable parameter, the second adjustable parameter, and the third adjustable parameter, respectively. The specific values ​​of the three adjustable parameters can be specific and situationally set according to actual conditions, and this application does not limit this; F is z t The set of all probability distribution density functions with finite second-order moments in . By establishing fuzzy sets for describing distribution uncertainty, various uncertainties in dynamic environments can be better considered and robust estimates of state variables can be provided.

[0136] Based on P M , we can establish the target minimization problem (minimax problem):

[0137]

[0138] Where ψ is y t to x t An affine estimator of , L is y t to x t The set of affine estimators of ; solving the target minimization and maximization problem can obtain the matching successful update result. Specifically, it can be solved to obtain:

[0139]

[0140] in, is the state variable x t The corresponding previous matching success update result, V t∣t is the state variable x t The corresponding covariance matrix is, For S xx The corresponding solution result at time t is: For S xy The corresponding solution result at time t is: For S yy The corresponding solution result at time t is: For S yx The corresponding solution result at time t, c x,t c x The corresponding solution result at time t, c y,t c y The corresponding solution result at time t.

[0141] Based on the previous matching successful update result and the previous reserved result, the previous update result can be determined, such as Figure 5 shown.

[0142] Afterwards, the target prediction result can be predicted using the distributed robust optimized Kalman filter based on the previous update results.

[0143] Specifically, the target prediction result can be obtained by referring to the following formula:

[0144]

[0145] V t+11∣t =F t V t∣t F t T +G t Q t G t T

[0146] in, is the target prediction result, Q t V t∣t The covariance of .

[0147] Step S104: Match the target detection result with the target prediction result to obtain a target matching result.

[0148] In an embodiment of the present application, after obtaining the target prediction result, the target detection result can be matched with the target prediction result to obtain a target matching result.

[0149] Specifically, the appearance feature extraction model can be used to extract appearance features from the target detection result to obtain the first appearance feature; the appearance feature extraction model can also be used to extract appearance features from the target prediction result to obtain the second appearance feature.

[0150] Afterwards, the cosine similarity between the first appearance feature and the second appearance feature may be calculated as the degree of similarity between the first appearance feature and the second appearance feature; based on the calculated degree of similarity, the target detection result and the target prediction result may be matched to obtain a target matching result.

[0151] The specific matching process can refer to the above description, using the matching strategies of cascade matching, Hungarian matching and secondary matching based on NWD, which will not be described in detail here.

[0152] In complex environments, the size and motion pattern of the target may change to a certain extent, but the appearance features of the target will remain basically stable. Therefore, matching the results by appearance features can improve the robustness and accuracy of the matching and lay the foundation for stable target tracking.

[0153] Step S105: Determine the target tracking result based on the target matching result.

[0154] In an embodiment of the present application, a target tracking result may be determined based on a target matching result; specifically, the target matching result may include a target matching success result and a target matching failure result.

[0155] For a successful target matching result, an updated motion trajectory can be determined based on the historical motion trajectory corresponding to the successful target matching result, such as Figure 6 shown.

[0156] In addition, the unique identifications (IDs) of the two successfully matched results may be associated with the same value to better associate the data.

[0157] For a target match failure result, a target retention result can be determined based on the target match failure result. The target retention result is a target match failure result that needs to be retained in the target video frame. Specifically, in the target match failure result, if the number of consecutive frames in which a certain result fails to match is greater than the frame number threshold (TLost), the tracking of the result can be terminated; if the number of consecutive frames in which the result fails to match is less than or equal to the frame number threshold, the result can be determined as a target retention result.

[0158] In addition, if there is a newly appeared tracking target in the target matching failure result, a corresponding ID can be set for the newly appeared tracking target so as to continue tracking the target in subsequent video frames.

[0159] In an embodiment of the present application, after obtaining the target matching result, the target prediction result can be updated by using the distributed robust optimized Kalman filter to obtain a target update result; the specific update process can refer to the above description, and the corresponding fuzzy set can be set to describe the distribution uncertainty, and the target minimization maximization problem can be set to solve to obtain the target update result, which will not be repeated here.

[0160] After obtaining the target update result, the distributed robust optimized Kalman filter can also be used to predict the subsequent prediction result; wherein the subsequent prediction result is the prediction result of the tracked target in the subsequent video frame, and the subsequent video frame is the video frame after the target video frame. The prediction process here can refer to the above description and will not be repeated here.

[0161] In a specific implementation of an embodiment of the present application, for the target retention result, the corresponding formula can be used to predict according to the aforementioned process, or its state can be kept unchanged, that is, the target retention result is directly used as its corresponding subsequent prediction result.

[0162] Based on this, the target detection results and target prediction results can be combined to continuously iterate and update the state of the tracked target, which helps to maintain the continuity and accuracy of the trajectory.

[0163] In summary, the embodiment of the present application obtains a target video frame; performs target detection on the target video frame to obtain a target detection result; determines a target prediction result corresponding to the target video frame based on a distributionally robust optimized Kalman filter; matches the target detection result with the target prediction result to obtain a target matching result; and determines a target tracking result based on the target matching result. In the embodiment of the present application, a distributionally robust optimized Kalman filter can be used to predict a target prediction result; accordingly, various uncertainties can be fully considered, thereby being able to provide more accurate target predictions in the face of unknown noise, which helps to improve the success rate and robustness of the tracking process.

[0164] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0165] Corresponding to a target tracking method described in the above embodiment, Figure 7 A structural diagram of an embodiment of a target tracking device provided in an embodiment of the present application is shown.

[0166] In an embodiment of the present application, a target tracking device may include:

[0167] The target video frame acquisition module 701 is used to acquire the target video frame;

[0168] The target detection module 702 is used to perform target detection on the target video frame to obtain a target detection result;

[0169] A prediction result determination module 703, configured to determine a target prediction result corresponding to the target video frame based on a distributed robust optimized Kalman filter;

[0170] A result matching module 704 is used to match the target detection result with the target prediction result to obtain a target matching result;

[0171] The tracking result determination module 705 is used to determine the target tracking result based on the target matching result.

[0172] In a specific implementation of the embodiment of the present application, the prediction result determination module includes:

[0173] A previous update result acquisition submodule, used to acquire a previous update result; wherein the previous update result is an update result of the tracking target in a previous video frame, and the previous video frame is a video frame before the target video frame;

[0174] The target prediction result prediction submodule is used to predict the target prediction result based on the previous update result using the distributed robust optimized Kalman filter.

[0175] In a specific implementation of the embodiment of the present application, the previous update result acquisition submodule includes:

[0176] A previous result acquisition unit, used to acquire a previous detection result and a previous prediction result; wherein the previous detection result is a detection result of a tracked target in the previous video frame, and the previous prediction result is a prediction result of a tracked target in the previous video frame;

[0177] A previous result matching unit, used for matching the previous detection result with the previous prediction result to obtain a previous matching result;

[0178] The previous matching result updating unit is used to update the previous matching result by using the distributed robust optimized Kalman filter to obtain the previous updated result.

[0179] In a specific implementation of the embodiment of the present application, the previous matching result includes a previous matching success result and a previous matching failure result;

[0180] The previous matching result updating unit comprises:

[0181] A previous matching result updating subunit is used to update the previous successful matching result by using the distributed robust optimized Kalman filter to obtain a successful matching update result;

[0182] A previously retained result determination subunit, configured to determine a previously retained result based on the previously failed matching result; wherein the previously retained result is the previously failed matching result retained in the previously video frame;

[0183] The previous update result determination subunit is used to determine the previous update result based on the matching successful update result and the previous reserved result.

[0184] In a specific implementation of the embodiment of the present application, the previous matching result updating subunit includes:

[0185] A fuzzy set establishment subunit, used to establish a fuzzy set based on the previous successful matching result; wherein the fuzzy set is used to describe distribution uncertainty;

[0186] A target problem determination subunit, used for determining a target minimization maximization problem based on the fuzzy set;

[0187] The target problem solving subunit is used to solve the target minimization and maximization problem to obtain the matching success update result.

[0188] In a specific implementation of the embodiment of the present application, the result matching module includes:

[0189] A first appearance feature extraction submodule, configured to extract appearance features from the target detection result using a preset appearance feature extraction model to obtain a first appearance feature; wherein the appearance feature extraction model is a pre-trained artificial intelligence model for performing appearance feature extraction;

[0190] A second appearance feature extraction submodule, configured to extract appearance features from the target prediction result using the appearance feature extraction model to obtain a second appearance feature;

[0191] a similarity calculation submodule, configured to calculate the similarity between the first appearance feature and the second appearance feature;

[0192] The result matching submodule is used to match the target detection result with the target prediction result based on the similarity level to obtain the target matching result.

[0193] In a specific implementation of the embodiment of the present application, the target tracking device further includes:

[0194] A target prediction result updating module is used to update the target prediction result based on the target matching result by using the distributed robust optimized Kalman filter to obtain a target update result;

[0195] A subsequent prediction result prediction module is used to predict subsequent prediction results based on the target update result using the distributed robust optimized Kalman filter; wherein the subsequent prediction result is the prediction result of the tracked target in the subsequent video frame, and the subsequent video frame is the next video frame after the target video frame.

[0196] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0197] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0198] Figure 8 A schematic block diagram of a terminal device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0199] like Figure 8As shown, the terminal device 8 of this embodiment includes: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80. When the processor 80 executes the computer program 82, the steps in the above-mentioned target tracking method embodiments are implemented, for example Figure 1 Alternatively, when the processor 80 executes the computer program 82, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 7 The functions of modules 701 to 705 are shown.

[0200] Exemplarily, the computer program 82 may be divided into one or more modules / units, which are stored in the memory 81 and executed by the processor 80 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 82 in the terminal device 8.

[0201] Those skilled in the art will understand that Figure 8 It is only an example of the terminal device 8 and does not constitute a limitation of the terminal device 8. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 8 may also include input and output devices, network access devices, buses, etc.

[0202] The processor 80 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 80 may be the nerve center and command center of the terminal device 8. The processor 80 may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0203] The memory 81 may be an internal storage unit of the terminal device 8, such as a hard disk or memory of the terminal device 8. The memory 81 may also be an external storage device of the terminal device 8, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 8. Further, the memory 81 may also include both an internal storage unit of the terminal device 8 and an external storage device. The memory 81 is used to store the computer program and other programs and data required by the terminal device 8. The memory 81 may also be used to temporarily store data that has been output or is to be output.

[0204] The terminal device 8 may also include a communication module, which may provide communication solutions including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth, Zigbee, mobile communication networks, global navigation satellite systems (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to network devices. The communication module may be one or more devices integrating at least one communication processing module. The communication module may include an antenna, which may have only one array element or an antenna array including multiple array elements. The communication module may receive electromagnetic waves through the antenna, frequency modulate and filter the electromagnetic wave signals, and send the processed signals to the processor. The communication module may also receive the signal to be sent from the processor, frequency modulate and amplify it, and convert it into electromagnetic waves for radiation through the antenna.

[0205] The terminal device 8 may further include a power management module, which may receive input from an external power source, a battery and / or a charger to power the processor, the memory, the communication module, and the like.

[0206] The terminal device 8 may further include a display module, which may be used to display information input by a user or information provided to a user. The display module may include a display panel, and optionally, the display panel may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel may cover the display panel, and when the touch panel detects a touch operation on or near it, it is transmitted to the processor to determine the type of the touch event, and then the processor provides a corresponding visual output on the display panel according to the type of the touch event.

[0207] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0208] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0209] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0210] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0211] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0212] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0213] An embodiment of the present application provides a computer program product. When the computer program product runs on the terminal device, the terminal device can implement the steps in the above-mentioned various method embodiments.

[0214] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0215] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A target tracking method, characterized in that: include: Get the target video frame; Performing target detection on the target video frame to obtain a target detection result; Determining a target prediction result corresponding to the target video frame based on a distributed robust optimized Kalman filter; Matching the target detection result with the target prediction result to obtain a target matching result; Based on the target matching result, a target tracking result is determined.

2. The target tracking method according to claim 1, characterized in that: The step of determining a target prediction result corresponding to the target video frame based on a distributed robust optimized Kalman filter comprises: Obtaining a previous update result; wherein the previous update result is an update result of the tracking target in a previous video frame, and the previous video frame is a video frame before the target video frame; Based on the previously updated result, the target prediction result is predicted using the distributed robust optimized Kalman filter.

3. The target tracking method according to claim 2, characterized in that: The obtaining of the previous update result includes: Acquire a previous detection result and a previous prediction result; wherein the previous detection result is a detection result of the tracked target in the previous video frame, and the previous prediction result is a prediction result of the tracked target in the previous video frame; Matching the previous detection result with the previous prediction result to obtain a previous matching result; The previous matching result is updated by using the distributed robust optimized Kalman filter to obtain the previous updated result.

4. The target tracking method according to claim 3, characterized in that: The previous matching results include previous matching success results and previous matching failure results; The method of updating the previous matching result by using the distributed robust optimized Kalman filter to obtain the previous updated result includes: Using the distributed robust optimized Kalman filter, the previous successful matching result is updated to obtain a successful matching update result; Based on the previous match failure result, determine a previous retained result; wherein the previous retained result is the previous match failure result retained in the previous video frame; The previous update result is determined based on the matching successful update result and the previous reserved result.

5. The target tracking method according to claim 4, characterized in that: The method of using the distributed robust optimized Kalman filter to update the previous successful matching result to obtain a successful matching update result includes: Based on the previous successful matching results, a fuzzy set is established; wherein the fuzzy set is used to describe distribution uncertainty; Based on the fuzzy set, determining the target minimization maximization problem; Solve the target minimization and maximization problem to obtain the matching success update result.

6. The target tracking method according to claim 1, characterized in that: The matching of the target detection result with the target prediction result to obtain a target matching result includes: Using a preset appearance feature extraction model, extracting appearance features from the target detection result to obtain a first appearance feature; wherein the appearance feature extraction model is a pre-trained artificial intelligence model for performing appearance feature extraction; Using the appearance feature extraction model, extracting the appearance feature of the target prediction result to obtain a second appearance feature; calculating a similarity between the first appearance feature and the second appearance feature; Based on the degree of similarity, the target detection result is matched with the target prediction result to obtain the target matching result.

7. The target tracking method according to any one of claims 1 to 6, characterized in that: After matching the target detection result with the target prediction result to obtain a target matching result, the method further includes: Based on the target matching result, the target prediction result is updated using the distributed robust optimized Kalman filter to obtain a target update result; Based on the target update result, the distributed robust optimized Kalman filter is used to predict subsequent prediction results; wherein the subsequent prediction results are prediction results of the tracked target in subsequent video frames, and the subsequent video frame is a video frame after the target video frame.

8. A target tracking device, characterized in that: include: A target video frame acquisition module, used to acquire a target video frame; A target detection module is used to perform target detection on the target video frame to obtain a target detection result; A prediction result determination module, used to determine a target prediction result corresponding to the target video frame based on a distributed robust optimized Kalman filter; A result matching module, used to match the target detection result with the target prediction result to obtain a target matching result; The tracking result determination module is used to determine the target tracking result based on the target matching result.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the target tracking method according to any one of claims 1 to 7 are implemented.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the target tracking method according to any one of claims 1 to 7 are implemented.

Images