Target tracking method and system based on fusion twin network and Kalman filter

Abstract

The present invention belongs to the field of target tracking technology, and specifically relates to a target tracking method and system based on a fusion twin network and Kalman filtering, which aims to solve the problem that the existing technology only uses Kalman filtering technology to perform tracking tasks in complex scenes, and has low tracking accuracy and adaptability to target deformation. The present invention includes: given the position and size of the target in the initial frame; using the target template and the search area as an image pair to extract features and determine the search area; a candidate area generation network generates a multi-scale candidate box, a feature score map, and a regression parameter response map; performing edge penalty in the search area; if there is a candidate box with a confidence level greater than a threshold after the edge penalty, it is output as a prediction result; otherwise, the target motion state and the predicted target position are corrected; and the target motion information and position information are updated. The present invention effectively reduces the probability of erroneous updates of target motion information and position information during the tracking process, thereby improving the tracking success rate under complex backgrounds.

Description

Technical Field

[0001] The present invention belongs to the technical field of target tracking, and specifically relates to a target tracking method and system based on a fusion twin network and Kalman filtering. Background Art

[0002] Object tracking is a hot topic in computer vision research and is widely used in scenarios such as mobile agents, video surveillance, intelligent transportation, and the military. Object tracking primarily addresses the problem of predicting the target's position and size in a continuous video sequence, given an initial frame, to obtain information about the target in the next frame, such as its position coordinates, trajectory, speed, and deformation scale. Developing both high-precision and robust object tracking algorithms remains a challenging task.

[0003] Traditional correlation filtering methods have made significant progress and gained widespread adoption in the field of target tracking. For example, the Minimum Output Sum of Squared Error filter (MOSSE) applies correlation filtering technology to target tracking, achieving real-time target tracking through efficient algorithms. However, these correlation filtering tracking methods often experience tracking drift or failure in complex scenarios, such as occlusion, rapid target motion, and background interference. This significantly limits their performance. Although related research has improved standard correlation filters by addressing edge effects (Spatially Regularized Discriminative Correlation Filters, SRDCF), continuous convolution operators for tracking (C-COT), and volumetric Kalman filtering, making them more suitable for scenarios such as occlusion and rapid target motion, these filtering methods still suffer from several drawbacks. First, the model update strategy remains difficult to adapt to complex scenarios. Second, when the target deforms significantly, the changes in target features are difficult for correlation filtering algorithms to accurately estimate.

[0004] In recent years, with the rapid development of deep learning, some researchers have proposed applying Siamese networks to target tracking. This effort stems largely from the powerful representational capabilities of deep features extracted using deep learning. Deep learning structures such as Fully-Convolutional Siamese Networks (SiamFC) and SiamRPN (Siamese Region Proposal Network) draw on the RPN structure used in target detection to address the inflexibility of SiamFC tracking boxes while also providing relatively good feature extraction capabilities for tracking targets.

[0005] In general, this field also needs a target tracking method that integrates the Kalman filter algorithm with the deep learning twin network, so as to reduce the probability of erroneous updating of target motion information and position information during the tracking process and improve the tracking success rate in complex backgrounds. Summary of the Invention

[0006] In order to solve the above-mentioned problems in the prior art, namely, the prior art only uses Kalman filtering technology for tracking tasks in complex scenes, and the tracking accuracy and target deformation adaptability are low, the present invention provides a target tracking method based on a fusion of a twin network and a Kalman filter, the target tracking method comprising:

[0007] Step S10, using the target in the k-1th frame of the video frame containing the target as the target template to track the target in the kth frame, and initializing the Kalman filter based on the target position and size in the k-1th frame;

[0008] Step S20: Based on the target in the k-1 frame, a candidate region generation network is used in the k frame to generate a search candidate frame. The target template and the search region are used as an image pair, and the tracking feature is extracted and the search region is generated through the SiamFC twin network.

[0009] Step S30, generating a multi-scale candidate box, a feature score map, and a regression parameter response map in the search area, and performing an edge penalty based on the target center position in the search area to obtain the confidence of each candidate box;

[0010] Step S40: If there is a candidate box with a confidence level greater than a set threshold, the candidate box is output as the prediction result, and the target motion information and position information are updated; otherwise, a cubic Kalman filter with multiple fading factors is used to predict and correct the target motion state, predict the target position, and update the target motion information and position information;

[0011] In step S50 , k is set to k+1, and the process jumps to step S20 to continue target tracking until k is the last frame of the video, thus completing target tracking of the video frame.

[0012] In some preferred embodiments, the Kalman filter initialization includes:

[0013] Initialize the target state initial value to X0, initialize the covariance matrix to Q0, initialize the measurement noise covariance matrix to R0, and initialize the prediction error covariance matrix to P0.

[0014] In some preferred embodiments, step S20 includes:

[0015] Step S21, taking the target of the k-1 frame as the center, the target size w k-1 ×h k-1 Crop the target template in the k-1th frame, w k-1 and h k-1 are the width and height of the target in the k-1th frame respectively;

[0016] Step S22: In the kth frame, the target is centered and the target size is w. k-1 ×h k-1 Crop the search area in the kth frame;

[0017] Step S23: normalize the target template and the search area to w sm ×h sm The method uses a SiamFC twin network to extract tracking features of the target and the area to be searched; the SiamFC twin network includes two neural networks with the same structure and shared weights.

[0018] In some preferred embodiments, the multi-scale candidate box includes s kinds of changes; the feature score map generates a feature for the template of each of the s kinds of changes, and the candidate box corresponding to the feature with a score greater than a set threshold is the target; the regression parameter response map generates a set of regression parameters for the template of each of the s kinds of changes, and the higher the regression parameter score, the higher the scale accuracy of the corresponding candidate box.

[0019] In some preferred embodiments, the set of regression parameters includes an x-direction offset and a y-direction offset of the center of the candidate box of the kth frame relative to the center of the candidate box of the k-1th frame, and a width and height of the candidate box of the kth frame.

[0020] In some preferred embodiments, step S40 uses a cubic Kalman filter with multiple fading factors to predict and correct the target motion state, including:

[0021] Step S411, predicting the target motion state:

[0022] Calculate the volume point of the correction stage of the k-1th frame, where the i-th correction stage volume point is:

[0023]

[0024] in, is the i-th correction stage volume point of the k-1-th frame, S k-1|k-1 is the weight function of k-1 frame, is the estimated value of the modified state prediction for the k-1 frame, is the volume point set, m is a positive even number representing the number of volume points in the volume point set, is the i-th volume point;

[0025] Step S412, correct the target motion state:

[0026] Calculate the volume point of the correction stage of the kth frame, where the volume point of the i-th correction stage is:

[0027]

[0028] in, is the i-th correction stage volume point of the k-th frame, u k-1 is the Kalman filter multi-fading factor of the k-1th frame;

[0029] Get the corrected state prediction estimate, prediction error covariance matrix, and state prediction value for the kth frame:

[0030]

[0031]

[0032]

[0033]

[0034]

[0035] in, is the estimated value of the corrected state prediction of the kth frame; P k|k-1 is the prediction error covariance matrix of the kth frame, T is all the moments that need to be accumulated, Q k is the initialization covariance matrix of the kth frame; K k is the Kalman gain of the kth frame, Z k is the measurement estimate of the k-th frame, is the measurement prediction estimate of the k-th frame; represents the product of the correction phase volume point set of the k-th frame and the multi-fading factor of the Kalman filter of the k-th frame;

[0036] Step S413: Update target motion state parameters; the target motion state parameters include a state transfer matrix, a covariance matrix, a state prediction matrix, and a volume point set.

[0037] In some preferred embodiments, the candidate region generation network is trained by:

[0038] Acquire multiple video frames containing the target as a sample set, and generate a multi-scale candidate box, a feature score map, and a regression parameter response map corresponding to each video frame through the methods corresponding to steps S10 to S30;

[0039] Set the threshold th pos and th reg , calculate the IoU between the multi-scale candidate box and the corresponding real box, if IoU>th pos , then the candidate box is classified as a positive sample; if IoU>th reg When , the candidate box is classified as a negative sample;

[0040] Iteratively training the candidate region generation network through positive samples and negative samples;

[0041] The total loss function of the candidate region generation network is:

[0042] loss = L cls +λL reg

[0043] Among them, L cls The cross entropy loss function for the classification branch of the candidate region generation network, L reg Smooth-based regression branch for candidate region generation network L1 The loss function constructed by the function, λ is a preset hyperparameter used to balance the classification branch and the regression branch.

[0044] In some preferred embodiments, the smooth L1 The function is:

[0045]

[0046] Among them, σ is the preset hyperparameter, and x is the difference between the actual output of the regression branch of the candidate region generation network and the sample label.

[0047] In some preferred embodiments, the regression branch of the candidate region generation network is based on smooth L1 The loss function constructed by the function is expressed as:

[0048]

[0049]

[0050]

[0051]

[0052]

[0053] Among them, A x 、A y 、A w and A h They are the x-coordinate, y-coordinate, width and height of the multi-scale candidate box center, G x , G y , G w and G h They are the x-coordinate, y-coordinate, width and height of the real frame center.

[0054] Another aspect of the present invention provides a target tracking system based on a fusion twin network and a Kalman filter, the target tracking system comprising:

[0055] an initialization module configured to use the target in the k-1th frame of the video frame containing the target as a target template to track the target in the kth frame, and to initialize a Kalman filter based on the target position and size in the k-1th frame;

[0056] The pairing and feature extraction module is configured to generate a search candidate box in the kth frame using a candidate region generation network based on the target in the k-1th frame, and use the target template and the search region as an image pair to perform tracking feature extraction and generate a search region through the SiamFC twin network;

[0057] a candidate box generation and edge penalty module, configured to generate multi-scale candidate boxes, feature score maps, and regression parameter response maps in the search area, and perform edge penalty based on the target center position in the search area to obtain the confidence of each candidate box;

[0058] The target tracking and information updating module is configured to output the candidate box as the prediction result if there is a candidate box with a confidence level greater than a set threshold, and update the target motion information and position information; otherwise, it uses a cubature Kalman filter with multiple fading factors to predict and correct the target motion state, predict the target position, and update the target motion information and position information;

[0059] The continuous tracking module is configured to set k=k+1, and jumps to the pairing and feature extraction module to continue target tracking until k is the last frame of the video, completing the target tracking of the video frame.

[0060] Beneficial effects of the present invention:

[0061] (1) Based on the target tracking method integrating twin networks and Kalman filtering, the present invention proposes a target motion information and position information update strategy, which can effectively reduce the probability of erroneous update of target motion information and position information during the tracking process.

[0062] (2) The present invention is based on a target tracking method that integrates a twin network and a Kalman filter. The candidate region generation network uses the features extracted by the twin network to generate multi-scale candidate frames, feature score maps, and regression parameter response maps. Potential candidate targets are selected in the detection stage. The multi-fading factor cubature Kalman filter method can be used to correct the target motion state and predict the target position, thereby improving the tracking success rate under complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0064] Figure 1 It is a flow chart of the target tracking method based on the fusion of twin network and Kalman filter of the present invention;

[0065] Figure 2 Schematic diagram of the twin network structure of an embodiment of the target tracking method based on the fusion of twin network and Kalman filter of the present invention;

[0066] Figure 3 Schematic diagram of edge penalty function of an embodiment of a target tracking method based on fusion of twin network and Kalman filter of the present invention;

[0067] Figure 4 This is an edge penalty effect diagram of an embodiment of the target tracking method based on the fusion of twin network and Kalman filter of the present invention;

[0068] Figure 5 It is a flow chart of target position correction using volumetric Kalman filtering in an embodiment of a target tracking method based on a fusion of a twin network and a Kalman filter according to the present invention. DETAILED DESCRIPTION

[0069] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0070] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0071] The present invention provides a target tracking method based on a fusion twin network and a Kalman filter, the target tracking method comprising:

[0072] Step S10, using the target in the k-1th frame of the video frame containing the target as the target template to track the target in the kth frame, and initializing the Kalman filter based on the target position and size in the k-1th frame;

[0073] Step S20: Based on the target in the k-1 frame, a candidate region generation network is used in the k frame to generate a search candidate frame. The target template and the search region are used as an image pair, and the tracking feature is extracted and the search region is generated through the SiamFC twin network.

[0074] Step S30, generating a multi-scale candidate box, a feature score map, and a regression parameter response map in the search area, and performing an edge penalty based on the target center position in the search area to obtain the confidence of each candidate box;

[0075] Step S40: If there is a candidate box with a confidence level greater than a set threshold, the candidate box is output as the prediction result, and the target motion information and position information are updated; otherwise, a cubic Kalman filter with multiple fading factors is used to predict and correct the target motion state, predict the target position, and update the target motion information and position information;

[0076] In step S50 , k is set to k+1, and the process jumps to step S20 to continue target tracking until k is the last frame of the video, thus completing target tracking of the video frame.

[0077] In order to more clearly illustrate the target tracking method based on the fusion of twin network and Kalman filter of the present invention, the following is combined with Figure 1 Each step in the embodiment of the present invention is described in detail.

[0078] The target tracking method based on the fusion of the twin network and the Kalman filter according to the first embodiment of the present invention includes steps S10 to S50, each of which is described in detail as follows:

[0079] Step S10, using the target in the k-1th frame of the video frame containing the target as the target template to track the target in the kth frame, and initializing the Kalman filter based on the target position and size in the k-1th frame, including:

[0080] Initialize the target state initial value to X0, initialize the covariance matrix to Q0, initialize the measurement noise covariance matrix to R0, and initialize the prediction error covariance matrix to P0.

[0081] Step S20: Based on the target of the k-1 frame, a candidate region generation network is used in the k frame to generate a search candidate frame. The target template and the search region are used as an image pair. The tracking feature is extracted and the search region is generated through the SiamFC twin network:

[0082] Step S21, taking the target of the k-1 frame as the center, the target size w k-1 ×h k-1 Crop the target template in the k-1th frame, w k-1 and h k-1 are the width and height of the target in the k-1th frame respectively;

[0083] Step S22: In the kth frame, the target is centered and the target size is w. k-1 ×h k-1 Crop the search area in the kth frame;

[0084] Step S23: normalize the target template and the search area to w sm ×h sm The method uses a SiamFC twin network to extract tracking features of the target and the area to be searched; the SiamFC twin network includes two neural networks with the same structure and shared weights.

[0085] like Figure 2 As shown in FIG, a schematic diagram of the twin network structure of an embodiment of the target tracking method based on the fusion of twin network and Kalman filter of the present invention is shown. The target template and the search area are input into the twin network with the same network structure and shared weights to extract the features of the target and the area to be searched. X1 and X2 are two pictures of the target template and the search area. is the convolutional neural network used to extract features, W is the weight shared by the convolutional neural network, and It is the output after the convolutional network extracts features.

[0086] Step S30: Generate a multi-scale candidate box, a feature score map, and a regression parameter response map in the search area, and perform edge penalty based on the target center position in the search area to obtain the confidence of each candidate box:

[0087] Multi-scale candidate boxes include s kinds of changes; feature score map For each of the s changes in the template, a feature is generated. The candidate box corresponding to the feature with a score greater than the set threshold is the target, that is, the higher the feature score, the higher the possibility that the corresponding candidate box is the target; the regression parameter response graph generates a set of regression parameters for each of the s changes in the template. The higher the regression parameter score, the higher the scale accuracy of the corresponding candidate box.

[0088] A set of regression parameters includes the x-direction offset dx of the center of the candidate box of the kth frame relative to the center of the candidate box of the k-1 frame, the y-direction offset dy of the center of the candidate box of the kth frame relative to the center of the candidate box of the k-1 frame, the width dw of the candidate box of the kth frame, and the height dh of the candidate box of the kth frame.

[0089] The candidate region generation network is trained as follows:

[0090] Acquire multiple video frames containing the target as a sample set, and generate a multi-scale candidate box, a feature score map, and a regression parameter response map corresponding to each video frame through the methods corresponding to steps S10 to S30;

[0091] Set the threshold th pos and th reg , calculate the IoU between the multi-scale candidate box and the corresponding real box, if IoU>th pos , then the candidate box is classified as a positive sample; if IoU>th reg When , the candidate box is classified as a negative sample;

[0092] Iteratively training the candidate region generation network through positive samples and negative samples;

[0093] The total loss function of the candidate region generation network is shown in formula (1):

[0094] loss = L cls +λL reg (1)

[0095] Among them, L cls The cross entropy loss function for the classification branch of the candidate region generation network, L reg Smooth-based regression branch for candidate region generation network L1 The loss function constructed by the function, λ is a preset hyperparameter used to balance the classification branch and the regression branch.

[0096] The smooth L1 The function is shown in formula (2):

[0097]

[0098] Among them, σ is the preset hyperparameter, and x is the difference between the actual output of the regression branch of the candidate region generation network and the sample label.

[0099] The regression branch of the candidate region generation network is based on smooth L1 The loss function constructed by the function is expressed as follows:

[0100]

[0101]

[0102]

[0103]

[0104]

[0105] Among them, A x 、A y 、A w and A h They are the x-coordinate, y-coordinate, width and height of the multi-scale candidate box center, G x , G y , G w and G h are the x-coordinate, y-coordinate, width and height of the center of the true frame, respectively; δ[0], δ[1], δ[2] and δ[3] are the regression labels.

[0106] like Figure 3 As shown, it is a schematic diagram of an edge penalty function of an embodiment of a target tracking method based on a fusion twin network and a Kalman filter. In one embodiment of the present invention, Figure 4 As shown, it is an edge penalty effect diagram of an embodiment of the target tracking method based on the fusion of twin network and Kalman filter. Figure 4 (a) is the sampling area map, and its corresponding feature score map is as follows Figure 4 As shown in (b), according to the target center Figure 4 (b) Apply Figure 3 The edge penalty function shown in the figure reduces the score of the candidate box far away from the center of the target. The feature score after edge penalty suppression is shown in the figure below. Figure 4 As shown in (c), the feature with the highest score is selected as the target and the target position is determined. Finally, the candidate box with the highest score after sorting is selected near the target position.

[0107] In step S40, if there is a candidate box with a confidence level greater than a set threshold, the candidate box is output as the prediction result, and the target motion information and position information are updated; otherwise, a cubature Kalman filter with multiple fading factors is used to predict and correct the target motion state, predict the target position, and update the target motion information and position information.

[0108] like Figure 5 FIG. 1 is a flow chart of target position correction using a cubature Kalman filter according to an embodiment of a target tracking method based on a fusion of a Siamese network and a Kalman filter. Step S50 uses a cubature Kalman filter with multiple fading factors to predict and correct the target motion state, including:

[0109] Step S411, predicting the target motion state:

[0110] The volume point calculation of the correction stage of the k-1th frame is performed, where the volume point of the i-th correction stage is as shown in formula (8):

[0111]

[0112] in, is the i-th correction stage volume point of the k-1-th frame, S k-1|k-1 is the weight function of k-1 frame, is the estimated value of the modified state prediction for the k-1 frame, is the volume point set, m is a positive even number representing the number of volume points in the volume point set, [1] i is the i-th volume point;

[0113] The i-th volume point [1] i As shown in formula (9):

[0114]

[0115] Step S412, correct the target motion state:

[0116] The volume point calculation of the correction stage of the kth frame is performed, where the volume point of the i-th correction stage is shown in formula (10):

[0117]

[0118] in, is the i-th correction stage volume point of the k-th frame, u k-1 is the Kalman filter multi-fading factor of the k-1th frame;

[0119] Obtain the corrected state prediction estimate, prediction error covariance matrix, and state prediction value of the kth frame, as shown in Equations (11) to (15):

[0120]

[0121]

[0122]

[0123]

[0124]

[0125] in, is the estimated value of the corrected state prediction of the kth frame; P k|k-1is the prediction error covariance matrix of the kth frame, T is all the moments that need to be accumulated, Q k is the initialization covariance matrix of the kth frame; K k is the Kalman gain of the kth frame, Z k is the measurement estimate of the k-th frame, is the measurement prediction estimate of the k-th frame; represents the product of the correction phase volume point set of the k-th frame and the multi-fading factor of the Kalman filter of the k-th frame;

[0126] Step S413: Update target motion state parameters; the target motion state parameters include a state transfer matrix, a covariance matrix, a state prediction matrix, and a volume point set.

[0127] In step S50 , k is set to k+1, and the process jumps to step S20 to continue target tracking until k is the last frame of the video, thus completing target tracking of the video frame.

[0128] Although the various steps in the above embodiment are described in the above-mentioned order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple changes are within the scope of protection of the present invention.

[0129] A target tracking system based on a fusion twin network and a Kalman filter according to a second embodiment of the present invention includes:

[0130] an initialization module configured to use the target in the k-1th frame of the video frame containing the target as a target template to track the target in the kth frame, and to initialize a Kalman filter based on the target position and size in the k-1th frame;

[0131] The pairing and feature extraction module is configured to generate a search region in the kth frame using a candidate region generation network based on the target in the k-1th frame, and perform tracking feature extraction through the Siamese network using the target template and the search region as an image pair;

[0132] A candidate generation module is configured to generate a multi-scale candidate box, a feature score map, and a regression parameter response map in the search area based on the extracted tracking features by a candidate region generation network;

[0133] an edge penalty module, configured to perform edge penalty based on the target center position in the search area to obtain the confidence of each candidate box;

[0134] The target tracking and information updating module is configured to output the candidate box as the prediction result if there is a candidate box with a confidence level greater than a set threshold, and update the target motion information and position information; otherwise, it uses a cubature Kalman filter with multiple fading factors to predict and correct the target motion state, predict the target position, and update the target motion information and position information;

[0135] The continuous tracking module is configured to set k=k+1, and jumps to the pairing and feature extraction module to continue target tracking until k is the last frame of the video, completing the target tracking of the video frame.

[0136] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the system described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0137] It should be noted that the target tracking system based on the fusion twin network and Kalman filter provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be decomposed or combined. For example, the modules of the above embodiment can be combined into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps and are not regarded as improper limitations on the present invention.

[0138] An electronic device according to a third embodiment of the present invention includes:

[0139] at least one processor; and

[0140] a memory communicatively connected to at least one of the processors; wherein,

[0141] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned target tracking method based on fusion twin network and Kalman filtering.

[0142] A computer-readable storage medium according to a fourth embodiment of the present invention stores computer instructions, which are used to be executed by the computer to implement the above-mentioned target tracking method based on the fusion twin network and Kalman filtering.

[0143] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes and related instructions of the storage device and processing device described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0144] Those skilled in the art should be able to appreciate that, in conjunction with the modules and method steps of each example described in the embodiments disclosed herein, it is possible to implement them with electronic hardware, computer software, or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0145] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.

[0146] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0147] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A target tracking method based on fusion of twin networks and Kalman filtering, characterized in that: The target tracking method comprises: Step S10, using the target in the k-1th frame of the video frame containing the target as the target template to track the target in the kth frame, and initializing the Kalman filter based on the target position and size in the k-1th frame; Step S20: Based on the target in the k-1 frame, a candidate region generation network is used in the k frame to generate a search candidate frame. The target template and the search region are used as an image pair, and the tracking feature is extracted and the search region is generated through the SiamFC twin network. Step S30, generating a multi-scale candidate box, a feature score map, and a regression parameter response map in the search area, and performing an edge penalty based on the target center position in the search area to obtain the confidence of each candidate box; Step S40: If there is a candidate box with a confidence level greater than a set threshold, the candidate box is output as the prediction result, and the target motion information and position information are updated; otherwise, a cubic Kalman filter with multiple fading factors is used to predict and correct the target motion state, predict the target position, and update the target motion information and position information; In step S50 , k is set to k+1, and the process jumps to step S20 to continue target tracking until k is the last frame of the video, thus completing target tracking of the video frame.

2. The target tracking method based on fusion twin network and Kalman filter according to claim 1 is characterized in that: The Kalman filter initialization includes: Initialize the target state initial value to X0, initialize the covariance matrix to Q0, initialize the measurement noise covariance matrix to R0, and initialize the prediction error covariance matrix to P0.

3. The target tracking method based on fusion of twin network and Kalman filter according to claim 1 is characterized in that: Step S20 includes: Step S21, taking the target of the k-1 frame as the center, the target size w k-1 ×h k-1 Crop the target template in the k-1th frame, w k-1 and h k-1 are the width and height of the target in the k-1th frame respectively; Step S22: In the kth frame, the target is centered and the target size is w. k-1 ×h k-1 Crop the search area in the kth frame; Step S23: normalize the target template and the search area to w sm ×h sm The method uses a SiamFC twin network to extract tracking features of the target and the area to be searched; the SiamFC twin network includes two neural networks with the same structure and shared weights.

4. The target tracking method based on fusion twin network and Kalman filter according to claim 1 is characterized in that: The multi-scale candidate box includes s kinds of changes; the feature score map generates a feature for the template of each of the s kinds of changes, and the candidate box corresponding to the feature with a score greater than a set threshold is the target; the regression parameter response map generates a set of regression parameters for the template of each of the s kinds of changes, and the higher the regression parameter score, the higher the scale accuracy of the corresponding candidate box.

5. The target tracking method based on fusion twin network and Kalman filter according to claim 4 is characterized in that: The set of regression parameters includes an x-direction offset and a y-direction offset of the center of the candidate box of the k-th frame relative to the center of the candidate box of the k-1 frame, and a width and a height of the candidate box of the k-th frame.

6. The target tracking method based on fusion twin network and Kalman filter according to claim 2 is characterized in that: In step S40, the target motion state is predicted and corrected using a cubature Kalman filter with multiple fading factors, including: Step S411, predicting the target motion state: Calculate the volume point of the correction stage of the k-1th frame, where the i-th correction stage volume point is: in, is the i-th correction stage volume point in the k-1th frame, s k-1|k-1 is the weight function of k-1 frames, is the estimated value of the modified state prediction for the k-1 frame, is the volume point set, m is a positive even number representing the number of volume points in the volume point set, is the i-th volume point; Step S412, correct the target motion state: Calculate the volume point of the correction stage of the kth frame, where the volume point of the i-th correction stage is: in, is the i-th correction stage volume point of the k-th frame, u k-1 is the Kalman filter multi-fading factor of the k-1th frame; Get the corrected state prediction estimate, prediction error covariance matrix, and state prediction value for the kth frame: in, is the estimated value of the corrected state prediction of the kth frame; P k|k-1 is the prediction error covariance matrix of the kth frame, T is all the moments that need to be accumulated, Q k is the initialization covariance matrix of the kth frame; K k is the Kalman gain of the kth frame, Z k is the measurement estimate of the k-th frame, is the measurement prediction estimate of the k-th frame; represents the product of the correction phase volume point set of the k-th frame and the multi-fading factor of the Kalman filter of the k-th frame; Step S413: Update target motion state parameters; the target motion state parameters include a state transfer matrix, a covariance matrix, a state prediction matrix, and a volume point set.

7. The target tracking method based on fusion of twin network and Kalman filter according to claim 1 is characterized in that: The candidate region generation network is trained as follows: Acquire multiple video frames containing the target as a sample set, and generate a multi-scale candidate box, a feature score map, and a regression parameter response map corresponding to each video frame through the methods corresponding to steps S10 to S30; Set the threshold th pos and th reg , calculate the IoU between the multi-scale candidate box and the corresponding real box, if IoU>th pos , then the candidate box is classified as a positive sample; if IoU>th reg When , the candidate box is classified as a negative sample; Iteratively training the candidate region generation network through positive samples and negative samples; The total loss function of the candidate region generation network is: loss=L cls +λL reg Among them, L cls The cross entropy loss function for the classification branch of the candidate region generation network, L reg Smooth-based regression branch for candidate region generation network L1 The loss function constructed by the function, λ is a preset hyperparameter used to balance the classification branch and the regression branch.

8. The target tracking method based on fusion of twin networks and Kalman filtering according to claim 7 is characterized in that: The smooth L1 The function is: Among them, σ is the preset hyperparameter, and x is the difference between the actual output of the regression branch of the candidate region generation network and the sample label.

9. The target tracking method based on fusion of twin networks and Kalman filtering according to claim 8, characterized in that: The regression branch of the candidate region generation network is based on smooth L1 The loss function constructed by the function is expressed as: Among them, A x 、A y 、A w and A h They are the x-coordinate, y-coordinate, width and height of the multi-scale candidate box center, G x , G y , G w and G h They are the x-coordinate, y-coordinate, width and height of the real frame center respectively.

10. A target tracking system based on fusion twin network and Kalman filter, characterized in that: The target tracking system comprises: an initialization module configured to use the target in the k-1th frame of the video frame containing the target as a target template to track the target in the kth frame, and to initialize a Kalman filter based on the target position and size in the k-1th frame; The pairing and feature extraction module is configured to generate a search candidate box in the kth frame using a candidate region generation network based on the target in the k-1th frame, and use the target template and the search region as an image pair to perform tracking feature extraction and generate a search region through the SiamFC twin network; a candidate box generation and edge penalty module, configured to generate multi-scale candidate boxes, feature score maps, and regression parameter response maps in the search area, and perform edge penalty based on the target center position in the search area to obtain the confidence of each candidate box; The target tracking and information updating module is configured to output the candidate box as the prediction result if there is a candidate box with a confidence level greater than a set threshold, and update the target motion information and position information; otherwise, it uses a cubature Kalman filter with multiple fading factors to predict and correct the target motion state, predict the target position, and update the target motion information and position information; The continuous tracking module is configured to set k=k+1, and jumps to the pairing and feature extraction module to continue target tracking until k is the last frame of the video, completing the target tracking of the video frame.

Images