Target tracking method, device, electronic device and readable storage medium

Abstract

The present application discloses a target tracking method, device, electronic device and readable storage medium, which are applied to the field of computer vision technology and are used to efficiently and accurately realize the tracking and detection of video targets. The method includes inputting the image frame to be processed and the image frame of the target to be tracked into the classification and quality fusion model and the boundary offset perception model; the classification and quality fusion model determines the confidence of the target to be tracked in the image frame to be processed based on the unified classification prediction score and quality estimation score; the boundary offset perception model converts the offset of the boundary from a continuous value to a discrete value, and determines the coarse positioning offset information according to the probability that the offset falls into multiple intervals divided by the discrete values; and calculates the fine positioning offset information of the offset in the target interval through a convolution operation, determines the regression frame according to the coarse positioning offset information and the fine positioning offset information, and determines the position of the target to be tracked in the image frame to be processed by selecting the regression frame corresponding to the position with the highest confidence.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a target tracking method, device, electronic device and readable storage medium. Background Art

[0002] Object tracking means framing a target in the first frame, and then predicting and marking the position of the target in subsequent frames. Object tracking integrates knowledge from multiple fields such as machine learning, optimization, and image processing, and is also the basis for completing further computer vision tasks such as autonomous driving, sports event broadcasting, video surveillance, and vehicle navigation.

[0003] The entire target tracking task process of video images can be decomposed into two subtasks: classification and regression. The classification subtask is to estimate the approximate position of the target, and the regression subtask is to provide an accurate target state and scale. Twin trackers are widely used in tracking tasks due to their good performance in speed and accuracy. Among them, deep learning-based twin trackers, such as SINT (Siamese Instance Search for Trancking) and SiamFC (target tracking algorithm based on fully convolutional twin networks), convert the formula calculation of video target tracking into a cross-correlation operation between the feature map of the tracked target and the feature map of the search area.

[0004] However, a single cross-correlation operation is obviously unable to cope with the situation where the scale of the tracking target changes dramatically, so the general strategy is to manually set different scales and ratios to perform matching, which will inevitably increase the computational burden of the tracker. In order to solve this kind of problem, the relevant technology will introduce the RPN module (Region Proposal Network). For example, SiamRPN avoids the time-consuming and computationally intensive multi-scale feature map extraction and cross-correlation calculation by introducing the RPN module of Faster RCNN, and jointly trains the classification task and regression task, achieving the most advanced results on multiple benchmarks. DaSiamRPN adopts the data augmentation strategy to address the data imbalance problem of non-semantic background and semantic distractor background in the training data. It makes the data in the ImageNet and COCO detection data sets into positive sample pairs to expand the types of training data sets, extracts pictures from different categories and the same category as negative samples, and makes difficult negative samples to improve the network's discrimination ability. SiamRPN++ offsets the target near the center point through uniformly distributed sampling, eliminates position bias, and enables modern networks to be used in tracking. SiamFC++ believes that directly using the confidence of the classification to select the bounding box will lead to the degradation of the tracker performance, so it introduces Estimation Quality Assessment and uses Centess to jointly classify the branch to select the bounding box, thereby improving the accuracy of the tracker. Although the twin tracker based on the RPN module can improve the computational efficiency, because it introduces a large number of anchor boxes set by prior knowledge to improve the accuracy performance, the twin tracker is very sensitive to the size, aspect ratio, number and other hyperparameters of the anchor box. At the same time, the size and aspect ratio of the anchor box are fixed, and the twin tracker is more difficult to handle candidate objects with large deformations, which limits the generalization ability of the twin tracker. In addition, in complex scenes, the essence of the existing box regression is a single Dirac distribution, which is not flexible enough. Although the tracking method of using Estimation Quality Assessment and the classification branch to jointly select the bounding box improves the accuracy of the tracker, in fact, there is inconsistency in the training and reasoning stages of the twin tracker. In the training stage, the classification branch and the quality judgment branch are optimized independently, while the reasoning stage selects the bounding box by the multiplication result, such as Figure 1 As shown, Figure 1 For SiamFC++, the method of using the quality estimation branch alone in the training and inference stages will inevitably lead to certain inconsistencies without end-to-end optimization, and the quality estimation branch is only performed on positive samples in the current tracker, which may cause negative samples to obtain high scores of estimated quality. That is, for existing methods with IoU quality estimation branches, there will be some defects in the case where the IoU score is extremely high but the actual prediction result is the background. Specifically, Figure 2 and Figure 3 As shown, Figure 2 Schematic diagram of target tracking frame with classification score of 0.108 and quality estimation score of 0.903; Figure 3 The following figure shows a target tracking box with a classification score of 0.082 and a quality estimation score of 0.899. A negative sample with a low classification score predicts an unreliable high quality score, causing the twin tracker to select it as the final result instead of a positive sample with a high classification score and a relatively low quality score. The same problem also exists between the classification subtask and the regression subtask. The bounding box corresponding to the position with the highest classification confidence score may not be the most accurate for the target to be tracked. Summary of the invention

[0005] The present application provides a target tracking method, device, electronic device and readable storage medium, which can effectively improve target tracking efficiency and target tracking accuracy at the same time.

[0006] To solve the above technical problems, the embodiments of the present invention provide the following technical solutions:

[0007] An embodiment of the present invention provides a target tracking method, including:

[0008] Pre-training a classification and quality fusion model and a boundary shift perception model; the classification label of the classification and quality fusion model is the IOU score value of the predicted regression box and the real box of the boundary shift perception model;

[0009] Inputting the image frame to be processed and the target image frame to be tracked into the classification and quality fusion model and the boundary offset perception model;

[0010] The classification and quality fusion model determines the confidence of the target to be tracked in the image frame to be processed based on the unified classification prediction score and quality estimation score; the boundary offset perception model converts the boundary offset from a continuous value to a discrete value, and determines the coarse positioning offset information according to the probability that the offset falls into multiple intervals divided by discrete values; and calculates the fine positioning offset information of the offset in the target interval through a convolution operation, and determines the regression frame according to the coarse positioning offset information and the fine positioning offset information;

[0011] According to the confidence and the regression frame, the position of the target to be tracked in the image frame to be processed is determined by selecting the regression frame corresponding to the position with the highest confidence.

[0012] Optionally, the step of inputting the image frame to be processed and the target image frame to be tracked into the classification and quality fusion model and the boundary offset perception model includes:

[0013] Inputting the feature maps of multiple target layers of the feature extraction network of the image frame to be processed and the feature extraction network of the target image frame to be tracked into the classification and quality fusion model respectively;

[0014] The classification and quality fusion model performs a convolution operation on each feature map, and obtains a classification and quality estimation score fusion score map by adding the convolution results pixel by pixel.

[0015] Optionally, determining the confidence of the target to be tracked in the image frame to be processed based on the unified classification prediction score and the quality estimation score includes:

[0016] The first loss function is called to determine the confidence of the target to be tracked in the image frame to be processed based on the unified classification prediction score and the quality estimation score; the first loss function is:

[0017]

[0018] Where S is the total number of samples, θ is the classification and quality estimation score fusion score map, |y-θ| 2 is the dynamic scaling factor, cls is the classification, qly is the quality estimate, L cls-qly represents the first loss function containing classification and quality estimation, y is the confidence of the target, and u is the u-th sample.

[0019] Optionally, determining the coarse positioning offset information according to the probability that the offset falls into a plurality of intervals divided by discrete values ​​includes:

[0020] The coarse positioning relational formula is called to calculate the coarse positioning offset information; the coarse positioning relational formula is:

[0021]

[0022] Where X is the coarse positioning offset information, x i is the i-th label in the discrete label set, n is the total number of intervals, P(x i ) is the probability that the offset falls into the interval corresponding to the i-th label;

[0023] The boundary offset perception model is transferred to the target interval where the current position is closest to the offset of the target edge of the real frame by using the interval cross entropy loss function; the interval cross entropy loss function is:

[0024]

[0025] Where, L interval (S(x i ), P(x i )) is the interval cross entropy loss function, S(x i) is the unique hot label of the offset of the interval, S(x i )∈{0,1}.

[0026] Optionally, determining a regression frame according to the coarse positioning offset information and the fine positioning offset information includes:

[0027] The offset calculation formula is called to calculate the offset of the regression frame; the offset calculation formula is:

[0028]

[0029] In the formula, is the offset of the regression frame, △X is the offset between the target to be tracked and the real frame in the target interval, and DX is the fine positioning offset information.

[0030] Optionally, determining a regression frame according to the coarse positioning offset information and the fine positioning offset information includes:

[0031] The optimization relation is called to optimize the boundary offset perception model as a whole; the optimization relation is:

[0032]

[0033] In the formula, pb is the regression box, gt is the real box of the target to be tracked, pos is the number of positive samples, and L reg (pb, gt) is the IOU loss function.

[0034] Optionally, after calling the optimization relation to perform overall optimization on the boundary offset perception model, the method further includes:

[0035] The reoptimization loss function is called to update the IOU loss function of the boundary offset perception model, and the reoptimization loss function is:

[0036] L reg-fin =θL reg (pb, gt)

[0037] Where, L reg-fin is the reoptimization loss function, θ is the fusion score map of the classification and quality estimation scores.

[0038] Another aspect of the present invention provides a target tracking device, including:

[0039] A model training module, used for pre-training a classification and quality fusion model and a boundary shift perception model; the classification label of the classification and quality fusion model is an IOU score value of a predicted regression box and a true box of the boundary shift perception model;

[0040] An image input module is used to input the image frame to be processed and the image frame of the target to be tracked into the classification and quality fusion model and the boundary offset perception model; the classification and quality fusion model determines the confidence of the target to be tracked in the image frame to be processed based on the unified classification prediction score and quality estimation score; the boundary offset perception model converts the offset of the boundary from a continuous value to a discrete value, and determines the coarse positioning offset information according to the probability that the offset falls into multiple intervals divided by the discrete values; and calculates the fine positioning offset information of the offset in the target interval through a convolution operation, and determines the regression frame according to the coarse positioning offset information and the fine positioning offset information;

[0041] The tracking processing module is used to determine the position of the target to be tracked in the image frame to be processed by selecting the regression frame corresponding to the position with the highest confidence according to the confidence and the regression frame.

[0042] An embodiment of the present invention further provides an electronic device, comprising a processor, wherein the processor is configured to implement the steps of the target tracking method as described in any of the preceding items when executing a computer program stored in a memory.

[0043] Finally, an embodiment of the present invention further provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the target tracking method as described in any of the preceding items are implemented.

[0044] The advantage of the technical solution provided by the present application is that the classification and quality fusion model jointly represents the classification prediction score and the quality estimation score as a unified whole, and the fusion score of the classification prediction score and the quality estimation score is used as the condition for the final classification judgment, so that it can be optimized end-to-end, eliminating the inconsistency between the training stage and the reasoning stage, thereby avoiding the need to introduce a large amount of prior knowledge and hyperparameter settings in the related technology RPN module, enhancing the communication between the classification branch and the regression branch, and being able to effectively cope with various challenges such as illumination changes, deformation, scale changes, and interference, and achieve high-precision and high-robustness target tracking. Further, the IOU score of the predicted frame and the real frame of the boundary offset perception model is used as the label of the classification branch, and a communication bridge is established between the classification and regression subtasks, which can eliminate the inconsistency between the two subtasks. In addition, the boundary offset perception model can obtain a more accurate prediction frame through coarse positioning + fine positioning, and at the same time avoids the introduction of prior knowledge and a large number of hyperparameter settings, which can effectively improve the target tracking efficiency and target tracking accuracy at the same time, and finally achieve efficient and accurate video target tracking detection.

[0045] In addition, the embodiments of the present invention also provide corresponding implementation devices, electronic devices and readable storage media for the target tracking method, which further make the method more practical, and the devices, electronic devices and readable storage media have corresponding advantages.

[0046] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A schematic diagram of a method for using a quality estimation branch separately in the training and reasoning stages of SiamFC++ in the related art provided by an embodiment of the present invention;

[0049] Figure 2 Schematic diagram of target tracking block with a classification score of 0.108 and a quality estimation score of 0.903 provided for an embodiment of the present invention;

[0050] Figure 3 Schematic diagram of target tracking block with a classification score of 0.082 and a quality estimation score of 0.899 provided for an embodiment of the present invention;

[0051] Figure 4 A flowchart of a target tracking method provided by an embodiment of the present invention;

[0052] Figure 5 A schematic diagram of a classification-quality estimation joint representation of a classification and quality fusion model provided in an embodiment of the present invention;

[0053] Figure 6 A schematic diagram of the structure principle of a boundary offset perception model provided by an embodiment of the present invention;

[0054] Figure 7 A schematic diagram of the structural framework of a target tracking method provided by an embodiment of the present invention;

[0055] Figure 8 A schematic diagram of the structural framework of the classification and quality fusion model and the boundary offset perception model provided in an embodiment of the present invention;

[0056] Fig. 9 A structural diagram of a specific implementation of a target tracking device provided by an embodiment of the present invention;

[0057] Fig.10 A structural diagram of a specific implementation of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0059] The terms "first", "second", "third", "fourth", etc. in the specification and claims of this application and the above drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0060] After introducing the technical solutions of the embodiments of the present invention, various non-limiting implementation methods of the present application are described in detail below.

[0061] See first Figure 4 , Figure 4 A flow chart of a target tracking method provided by an embodiment of the present invention, the embodiment of the present invention may include the following contents:

[0062] S401: Pre-training classification and quality fusion model and boundary offset perception model.

[0063] In this embodiment, the classification and quality fusion model can jointly represent the classification prediction score and the quality estimation score as a unified whole, and can determine the approximate position of the target to be tracked in each frame of the video and the corresponding confidence. For the supervision label y∈[0,1] of the classification and quality fusion model, the classification label can be the IOU (confidence) score value of the predicted regression box of the target to be tracked output by the boundary offset perception model and the real box of the target to be tracked. For the training sample of the classification and quality fusion model, the label of the positive sample can be the IOU score value of the predicted regression box of the target to be tracked output by the boundary offset perception model and the real box of the target to be tracked, and the score of the negative sample is set to 0, so that the positive and negative samples are consistent and reliable. The boundary offset perception model is used to process the input image and determine the regression box of the target to be tracked in each frame of the video.

[0064] S402: Inputting the image frame to be processed and the target image frame to be tracked into the classification and quality fusion model and the boundary offset perception model.

[0065] In this embodiment, the image frame to be processed is an image frame for target tracking, and the target image frame to be tracked is the search image frame, that is, the target that needs to be searched from the image frame to be processed. The classification and quality fusion model determines the confidence of the target to be tracked in the image frame to be processed based on the unified classification prediction score and the quality estimation score. The boundary offset perception model converts the offset of the boundary from a continuous value to a discrete value, divides these discrete values ​​into multiple intervals, and determines the coarse positioning offset information according to the probability that the offset falls into these multiple intervals divided by the discrete values. The fine positioning offset information of the offset in the target interval is calculated by a convolution operation, and the regression frame is determined based on the coarse positioning offset information and the fine positioning offset information. Figure 5-Figure 8 As shown, Figure 6 In the figure, L, T, R, and D represent the coarse positioning of the left, upper, right, and lower boundary offsets of the regression box, respectively. DL, DT, DR, and DD represent the offsets of each bounding box within a certain interval, and the offsets of the final regression box are obtained by adding them together.

[0066] S403: According to the confidence and the regression frame, the position of the target to be tracked in the image frame to be processed is determined by selecting the regression frame corresponding to the position with the highest confidence.

[0067] After determining the confidence of the approximate position of the target to be tracked in the image frame to be processed based on the classification and quality fusion model, for the regression box predicted by the boundary offset perception model, the position of the target to be tracked in the image frame to be processed is determined by selecting the regression box corresponding to the position with the highest confidence, thereby avoiding a negative sample with a low classification score, predicting an unreliable high quality score, and causing the erroneous result to be used as the final target tracking result.

[0068] In the technical solution provided by the embodiment of the present invention, the classification and quality fusion model jointly represents the classification prediction score and the quality estimation score as a unified whole, and the fusion score of the classification prediction score and the quality estimation score is used as the condition for the final classification judgment, so that it can be optimized end-to-end, eliminating the inconsistency between the training stage and the reasoning stage, thereby avoiding the need to introduce a large amount of prior knowledge and hyperparameter settings in the related technology RPN module, enhancing the communication between the classification branch and the regression branch, and being able to effectively cope with various challenges such as illumination changes, deformation, scale changes, and interference effects, and achieve high-precision and high-robustness target tracking. Further, the IOU score of the predicted frame and the real frame of the boundary offset perception model is used as the label of the classification branch, and a communication bridge is established between the classification and regression subtasks, which can eliminate the inconsistency between the two subtasks. In addition, the boundary offset perception model can obtain a more accurate prediction frame through the method of coarse positioning + fine positioning, and at the same time avoids the introduction of prior knowledge and the setting of a large number of hyperparameters, which can effectively improve the target tracking efficiency and target tracking accuracy at the same time, and finally achieve efficient and accurate video target tracking detection.

[0069] It should be noted that there is no strict order of execution between the steps in this application. As long as they comply with the logical order, these steps can be executed simultaneously or in a certain preset order. Figure 4 This is just a schematic and does not mean that this is the only execution order.

[0070] In the above embodiment, there is no limitation on how to execute step S402. In this embodiment, an image processing flow of a classification and quality fusion model and a boundary offset perception model is provided, which may include the following contents:

[0071] Obtain the image frame to be processed and the image frame of the target to be tracked, and extract the feature maps of multiple target layers of the feature extraction network of the image frame to be processed and the target to be tracked through any deep learning network, for example, the feature maps of the third layer, the fourth layer and the fifth layer of the feature extraction network of the image frame to be processed and the target to be tracked can be extracted. Then, the feature maps of multiple target layers of the feature extraction network of the image frame to be processed and the feature extraction network of the target image frame to be tracked can be respectively input into the classification and quality fusion model; the classification and quality fusion model can perform convolution operations on each feature map according to the required convolution kernel size, such as 1*1, and add the convolution results pixel by pixel, such as sigmoid processing, to finally obtain the classification and quality estimation score fusion score map. Based on the unified classification prediction score and quality estimation score, that is, the classification and quality estimation score fusion score map, the classification and quality fusion model determines the confidence of the target to be tracked in the image frame to be processed by calling the first loss function. The first loss function can be expressed as:

[0072]

[0073] Where S is the total number of samples, θ is the fusion score map of classification and quality estimation scores, |y-θ| 2 is a dynamic scaling factor that can reduce the weight of simple samples during the training phase and allow the model to quickly focus on difficult samples. cls is classification, qly is quality estimation, and L cls-qly represents the first loss function containing classification and quality estimation, y is the confidence of the target, and u is the u-th sample.

[0074] The process of determining the regression frame for the boundary offset perception model may include: dividing the continuous offsets of the four sides of the predicted regression frame into discrete offsets, setting the candidate region of the offset, for example, the candidate region of the offset [0, 120] may be divided into a plurality of, for example, 15 independent intervals, each of which has a width of 8. Of course, [120, +∞] is also regarded as 1 interval, and the total number of intervals n = 16, and the corresponding discrete label set can be set to L∈{x1, x2, ..., x m , x m+1 , …, x n , x n+1}, x m In the discrete label set, the label of the mth sample is represented. The discrete label is the interval label, and t is the offset between the current position and a certain edge of the real box. m , x m+1 If w m =|tx m | is the offset t and the interval x m The distance, then when w m <w m+1 When the discrete label is x m , otherwise x m+1 . P(x i ) indicates that the offset falls on label x i The probability of the corresponding interval and satisfies By using discrete integration, the coarse offset can be calculated, and the coarse positioning relationship can be called to calculate the coarse positioning offset information; the coarse positioning relationship can be expressed as:

[0075]

[0076] Where X is the coarse positioning offset information, x i is the i-th label in the discrete label set, n is the total number of intervals, P(x i ) is the probability that the offset falls into the interval corresponding to the i-th label.

[0077] In order to allow the model to shift its attention to the interval closest to the offset more quickly, the interval cross entropy loss function can be used for optimization, so that the boundary offset perception model can be transferred to the target interval closest to the offset between the current position and the target edge of the real box. The interval cross entropy loss function can be expressed as:

[0078]

[0079] Where, L interval (S(x i ), P(x i )) is the interval cross entropy loss function, S(x i ) is the unique hot label of the offset of the interval, S(x i )∈{0,1}. When the offset is in the interval x i 1 if the condition is true, otherwise 0.

[0080] After determining the coarse positioning offset information and the fine positioning offset information, the offset calculation formula can be called to calculate the offset of the regression frame; the offset calculation formula can be expressed as:

[0081]

[0082] In the formula, is the offset of the regression box, △X is the offset between the target to be tracked and the real box in the target interval, and DX is the fine positioning offset information.

[0083] In order to ensure that the final regression box corresponds to the highest confidence score, the optimization relationship can also be called to optimize the boundary offset perception model as a whole; the optimization relationship can be:

[0084]

[0085] In the formula, pb is the regression box, i.e. the predicted bounding box, gt is the real box of the target to be tracked, i.e. the ground-truth box, pos is the number of positive samples, and L is reg (pb, gt) is the IOU loss function.

[0086] In order to build a bridge of communication between the classification and quality fusion model and the boundary shift perception model and eliminate the inconsistency between the classification and quality fusion model and the boundary shift perception model, the reoptimization loss function can also be called to update the IOU loss function of the boundary shift perception model. The reoptimization loss function is:

[0087] L reg-fin =θL reg (pb, gt)

[0088] Where, L reg-finTo re-optimize the loss function, that is, the final regression loss, is the final target to be optimized, and θ is the fusion score map of classification and quality estimation scores.

[0089] In order to make the technical personnel in the relevant field more clearly understand the technical solution of the present application, the present application is also combined with Figure 7 and Figure 8 , giving an illustrative example of the present application, may include the following contents:

[0090] A1: Set the supervision label of the classification and quality fusion model, that is, the classification branch. Use the IOU score between the bounding box predicted by the boundary offset perception model and the real box of the target as the label of the positive sample, and set the score of the negative sample to 0.

[0091] A2: The feature maps R3(A, B), R4(A, B), and R5(A, B) of the tracking video template frame and the search frame are input into the classification and quality fusion model respectively, and a 1x1 convolution kernel is used for convolution operation, and then a sigmoid process is performed, and finally the classification and quality estimation score fusion score map C = (25×25) is obtained. 1 , R3, R4, and R5 represent the feature maps output by the 3rd, 4th, and 5th layers of the feature extraction network, respectively. A and B represent the search frame and template frame, respectively. 25 is the size of the feature map, and 1 is the number of feature channels.

[0092] A3: Use loss function:

[0093]

[0094] A4: The boundary offset perception model is composed of coarse positioning and fine positioning of the regression branch. For coarse positioning, the coarse positioning X is expressed as:

[0095]

[0096] A5: Use interval cross entropy loss function:

[0097]

[0098] A6: After performing A4 and A5 operations on the extracted image features, the coarsely positioned offset perception map R is obtained. P =(25×25) 4 , 4 indicates the coarse positioning of the four edges of the prediction box.

[0099] A7: After performing a 1x1 convolution operation on the extracted image features, we get the refined offset DX of the four edges in the interval, which is the offset between the target and the real border in the positioning interval. The predicted offset finally output by the boundary offset perception model

[0100]

[0101] A8: Use the IOU loss function to optimize the boundary offset perception model as a whole:

[0102]

[0103] A9: Multiply the prediction score output by the classification and quality fusion model with the IOU loss function of the boundary shift perception model and reweight it to alleviate the inconsistency between the classification and regression branches.

[0104] A10: In order to verify the effectiveness, the entire technical solution was verified on multiple authoritative datasets such as VOT2016, OTB100, UAV123, and compared with the performance of related methods on the above datasets. The experimental results prove that the technical solution provided in this application can effectively improve the accuracy and efficiency of target tracking.

[0105] The embodiment of the present invention also provides a corresponding device for the target tracking method, which further makes the method more practical. The device can be described from the perspective of functional modules and hardware. The target tracking device provided by the embodiment of the present invention is introduced below. The target tracking device described below and the target tracking method described above can be referred to each other.

[0106] From the perspective of functional modules, see Fig. 9 , Fig. 9 A structural diagram of a target tracking device provided by an embodiment of the present invention in a specific implementation manner, the device may include:

[0107] The model training module 901 is used to pre-train the classification and quality fusion model and the boundary shift perception model; the classification label of the classification and quality fusion model is the IOU score value of the predicted regression box and the real box of the boundary shift perception model;

[0108] The image input module 902 is used to input the image frame to be processed and the image frame of the target to be tracked into the classification and quality fusion model and the boundary offset perception model; the classification and quality fusion model determines the confidence of the target to be tracked in the image frame to be processed based on the unified classification prediction score and the quality estimation score; the boundary offset perception model converts the offset of the boundary from a continuous value to a discrete value, and determines the coarse positioning offset information according to the probability that the offset falls into multiple intervals divided by the discrete values; and calculates the fine positioning offset information of the offset in the target interval through a convolution operation, and determines the regression frame according to the coarse positioning offset information and the fine positioning offset information;

[0109] The tracking processing module 903 is used to determine the position of the target to be tracked in the image frame to be processed by selecting the regression frame corresponding to the position with the highest confidence according to the confidence and the regression frame.

[0110] Optionally, in some implementations of the present embodiment, the image input module 902 may be used to: input feature maps of multiple target layers of a feature extraction network of an image frame to be processed and a feature extraction network of a target image frame to be tracked into a classification and quality fusion model respectively; the classification and quality fusion model performs a convolution operation on each feature map, and obtains a classification and quality estimation score fusion score map by adding the convolution results pixel by pixel.

[0111] As an optional implementation, the image input module 902 may be further used to: call a first loss function to determine the confidence of the target to be tracked in the image frame to be processed based on the unified classification prediction score and quality estimation score; the first loss function is:

[0112]

[0113] Where S is the total number of samples, θ is the fusion score map of classification and quality estimation scores, |y-θ| 2 is the dynamic scaling factor, cls is the classification, qly is the quality estimate, L cls-qly represents the first loss function containing classification and quality estimation, y is the confidence of the target, and u is the u-th sample.

[0114] Optionally, in some implementations of this embodiment, the image input module 902 may be further used to: call a coarse positioning relationship to calculate coarse positioning offset information; the coarse positioning relationship is:

[0115]

[0116] Where X is the coarse positioning offset information, x i is the i-th label in the discrete label set, n is the total number of intervals, P(x i ) is the probability that the offset falls into the interval corresponding to the i-th label;

[0117] The interval cross entropy loss function is used to transfer the boundary offset perception model to the target interval where the current position is closest to the offset of the target edge of the real frame; the interval cross entropy loss function is:

[0118]

[0119] Where, L interval (S(x i ), P(x i )) is the interval cross entropy loss function, S(x i ) is the unique hot label of the offset of the interval, S(x i )∈{0,1}.

[0120] As an optional implementation of the above embodiment, the above image input module 902 can also be used to: call the offset calculation relationship to calculate the offset of the regression frame; the offset calculation relationship is:

[0121]

[0122] In the formula, is the offset of the regression box, △X is the offset between the target to be tracked and the real box in the target interval, and DX is the fine positioning offset information.

[0123] As another optional implementation of the above embodiment, the above image input module 902 may be further used to: call the optimization relationship to optimize the boundary offset perception model as a whole; the optimization relationship is:

[0124]

[0125] In the formula, pb is the regression box, gt is the real box of the target to be tracked, pos is the number of positive samples, and L reg (pb, gt) is the IOU loss function.

[0126] As another optional implementation of the above embodiment, the above image input module 902 may be further used to: call a reoptimization loss function to update the IOU loss function of the boundary offset perception model, and the reoptimization loss function is:

[0127] L reg-fin =θL reg (pb, gt)

[0128] Where, L reg-fin is the reoptimization loss function, and θ is the fusion score map of classification and quality estimation scores.

[0129] The functions of the functional modules of the target tracking device described in the embodiment of the present invention can be specifically implemented according to the method in the above method embodiment. The specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here.

[0130] It can be seen from the above that the embodiments of the present invention can effectively improve both the target tracking efficiency and the target tracking accuracy.

[0131] The target tracking device mentioned above is described from the perspective of functional modules. Furthermore, the present application also provides an electronic device, which is described from the perspective of hardware. Fig.10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application in one implementation manner. Fig.10As shown, the electronic device includes a memory 100 for storing a computer program; and a processor 101 for implementing the steps of the target tracking method mentioned in any of the above embodiments when executing the computer program.

[0132] Among them, the processor 101 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the processor 101 may also be a controller, a microcontroller, a microprocessor or other data processing chip. The processor 101 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 101 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 101 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 101 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0133] The memory 100 may include one or more computer-readable storage media, which may be non-transitory. The memory 100 may also include a high-speed random access memory and a non-volatile memory, such as one or more disk storage devices and flash storage devices. The memory 100 may be an internal storage unit of an electronic device in some embodiments, such as a hard disk of a server. The memory 100 may also be an external storage device of an electronic device in other embodiments, such as a plug-in hard disk equipped on a server, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 100 may also include both an internal storage unit of an electronic device and an external storage device. The memory 100 may not only be used to store application software and various types of data installed in the electronic device, such as: the code of the program for executing the vulnerability processing method, etc., but may also be used to temporarily store data that has been output or is to be output. In this embodiment, the memory 100 is at least used to store the following computer program 1001, wherein, after the computer program is loaded and executed by the processor 101, the relevant steps of the target tracking method disclosed in any of the aforementioned embodiments can be implemented. In addition, the resources stored in the memory 100 may also include an operating system 1002 and data 1003, etc., and the storage method may be temporary storage or permanent storage. The operating system 1002 may include Windows, Unix, Linux, etc. The data 1003 may include but is not limited to data corresponding to the target tracking result, etc.

[0134] In some embodiments, the electronic device may further include a display screen 102, an input / output interface 103, a communication interface 104 or a network interface, a power supply 105 and a communication bus 106. Among them, the display screen 102 and the input / output interface 103, such as a keyboard, belong to the user interface, and the optional user interface may also include a standard wired interface, a wireless interface, etc. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface. The communication interface 104 may optionally include a wired interface and / or a wireless interface, such as a WI-FI interface, a Bluetooth interface, etc., which are generally used to establish a communication connection between the electronic device and other electronic devices. The communication bus 106 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0135] Those skilled in the art will understand that Fig.10 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, for example, it may also include a sensor 107 to implement various functions.

[0136] The functions of the functional modules of the electronic device described in the embodiment of the present invention can be specifically implemented according to the method in the above method embodiment. The specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here.

[0137] It can be seen from the above that the embodiments of the present invention can effectively improve both the target tracking efficiency and the target tracking accuracy.

[0138] It is understandable that if the target tracking method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electrically erasable programmable ROM, register, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, removable disk, CD-ROM, disk or optical disk and other media that can store program code.

[0139] Based on this, an embodiment of the present invention further provides a readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the target tracking method described in any of the above embodiments are performed.

[0140] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the hardware disclosed in the embodiments, including devices and electronic devices, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0141] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0142] The above is a detailed introduction to a target tracking method, device, electronic device and readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A target tracking method, characterized in that: include: Pre-training a classification and quality fusion model and a boundary shift perception model; the classification label of the classification and quality fusion model is the IOU score value of the predicted regression box and the real box of the boundary shift perception model; Inputting the image frame to be processed and the target image frame to be tracked into the classification and quality fusion model and the boundary offset perception model; The classification and quality fusion model determines the confidence of the target to be tracked in the image frame to be processed based on the unified classification prediction score and quality estimation score; the boundary offset perception model converts the boundary offset from a continuous value to a discrete value, and determines the coarse positioning offset information according to the probability that the offset falls into multiple intervals divided by discrete values; and calculates the fine positioning offset information of the offset in the target interval through a convolution operation, and determines the regression frame according to the coarse positioning offset information and the fine positioning offset information; According to the confidence and the regression frame, the position of the target to be tracked in the image frame to be processed is determined by selecting the regression frame corresponding to the position with the highest confidence.

2. The target tracking method according to claim 1, characterized in that: The step of inputting the image frame to be processed and the target image frame to be tracked into the classification and quality fusion model and the boundary offset perception model comprises: Inputting the feature maps of multiple target layers of the feature extraction network of the image frame to be processed and the feature extraction network of the target image frame to be tracked into the classification and quality fusion model respectively; The classification and quality fusion model performs a convolution operation on each feature map, and obtains a classification and quality estimation score fusion score map by adding the convolution results pixel by pixel.

3. The target tracking method according to claim 2, characterized in that: The step of determining the confidence of the target to be tracked in the image frame to be processed based on the unified classification prediction score and the quality estimation score includes: The first loss function is called to determine the confidence of the target to be tracked in the image frame to be processed based on the unified classification prediction score and the quality estimation score; the first loss function is: Where S is the total number of samples, θ is the classification and quality estimation score fusion score map, |y-θ| 2 is the dynamic scaling factor, cls is the classification, qly is the quality estimate, L cls-qly represents the first loss function containing classification and quality estimation, y is the confidence of the target, and u is the u-th sample.

4. The target tracking method according to any one of claims 1 to 3, characterized in that: The determining of the coarse positioning offset information according to the probability that the offset falls into a plurality of intervals divided by discrete values ​​comprises: The coarse positioning relational formula is called to calculate the coarse positioning offset information; the coarse positioning relational formula is: Where X is the coarse positioning offset information, x i is the i-th label in the discrete label set, n is the total number of intervals, P(x i ) is the probability that the offset falls into the interval corresponding to the i-th label; The boundary offset perception model is transferred to the target interval where the current position is closest to the offset of the target edge of the real frame by using the interval cross entropy loss function; the interval cross entropy loss function is: Where, L interval (S(x i ), P(x i )) is the interval cross entropy loss function, S(x i ) is the unique hot label of the offset of the interval, S(x i )∈{0,1}.

5. The target tracking method according to claim 4, characterized in that: The determining of the regression frame according to the coarse positioning offset information and the fine positioning offset information includes: The offset calculation formula is called to calculate the offset of the regression frame; the offset calculation formula is: In the formula, is the offset of the regression frame, △X is the offset between the target to be tracked and the real frame in the target interval, and DX is the fine positioning offset information.

6. The target tracking method according to claim 5, characterized in that: The determining of the regression frame according to the coarse positioning offset information and the fine positioning offset information includes: The optimization relation is called to optimize the boundary offset perception model as a whole; the optimization relation is: In the formula, pb is the regression box, gt is the real box of the target to be tracked, pos is the number of positive samples, and L reg (pb, gt) is the IOU loss function.

7. The target tracking method according to claim 6, characterized in that: After the optimization relation is called to optimize the boundary offset perception model as a whole, the method further includes: The reoptimization loss function is called to update the IOU loss function of the boundary offset perception model, and the reoptimization loss function is: L reg-fin =θL reg (pb,gt) Where, L reg-fin is the reoptimization loss function, θ is the fusion score map of the classification and quality estimation scores.

8. A target tracking device, characterized in that: include: A model training module, used for pre-training a classification and quality fusion model and a boundary shift perception model; the classification label of the classification and quality fusion model is an IOU score value of a predicted regression box and a true box of the boundary shift perception model; An image input module is used to input the image frame to be processed and the image frame of the target to be tracked into the classification and quality fusion model and the boundary offset perception model; the classification and quality fusion model determines the confidence of the target to be tracked in the image frame to be processed based on the unified classification prediction score and quality estimation score; the boundary offset perception model converts the offset of the boundary from a continuous value to a discrete value, and determines the coarse positioning offset information according to the probability that the offset falls into multiple intervals divided by the discrete values; and calculates the fine positioning offset information of the offset in the target interval through a convolution operation, and determines the regression frame according to the coarse positioning offset information and the fine positioning offset information; The tracking processing module is used to determine the position of the target to be tracked in the image frame to be processed by selecting the regression frame corresponding to the position with the highest confidence according to the confidence and the regression frame.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor is used to implement the steps of the target tracking method according to any one of claims 1 to 7 when executing a computer program stored in the memory.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the target tracking method according to any one of claims 1 to 7 are implemented.

Images