Target tracking method, device, equipment and storage medium

Abstract

The present application relates to the field of computer, provide a kind of target tracking method, device, equipment and storage medium, the method comprises: based on target tracking module obtains the tracking result of current frame image and last frame image;If the similarity degree of current frame tracking result and last frame tracking result is less than preset degree, then whether the next frame image contains target to be tracked by target identification module is identified;If it is identified that the next frame image contains target to be tracked, then output first target pixel bounding box of target to be tracked in the next frame image, and first target pixel bounding box is classified based on target classification module, and the re-tracking target is determined.This embodiment of the present application provides target tracking method to realize the automatic closed loop of target identification to target tracking, improves target tracking robustness.

Description

Technical Field

[0001] This invention relates to the field of computers, and more particularly to a target tracking method, apparatus, device, and storage medium. Background Technology

[0002] Target tracking is a crucial research area in computer vision, and the automatic tracking of moving ground targets by UAVs plays a vital role and has strategic requirements in fields such as reconnaissance and intelligence analysis. However, factors such as the speed, attitude, and position of the UAV can lead to issues like image jitter and unstable video streams; complex environments can cause targets to be easily obscured by the background; and some targets move quickly and are highly maneuverable. All of these factors increase the difficulty of automatic target tracking on UAVs, resulting in low robustness in target tracking. Summary of the Invention

[0003] This invention provides a target tracking method, apparatus, device, and storage medium, aiming to achieve automatic closed-loop from target identification to target tracking and improve the robustness of target tracking.

[0004] In a first aspect, the present invention provides a target tracking method, comprising:

[0005] The tracking results of the current frame image and the previous frame image are obtained based on the target tracking module;

[0006] If the similarity between the current frame tracking result and the previous frame tracking result is less than a preset level, the target recognition module will identify whether the next frame image contains the target to be tracked.

[0007] If the target to be tracked is detected in the next frame, the first target pixel bounding box of the target to be tracked in the next frame is output, and the first target pixel bounding box is classified based on the target classification module to determine the target to be tracked again.

[0008] In one embodiment, classifying the first target pixel bounding box based on the target classification module to determine the target to be re-tracked includes:

[0009] The target classification module classifies the bounding box of the first target pixel to obtain multiple samples to be processed.

[0010] The multiple samples to be processed are classified according to a confidence threshold to obtain a positive sample set, wherein the confidence of the samples to be processed in the positive sample set is greater than the confidence threshold.

[0011] The target to be re-tracked is determined based on the positive sample set.

[0012] Determining the re-tracking target based on the positive sample set includes:

[0013] The samples to be processed in the positive sample set are sorted in descending order of confidence level to obtain the sorted samples to be processed.

[0014] The sample with the highest confidence among the sorted samples to be processed is determined as the target sample;

[0015] The pixel bounding box of the target sample is determined as the second target pixel bounding box, and the re-tracking target is determined based on the second target pixel bounding box.

[0016] After classifying the bounding box of the first target pixel based on the target classification module and determining the target to be tracked again, the method further includes:

[0017] Collect a set of positive and negative samples from the current frame image of the re-tracked target;

[0018] The parameters in the target classification module are updated and trained based on the positive and negative sample sets to obtain the target classification module with updated parameters.

[0019] The positive and negative sample set includes positive samples and negative samples. The positive samples are samples containing the target image region of the re-tracked target, and the negative samples are samples from other image regions besides the target image region.

[0020] After obtaining the tracking results of the current frame image and the previous frame image based on the target tracking module, the method further includes:

[0021] If the similarity between the current frame tracking result and the previous frame tracking result is greater than or equal to the preset level, then the positive and negative sample sets of each frame image of the target to be tracked are collected.

[0022] The parameters in the initial classification module are updated and trained based on the positive and negative sample sets to obtain the target classification module.

[0023] The positive and negative sample set includes positive samples and negative samples. The positive samples are samples containing the target image region of the target to be tracked, and the negative samples are samples from other image regions besides the target image region.

[0024] Before updating and training the parameters in the initial classification module based on the positive and negative sample sets to obtain the target classification module, the method further includes:

[0025] The classifier is pre-trained and initialized to obtain the initial classification module, wherein the classifier is a backbone network composed of three fully connected layers.

[0026] The pixel bounding box in the current frame recognition result is obtained based on the target recognition module, and the specific steps include:

[0027] The target recognition module performs target recognition on the current frame image to determine whether the current frame image contains the target to be tracked.

[0028] If the target to be tracked is contained in the current frame image, then the pixel coordinate center and bounding box size of the target to be tracked in the current frame image are output;

[0029] The pixel coordinate center and the bounding box size are determined as the pixel bounding box of the target to be tracked in the current frame image.

[0030] In a second aspect, the present invention provides a target tracking device comprising:

[0031] The tracking unit is used to obtain the tracking results of the current frame image and the previous frame image based on the target tracking module;

[0032] The recognition unit is used to identify whether the target to be tracked is contained in the next frame image if the similarity between the current frame tracking result and the previous frame tracking result is less than a preset level.

[0033] The classification unit is used to output the first target pixel bounding box of the target in the next frame image if the target to be tracked is detected, and classify the first target pixel bounding box based on the target classification module to determine the target to be tracked again.

[0034] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target tracking method described in the first aspect.

[0035] Fourthly, the present invention also provides a non-transitory computer-readable storage medium comprising a computer program that, when executed by the processor, implements the target tracking method described in the first aspect.

[0036] Fifthly, the present invention also provides a computer program product comprising a computer program that, when executed by the processor, implements the target tracking method described in the first aspect.

[0037] The target tracking method, apparatus, device, and storage medium provided by this invention obtain the tracking results of the current frame image and the previous frame image based on the target tracking module; if the similarity between the current frame tracking result and the previous frame tracking result is less than a preset level, the target recognition module identifies whether the next frame image contains the target to be tracked; if the next frame image contains the target to be tracked, the first target pixel bounding box of the target to be tracked in the next frame image is output, and the first target pixel bounding box is classified based on the target classification module to determine the target to be tracked again.

[0038] During target tracking, the target tracking module combines the tracking results from the previous frame and the current frame to track the target. If tracking fails based on the current frame's tracking result, the target recognition module re-identifies the target in the next frame. Then, based on the target classification module and the recognition results from the next frame, the target is classified to determine the target to be tracked again. This achieves an automatic closed loop from target recognition to target tracking, improving the robustness of target tracking. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of the target tracking method provided by the present invention;

[0041] Figure 2 This is a structural diagram of the target tracking device provided by the present invention;

[0042] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0044] It should be noted that in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly, for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two elements. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0045] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0046] Furthermore, combined Figures 1 to 3 This invention describes the target tracking method, apparatus, device, and storage medium provided by the present invention. Figure 1 This is a flowchart of the target tracking method provided by the present invention; Figure 2 This is a structural diagram of the target tracking device provided by the present invention; Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0047] This invention provides an embodiment of a target tracking method. It should be noted that although the logical order is shown in the flowchart, under certain data conditions, the steps shown or described may be performed in a different order than that shown here.

[0048] This invention uses an electronic device as the execution subject for example. The electronic device in this invention includes, but is not limited to, a host computer, a ground station, and an airborne mission computer. This invention uses a target tracking framework as an example of an electronic device for illustration.

[0049] Reference Figure 1 , Figure 1 This is a flowchart of the target tracking method provided by the present invention. The target tracking method provided by the embodiments of the present invention includes:

[0050] S101, based on the target tracking module, obtain the tracking results of the current frame image and the previous frame image.

[0051] It should be noted that one of the application scenarios of the target tracking method provided in this embodiment of the invention is automatic target tracking on a drone.

[0052] Furthermore, the target tracking framework includes, but is not limited to, a target recognition module, a target tracking module, and a target classification module.

[0053] It should be further explained that the main functions of each module are as follows: the main function of the target recognition module is to identify whether the image contains the target to be tracked; the main function of the target tracking module is to track the target to be tracked; and the main function of the target classification module is to classify the target to be tracked.

[0054] Specifically, the target recognition module of the target tracking framework performs target recognition on the acquired current frame image according to the set target information to determine whether the current frame image contains a target to be tracked. The target information includes, but is not limited to, target type information, target size information, and target position information.

[0055] If it is determined that the current frame image contains a target to be tracked, that is, the target recognition module has identified the target of interest, the target tracking framework will output the pixel bounding box of the target to be tracked in the current frame image. The pixel bounding box includes the pixel coordinate center and the size of the bounding box.

[0056] Furthermore, the target classification module obtains the pixel bounding box of the target to be tracked in the current frame image and immediately initializes the target classification module.

[0057] Furthermore, the target tracking module also acquires the pixel bounding box of the target in the current frame image and immediately initializes the target tracking module. Further, the target tracking module obtains the tracking results from the current frame image and the previous frame image.

[0058] S102, if the similarity between the current frame tracking result and the previous frame tracking result is less than a preset level, then the target recognition module identifies whether the next frame image contains the target to be tracked.

[0059] Furthermore, the target tracking framework determines the similarity between the current frame tracking result and the previous frame tracking result, and compares the similarity between the current frame tracking result and the previous frame tracking result with a preset degree to determine the magnitude relationship between the similarity between the current frame tracking result and the previous frame tracking result and the preset degree. The preset degree is set by the technician, and the magnitude relationship can be that the similarity is greater than or equal to the preset degree, or the similarity is less than the preset degree.

[0060] If the similarity between the current frame tracking result and the previous frame tracking result is less than a preset level, the target tracking framework determines that the target tracking module has failed to track the target to be tracked. Therefore, the target tracking framework will immediately switch to the target recognition module.

[0061] Furthermore, the target recognition module performs target recognition on the next frame image to identify whether the next frame image contains the target to be tracked, that is, the target recognition module identifies whether the next frame image contains the target of interest.

[0062] S103, if the target to be tracked is detected in the next frame image, the first target pixel bounding box of the target to be tracked in the next frame image is output, and the first target pixel bounding box is classified based on the target classification module to determine the target to be tracked again.

[0063] It should be noted that when tracking fails, the online learning module of the target classification module is first started, that is, the parameters of the target classification module are updated using a dynamically updated template library. In order to reduce the computational load of the target classification module, the online update iteration of the target classification module is 10 times. The updated target classification module can then perform classification.

[0064] Furthermore, if the target to be tracked is detected in the next frame, that is, if the target of interest is determined to be present in the next frame, the target tracking framework outputs the first target pixel bounding box of the target in the next frame. Further, the target classification module classifies the first target pixel bounding box, that is, classifies it as a real target and background (or a false target), thus identifying the target to be tracked again. After identifying the target to be tracked again, the system switches to the target tracking module, which continuously tracks the target.

[0065] The target tracking method provided in this embodiment of the invention obtains the tracking results of the current frame image and the previous frame image based on the target tracking module; if the similarity between the current frame tracking result and the previous frame tracking result is less than a preset level, the target recognition module identifies whether the next frame image contains the target to be tracked; if the next frame image contains the target to be tracked, the first target pixel bounding box of the target to be tracked in the next frame image is output, and the first target pixel bounding box is classified based on the target classification module to determine the target to be tracked again.

[0066] During target tracking, the target tracking module combines the tracking results from the previous frame and the current frame to track the target. If tracking fails based on the current frame's tracking result, the target recognition module re-identifies the target in the next frame. Then, based on the target classification module and the recognition results from the next frame, the target is classified to determine the target to be tracked again. This achieves an automatic closed loop from target recognition to target tracking, improving the robustness of target tracking.

[0067] Furthermore, S103 describes classifying the bounding box of the first target pixel based on the target classification module and determining the target to be tracked again. During the continuous tracking of the target, the parameters in the target classification module need to be continuously updated, as detailed below:

[0068] Collect a set of positive and negative samples from the current frame image of the re-tracked target;

[0069] The parameters in the target classification module are updated and trained based on the positive and negative sample sets to obtain the target classification module with updated parameters.

[0070] Specifically, during the continuous tracking of the re-tracked target, the target tracking framework collects a set of positive and negative samples from the current frame image of the re-tracked target. It should be noted that the positive and negative sample set includes positive samples and negative samples; positive samples are samples from the target image region containing the re-tracked target, and negative samples are samples from other image regions besides the target image region. Furthermore, the target tracking framework uses the positive and negative sample sets to update and train the parameters in the target classification module, resulting in a target classification module with updated parameters.

[0071] In this embodiment of the invention, the status information generated by the target tracking module during the tracking process is used to continuously update the parameters in the target classification module. This results in the target classification module having higher accuracy in classification after the parameter update, and more accurately classifying the target to be tracked.

[0072] Furthermore, in S102, if the target tracking module does not fail to track the target, that is, it continuously and successfully tracks the target, then the initial classification module is continuously updated with the status information generated by the target tracking module during the tracking process, resulting in the target classification module used in S102. The specific analysis is as follows:

[0073] If the similarity between the current frame tracking result and the previous frame tracking result is greater than or equal to the preset level, then the positive and negative sample sets of each frame image of the target to be tracked are collected.

[0074] The parameters in the initial classification module are updated and trained based on the positive and negative sample sets to obtain the target classification module.

[0075] Specifically, if the similarity between the current frame tracking result and the previous frame tracking result is greater than or equal to a preset level, the target tracking framework determines that the target tracking module has successfully tracked the target to be tracked. Therefore, the target tracking framework will continuously collect positive and negative sample sets of each frame image of the target to be tracked. The positive and negative sample sets include positive samples and negative samples. Positive samples are samples of the target image region containing the target to be tracked, and negative samples are samples of other image regions besides the target image region.

[0076] Furthermore, the target tracking framework updates and trains the parameters in the initial classification module using positive and negative sample sets to obtain the target classification module. If tracking is successful, the parameters in the initial classification module are continuously updated until tracking fails, at which point step S102 is switched.

[0077] Furthermore, the target tracking module is updated during the continuous tracking of the target, which is also the online update of the target classification module. The online update process is as follows:

[0078] During continuous tracking of the target, the target tracking module continuously generates state information of the target in new frame images. When the tracking algorithm in the target tracking module deems the current tracking result reliable, i.e., the target is being tracked successfully, it can dynamically update the sample library of the target classification module. Specifically, using the current tracking result and the current frame image, a batch of positive and negative target pixel bounding boxes are generated, with 50 positive samples and 200 negative samples. Simultaneously, the RoiAlign module extracts the features of these positive and negative samples, and then updates the initial classification module's sample library using a first-in-first-out (FIFO) rule, resulting in the target classification module.

[0079] In this embodiment of the invention, the parameters in the initial classification module are continuously updated by the status information generated by the target tracking module during the tracking process. This results in the updated target classification module having higher accuracy in classification and more accurately classifying the target to be tracked.

[0080] Furthermore, before updating the parameters in the initial classification module, the initial classification module needs to be trained. The specific training process is as follows:

[0081] The classifier is pre-trained and initialized to obtain the initial classification module, wherein the classifier is a backbone network composed of three fully connected layers.

[0082] The target classification algorithm in the target tracking framework is a lightweight target recognition algorithm that balances recognition performance and computational efficiency on the UAV. AlexNet or VGG-m is chosen as the backbone of the lightweight target recognition network, and a three-layer fully connected network is used as the backbone of the lightweight target classification network. The output layer consists of binary neurons corresponding to real and fake targets. The last fully connected layer contains multiple branches corresponding to different domains (video sequences). Different domain branches share the weights of the preceding convolutional layers and two fully connected layers. Each branch has two neurons, indicating that it classifies the input sample as either background or target.

[0083] To improve the running speed of the classification module, a region of interest feature extraction module, RoIAlign, is added after the last convolutional layer.

[0084] Using this module, feature extraction on each image only needs to be performed once during model training and testing. Then, for regions of interest of different sizes, this module can quickly map the features of that region to a feature map of the same size. In this object tracking framework, the feature size of all regions of interest is 3*3*512.

[0085] The online object classification module is described in detail based on three parts: model pre-training, online training, and testing. The specific analysis is as follows:

[0086] Model pre-training: The classifier weights were pre-trained on the ImageNet-VID large video dataset, which contains 4417 video sequences. The entire model was iterated 1000 times during offline training. In each epoch, the model randomly selected a classification branch for training until all branches were trained, completing one epoch. The model was trained using the stochastic gradient descent algorithm.

[0087] Online model training: Online training of classification models consists of two stages: model initialization and online update, which are analyzed in detail below:

[0088] Model Initialization: The multi-domain branches of the model pre-training phase are replaced with a single branch. During initialization, for the specific sequence being tracked, 500 positive samples and 5000 negative samples are extracted using given initial target information (including target position and size) and IoU criteria. In addition, for the initial frame of the sequence, samples are augmented through image rotation and mirror flipping. The number of positive and negative samples obtained through rotation is 50 and 100 respectively, and the number of positive and negative samples obtained through mirror flipping is also 50 and 100 respectively. Similarly, the three fully connected layers following the model are trained using stochastic gradient descent with a learning rate set to 0.0001, and the weights of the convolutional layers are frozen. Thus, the classifier is initialized as a classification model related to the currently tracked sequence, resulting in the initial classification module.

[0089] Online model update: During target tracking, the target tracking module continuously generates target state information in new images. When the tracking algorithm deems the current result reliable, it dynamically updates the target classification module's sample library. Specifically, using the current tracking results and test images, a batch of positive and negative target pixel bounding boxes are generated; the number of positive samples is 50, and the number of negative samples is 200. Simultaneously, the RoiAlign module extracts features from these positive and negative samples. Then, the classifier's sample library is updated using a first-in, first-out (FIFO) rule.

[0090] Furthermore, the pixel bounding box in the current frame recognition result recorded in S101 is obtained based on the target recognition module, and the determination method is analyzed in detail below:

[0091] The target recognition module performs target recognition on the current frame image to determine whether the current frame image contains the target to be tracked.

[0092] If the target to be tracked is contained in the current frame image, then the pixel coordinate center and bounding box size of the target to be tracked in the current frame image are output;

[0093] The pixel coordinate center and the bounding box size are determined as the pixel bounding box of the target to be tracked in the current frame image.

[0094] It should be noted that the target recognition algorithm used in the target tracking framework is a lightweight target recognition algorithm that balances recognition performance and computing efficiency on the UAV, such as Mobilenet-SSD, YOLO-v4, and YOLO-v5.

[0095] Specifically, the target recognition module performs target recognition on the current frame image to determine whether the current frame image contains the target to be tracked. If it is determined that the current frame image contains the target to be tracked, the target tracking framework outputs the pixel coordinate center and bounding box size of the target to be tracked in the current frame image. Further, the target tracking framework determines the pixel bounding box of the target to be tracked in the current frame image based on the pixel coordinate center and the bounding box size.

[0096] If it is determined that the target to be tracked is not contained in the current frame image, the target recognition module will continuously perform target recognition on the current frame image until the target to be tracked is identified.

[0097] In one embodiment, taking YOLO-v5 as an example, the confidence level for target recognition is set to 0.1, and the IoU threshold for the non-maximum suppression module is 0.45. Secondly, class labels (classes) for specific target selection are also required as input. The output of the target recognition module is the target detection results with class information, which can be viewed as a batch of labeled regions of interest. When the detection results are not empty, it means that a possible target exists in that frame.

[0098] According to the embodiments of the present invention, the pixel bounding box of the target to be tracked in the image is accurately determined based on the pixel coordinate center of the target and the bounding box size.

[0099] Furthermore, the specific analysis described in S103 regarding the classification of the first target pixel bounding box based on the target classification module to determine the target to be re-tracked is as follows:

[0100] The target classification module classifies the bounding box of the first target pixel to obtain multiple samples to be processed.

[0101] The multiple samples to be processed are classified according to a confidence threshold to obtain a positive sample set, wherein the confidence of the samples to be processed in the positive sample set is greater than the confidence threshold.

[0102] The target to be re-tracked is determined based on the positive sample set.

[0103] Specifically, the target classification module classifies the bounding box of the first target pixel to obtain multiple samples to be processed. The number of samples to be processed is determined by the number of bounding boxes of the first target pixel. It should be noted that each sample to be processed has its own confidence level. Furthermore, the number of samples to be processed may also be one; this embodiment of the invention provides multiple examples.

[0104] Furthermore, the target classification module compares the confidence score of each sample to a confidence threshold, and classifies multiple samples based on the confidence threshold to determine all positive samples. Specifically, the target classification module classifies samples with a confidence score greater than the confidence threshold as positive samples, and samples with a confidence score less than or equal to the confidence threshold as negative samples. The confidence threshold is set according to the actual situation. In one embodiment, the confidence threshold is 0; samples with a confidence score greater than 0 are classified as positive samples, and samples with a confidence score less than or equal to 0 are classified as negative samples. Further, the target tracking framework aggregates all positive samples to obtain a positive sample set.

[0105] Furthermore, the target tracking framework determines whether the number of samples to be processed in the positive sample set is greater than a preset number. The preset number is set according to the actual situation. For example, if the preset number is 0, it can be understood as whether the number of samples to be processed in the positive sample set is greater than 0.

[0106] If the number of samples to be processed in the positive sample set is greater than the preset number, that is, the number of samples to be processed in the positive sample set is greater than 0, the target tracking framework will determine the target to be tracked again based on the positive sample set.

[0107] In this embodiment of the invention, the target classification module accurately classifies the target to be tracked again.

[0108] The specific analysis for determining the re-tracking target based on the positive sample set is as follows:

[0109] The samples to be processed in the positive sample set are sorted in descending order of confidence level to obtain the sorted samples to be processed.

[0110] The sample with the highest confidence among the sorted samples to be processed is determined as the target sample;

[0111] The pixel bounding box of the target sample is determined as the second target pixel bounding box, and the re-tracking target is determined based on the second target pixel bounding box.

[0112] Specifically, the target tracking framework sorts the samples to be processed in the positive sample set according to their confidence level from high to low, thus obtaining the sorted samples to be processed.

[0113] Furthermore, the target tracking framework identifies the sample with the highest confidence among the sorted samples as the target sample.

[0114] Furthermore, the target tracking framework determines the pixel bounding box of the target sample as the second target pixel bounding box. Finally, the target tracking framework classifies the target to be tracked again based on the second target pixel bounding box.

[0115] In this embodiment of the invention, the target to be re-tracked is classified based on the pixel bounding box of the positive sample with the highest confidence level, thus accurately classifying the target to be re-tracked.

[0116] Furthermore, the target tracking method provided in this embodiment of the invention can be understood as follows:

[0117] During tracking, if the target is successfully tracked, the parameters of the target classification module are continuously updated. If target tracking fails, in the next frame, the target tracking framework first starts the target recognition module and updates the model weights of the target classification module using the sample database.

[0118] If the target recognition module returns an empty result, the target re-detection for that frame fails, and the process continues with the next frame. If the target recognition module returns a non-empty result, the automatic target tracking framework starts the updated target classification module to perform target re-recognition.

[0119] If the target classification module returns an empty result, the target re-identification for that frame fails, and processing continues to the next frame; otherwise, the re-identification result is used as the state of the target in that frame, and the target tracking module is restarted in the next frame.

[0120] Furthermore, the target tracking method provided in this embodiment of the invention can be understood as follows:

[0121] Step 1: The system starts running. First, it enters the target recognition module. Based on the given initial frame image and the set target category information, the target recognition algorithm identifies the target of interest in the image. Based on the result of the target recognition algorithm, the target to be tracked is selected, and the target information is sent to the classification module and the tracking module respectively, proceeding to Step 2.

[0122] Step two: The target classification module initializes based on the selected target information; the target tracking module initializes based on the selected target information, then proceeds to step three.

[0123] Step 3: For a new image frame, the target tracking module performs target tracking based on the tracking results of the previous frame and the results of the current image frame. When the target tracking module successfully tracks the target, it returns the tracking result to the target tracking frame. If the current frame is the last frame of the sequence, proceed to step 6; otherwise, continue with step 3. When tracking fails, if the current frame is the last frame of the sequence, proceed to step 6; otherwise, proceed to step 4.

[0124] Step 4: The system enters the target recognition module. First, for a new image frame, the system runs the target recognition algorithm. Second, it determines whether the result of the target recognition algorithm contains the target of interest. If so, the recognition result is sent to the target classification module and the system proceeds to Step 5. Otherwise, the system returns an empty value. If the current image is the last frame of the sequence, the system proceeds to Step 6. Otherwise, the system continues to execute Step 4.

[0125] Step 5: The system performs online classification based on the valid results returned by the target recognition module. First, the target classification module classifies the generated samples, assigning a confidence level greater than a confidence threshold to positive samples. When the set of positive samples in the classification results is not empty, the algorithm reorders these positive samples according to their confidence levels, finds the positive sample with the highest confidence, and returns the target box of the highest-confidence positive sample as a successful re-identification result to the target tracking module. If the current image is the last frame of the sequence, proceed to Step 6; otherwise, proceed to Step 3. When all classification results are negative, if the current image is the last frame of the sequence, proceed to Step 6; otherwise, proceed to Step 4.

[0126] Step Six: System operation ends.

[0127] Furthermore, the target tracking device provided by the present invention will be described below, and the target tracking device and the target tracking method can be referred to in correspondence with each other.

[0128] like Figure 2 As shown, Figure 2 This is a structural diagram of the target tracking device provided by the present invention. The target tracking device includes:

[0129] The tracking unit 201 is used to obtain the tracking results of the current frame image and the previous frame image based on the target tracking module;

[0130] The identification unit 202 is used to identify whether the target to be tracked is contained in the next frame image by means of the target identification module if the similarity between the current frame tracking result and the previous frame tracking result is less than a preset level.

[0131] The classification unit 203 is used to output the first target pixel bounding box of the target in the next frame image if the target to be tracked is detected in the next frame image, and classify the first target pixel bounding box based on the target classification module to determine the target to be tracked again.

[0132] Furthermore, the tracking unit 201 is also used for:

[0133] The target recognition module performs target recognition on the current frame image to determine whether the current frame image contains the target to be tracked.

[0134] If the target to be tracked is contained in the current frame image, then the pixel coordinate center and bounding box size of the target to be tracked in the current frame image are output;

[0135] The pixel coordinate center and the bounding box size are determined as the pixel bounding box of the target to be tracked in the current frame image.

[0136] Furthermore, classification unit 203 is also used for:

[0137] The target classification module classifies the bounding box of the first target pixel to obtain multiple samples to be processed.

[0138] The multiple samples to be processed are classified according to a confidence threshold to obtain a positive sample set, wherein the confidence of the samples to be processed in the positive sample set is greater than the confidence threshold.

[0139] The target to be re-tracked is determined based on the positive sample set.

[0140] Furthermore, classification unit 203 is also used for:

[0141] The samples to be processed in the positive sample set are sorted in descending order of confidence level to obtain the sorted samples to be processed.

[0142] The sample with the highest confidence among the sorted samples to be processed is determined as the target sample;

[0143] The pixel bounding box of the target sample is determined as the second target pixel bounding box, and the re-tracking target is determined based on the second target pixel bounding box.

[0144] Furthermore, the target tracking device also includes: a training unit, used for:

[0145] Collect a set of positive and negative samples from the current frame image of the re-tracked target;

[0146] The parameters in the target classification module are updated and trained based on the positive and negative sample sets to obtain the target classification module with updated parameters.

[0147] The positive and negative sample set includes positive samples and negative samples. The positive samples are samples containing the target image region of the re-tracked target, and the negative samples are samples from other image regions besides the target image region.

[0148] Furthermore, the training unit is also used for:

[0149] If the similarity between the current frame tracking result and the previous frame tracking result is greater than or equal to the preset level, then the positive and negative sample sets of each frame image of the target to be tracked are collected.

[0150] The parameters in the initial classification module are updated and trained based on the positive and negative sample sets to obtain the target classification module.

[0151] The positive and negative sample set includes positive samples and negative samples. The positive samples are samples containing the target image region of the target to be tracked, and the negative samples are samples from other image regions besides the target image region.

[0152] Furthermore, the training unit is also used for:

[0153] The classifier is pre-trained and initialized to obtain the initial classification module, wherein the classifier is a backbone network composed of three fully connected layers.

[0154] The specific embodiments of the target tracking device provided by the present invention are basically the same as the embodiments of the target tracking method described above, and will not be repeated here.

[0155] Figure 3 An example of a physical structure diagram of an electronic device is shown, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a target tracking method, which includes:

[0156] The tracking results of the current frame image and the previous frame image are obtained based on the target tracking module;

[0157] If the similarity between the current frame tracking result and the previous frame tracking result is less than a preset level, the target recognition module will identify whether the next frame image contains the target to be tracked.

[0158] If the target to be tracked is detected in the next frame, the first target pixel bounding box of the target to be tracked in the next frame is output, and the first target pixel bounding box is classified based on the target classification module to determine the target to be tracked again.

[0159] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] On the other hand, the present invention also provides a computer program product, comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, enable the computer to perform the target tracking method provided by the above methods, the method comprising:

[0161] The tracking results of the current frame image and the previous frame image are obtained based on the target tracking module;

[0162] If the similarity between the current frame tracking result and the previous frame tracking result is less than a preset level, the target recognition module will identify whether the next frame image contains the target to be tracked.

[0163] If the target to be tracked is detected in the next frame, the first target pixel bounding box of the target to be tracked in the next frame is output, and the first target pixel bounding box is classified based on the target classification module to determine the target to be tracked again.

[0164] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the target tracking methods provided above, the method comprising:

[0165] The tracking results of the current frame image and the previous frame image are obtained based on the target tracking module;

[0166] If the similarity between the current frame tracking result and the previous frame tracking result is less than a preset level, the target recognition module will identify whether the next frame image contains the target to be tracked.

[0167] If the target to be tracked is detected in the next frame, the first target pixel bounding box of the target to be tracked in the next frame is output, and the first target pixel bounding box is classified based on the target classification module to determine the target to be tracked again.

[0168] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0169] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target tracking method, characterized in that, include: The tracking results of the current frame image and the previous frame image are obtained based on the target tracking module; If the similarity between the current frame tracking result and the previous frame tracking result is less than a preset level, the target recognition module will identify whether the next frame image contains the target to be tracked. If the target to be tracked is detected in the next frame, the first target pixel bounding box of the target to be tracked in the next frame is output, and the first target pixel bounding box is classified based on the target classification module to determine the target to be tracked again. After classifying the bounding box of the first target pixel based on the target classification module and determining the target to be tracked again, the method further includes: Collect a set of positive and negative samples from the current frame image of the re-tracked target; The parameters in the target classification module are updated and trained based on the positive and negative sample sets to obtain the target classification module with updated parameters. The positive and negative sample set includes positive samples and negative samples. The positive samples are samples containing the target image region of the re-tracked target, and the negative samples are samples from other image regions besides the target image region. After obtaining the tracking results of the current frame image and the previous frame image based on the target tracking module, the method further includes: If the similarity between the current frame tracking result and the previous frame tracking result is greater than or equal to the preset level, then the positive and negative sample sets of each frame image of the target to be tracked are collected. The parameters in the initial classification module are updated and trained based on the positive and negative sample sets to obtain the target classification module. The positive and negative sample set includes positive samples and negative samples. The positive samples are samples containing the target image region of the target to be tracked, and the negative samples are samples of other image regions besides the target image region. Before updating and training the parameters in the initial classification module based on the positive and negative sample sets to obtain the target classification module, the method further includes: The classifier is pre-trained and initialized to obtain the initial classification module, wherein the classifier is a backbone network composed of three fully connected layers; The pixel bounding box in the current frame recognition result is obtained based on the target recognition module, and the specific steps include: The target recognition module performs target recognition on the current frame image to determine whether the current frame image contains the target to be tracked. If the target to be tracked is contained in the current frame image, then the pixel coordinate center and bounding box size of the target to be tracked in the current frame image are output; The pixel coordinate center and the bounding box size are determined as the pixel bounding box of the target to be tracked in the current frame image.

2. The target tracking method according to claim 1, characterized in that, The step of classifying the bounding box of the first target pixel based on the target classification module to determine the target to be tracked again includes: The target classification module classifies the bounding box of the first target pixel to obtain multiple samples to be processed. The multiple samples to be processed are classified according to a confidence threshold to obtain a positive sample set, wherein the confidence of the samples to be processed in the positive sample set is greater than the confidence threshold. The target to be re-tracked is determined based on the positive sample set.

3. The target tracking method according to claim 2, characterized in that, Determining the re-tracking target based on the positive sample set includes: The samples to be processed in the positive sample set are sorted in descending order of confidence level to obtain the sorted samples to be processed. The sample with the highest confidence among the sorted samples to be processed is determined as the target sample; The pixel bounding box of the target sample is determined as the second target pixel bounding box, and the re-tracking target is determined based on the second target pixel bounding box.

4. A target tracking device, characterized in that, include: The tracking unit is used to obtain the tracking results of the current frame image and the previous frame image based on the target tracking module; The recognition unit is used to identify whether the target to be tracked is contained in the next frame image if the similarity between the current frame tracking result and the previous frame tracking result is less than a preset level. The classification unit is used to output the first target pixel bounding box of the target in the next frame image if the target to be tracked is detected in the next frame image, and classify the first target pixel bounding box based on the target classification module to determine the target to be tracked again. The training unit is used to: collect a set of positive and negative samples of the current frame image of the re-tracked target; update and train the parameters in the target classification module based on the set of positive and negative samples to obtain the target classification module with updated parameters; wherein, the set of positive and negative samples includes positive samples and negative samples, the positive samples are samples containing the target image region of the re-tracked target, and the negative samples are samples of other image regions besides the target image region; The training unit is further configured to: if the similarity between the current frame tracking result and the previous frame tracking result is greater than or equal to the preset level, then collect a set of positive and negative samples for each frame of the target to be tracked; update and train the parameters in the initial classification module based on the set of positive and negative samples to obtain the target classification module; wherein, the set of positive and negative samples includes positive samples and negative samples, the positive samples are samples containing the target image region of the target to be tracked, and the negative samples are samples of other image regions besides the target image region; The training unit is further configured to: perform model pre-training and model initialization on the classifier to obtain the initial classification module, wherein the classifier is a backbone network composed of three fully connected layers; The tracking unit is further configured to: perform target recognition on the current frame image through the target recognition module to determine whether the current frame image contains the target to be tracked; if the current frame image contains the target to be tracked, output the pixel coordinate center and bounding box size of the target to be tracked in the current frame image; and determine the pixel coordinate center and the bounding box size as the pixel bounding box of the target to be tracked in the current frame image.

5. An electronic device, the electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the target tracking method according to any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the target tracking method according to any one of claims 1 to 3.

Images