Target tracking method, device, terminal device and readable storage medium

Through the combination of feature pyramid network and regional candidate subnetwork, the feature maps of templates and detection frames are extracted, which solves the problem of insufficient timeliness in deep learning in target tracking, and achieves high-precision small-target tracking.

CN111815677BActive Publication Date: 2024-11-26广州新华学院
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010661194.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-07-10
Publication Date
2024-11-26
Estimated Expiration
2040-07-10

AI Technical Summary

Technical Problem

In the field of target tracking, deep learning has problems such as large training samples and long online update time, which leads to insufficient timeliness and is difficult to achieve efficient tracking.

Method used

The feature pyramid network is used to extract template feature maps of different scales from template frames, and the detection feature maps of corresponding scales are extracted from the detection frames in combination with the regional candidate subnet. The scores of alternative images are calculated through convolution operations, classification and regression branches. and location, filter out the tracking target.

Benefits of technology

Effectively integrating low-level high-resolution information and high-level semantic information improves target positioning accuracy, especially in small-target tracking, achieving accurate tracking of small-targets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111815677B_ABST
    Figure CN111815677B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention discloses a target tracking method, device, terminal device and readable storage medium, which includes extracting n template feature maps of different scales from a template frame using a feature pyramid network; extracting n detection feature maps of different scales from a detection frame using a feature pyramid network; determining a pre-selected number of candidate images and scores and positions corresponding to each candidate image on the detection feature map according to the i-th template feature map and the detection feature map of the same scale as the i-th template feature map using the i-th region candidate subnetwork; after the n-th region candidate subnetworks process the n-th template feature map and the corresponding detection feature map, determining the top m candidate images with the highest scores among all candidate images according to the scores of each candidate image as tracking targets, and the positions corresponding to the tracking targets as the target positions. This solution realizes accurate tracking of small targets.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of target tracking, and in particular to a target tracking method, device, terminal equipment and readable storage medium. Background Art

[0002] In recent years, deep learning has begun to enter the field of target tracking, attracting the attention of more and more scholars. The low-level features of the tracking image have a higher resolution, which is convenient for accurate positioning of the target, while the high-level features contain more semantic information, can handle large target changes and prevent the tracker from drifting, and are more convenient for range positioning of the target. Therefore, deep learning can better extract the features of the target and better express the target. However, due to the problems of large training samples and long online update time in deep learning, there are still considerable challenges in the timeliness of tracking. Summary of the invention

[0003] In view of the above problems, the present invention proposes a target tracking method, apparatus, terminal device and readable storage medium.

[0004] A first embodiment of the present invention provides a target tracking method, the method comprising:

[0005] Use the feature pyramid network to extract n template feature maps of different scales from the template frame;

[0006] Extracting n detection feature maps of different scales corresponding to the scales of the n template feature maps from the detection frame using a feature pyramid network;

[0007] Determine, using the i-th region candidate subnetwork according to the i-th template feature map and the detection feature map of the same scale as the i-th template feature map, a pre-selected number of candidate images, and the scores and positions corresponding to each candidate image on the detection feature map;

[0008] When the n region candidate subnetworks have completed processing the n template feature maps and the corresponding detection feature maps, the top m candidate images with the highest scores are determined as tracking targets among all the candidate images according to the scores of the candidate images, and the positions corresponding to the tracking targets are used as the target positions.

[0009] The target tracking method proposed in the second embodiment of the present invention, wherein the method uses the i-th region candidate subnetwork to determine a pre-selected number of candidate images on the detection feature map according to the i-th template feature map and the detection feature map with the same scale as the i-th template feature map, as well as the scores and positions corresponding to each candidate image, including:

[0010] Performing a convolution operation on the i-th template feature map to obtain a template sample set of a first preset size and a template sample set of a second preset size;

[0011] Performing a convolution operation on a detection feature map having the same scale as the i-th template feature map to obtain a detection sample set of a third preset size and a detection sample set of a fourth preset size;

[0012] The classification branch of the i-th region candidate subnetwork calculates the score of each candidate image in the detection sample set of the third preset size according to the template sample set of the first preset size and the detection sample set of the third preset size;

[0013] The regression branch of the i-th region candidate subnetwork determines the position of each sample in the detection sample set of the fourth preset size according to the template sample set of the second preset size and the detection sample set of the fourth preset size;

[0014] The positions corresponding to the candidate images are determined according to the positions of the samples in the detection sample set of the fourth preset size, and the detection sample set of the third preset size is equal to the detection sample set of the fourth preset size.

[0015] The target tracking method provided in the third embodiment of the present invention further includes:

[0016] Determine the position response values ​​of the target positions corresponding to the first m candidate images;

[0017] When the maximum position response value is greater than a preset response threshold, the candidate image corresponding to the maximum position response value is used as the template frame.

[0018] In the above target tracking method, the response value is calculated according to the following formula:

[0019] in, represents the position response value, t * represents the target position, y(t * ) represents the target position t * , t represents the interference position closest to the target position, y(t) represents the response result of the interference position t, and Δ is a quadratic continuously differentiable function.

[0020] The target tracking method described in the above embodiment uses a training sample set to pre-train a target tracking model corresponding to the target tracking method until the error loss of the target tracking model is less than a preset error threshold;

[0021] The error loss is calculated using the following loss function:

[0022] in, represents the error loss, β i represents the weight coefficient of the candidate sub-network of the i-th region, μ represents the attenuation parameter, Represents the weighted response value of the n region candidate sub-networks;

[0023] The weighted response value calculation formula after weighting is as follows:

[0024] Among them, y β (t * ) represents the target position t * The weighted response result of , t represents the interference position closest to the target position, y β (t) represents the weighted response result of the interference position t, Δ is a quadratic continuous differentiable function;

[0025] The weighted response result calculation formula is as follows:

[0026] s i (t) represents the response result of the candidate sub-network of the i-th region.

[0027] In the target tracking method described in the above embodiment, the n different scales include at least one of a 32×32 pixel scale, a 64×64 pixel scale, a 128×128 pixel scale, and a 256×256 pixel scale.

[0028] A fourth embodiment of the present invention provides a target tracking device, the device comprising:

[0029] The template feature map acquisition module is used to extract n template feature maps of different scales from the template frame using a feature pyramid network;

[0030] A detection feature map acquisition module, used for extracting n detection feature maps of different scales corresponding to the scales of the n template feature maps from the detection frame by using a feature pyramid network;

[0031] A candidate image determination module is used to determine a pre-selected number of candidate images, and a score and position corresponding to each candidate image on the detection feature map according to the i-th template feature map and the detection feature map with the same scale as the i-th template feature map using the i-th region candidate sub-network;

[0032] The tracking target determination module is used to determine the top m candidate images with the highest scores among all the candidate images as tracking targets after the n region candidate sub-networks have processed the n template feature maps and the corresponding detection feature maps, based on the scores of the candidate images, and the positions corresponding to the tracking targets are used as the target positions.

[0033] The above-mentioned alternative image determination module includes:

[0034] A template sample set acquisition unit, configured to perform a convolution operation on the i-th template feature map to acquire a template sample set of a first preset size and a template sample set of a second preset size;

[0035] A detection sample set acquisition unit, used for performing a convolution operation on a detection feature map having the same scale as the i-th template feature map to acquire a detection sample set of a third preset size and a detection sample set of a fourth preset size;

[0036] a candidate image score calculation unit, configured for the classification branch of the i-th region candidate subnetwork to calculate the score of each candidate image in the detection sample set of the third preset size according to the template sample set of the first preset size and the detection sample set of the third preset size;

[0037] a sample position determination unit, configured for the regression branch of the i-th region candidate subnetwork to determine the position of each sample in the detection sample set of the fourth preset size according to the template sample set of the second preset size and the detection sample set of the fourth preset size;

[0038] The candidate image position determination unit is used to determine the position corresponding to each candidate image according to the position of each sample in the detection sample set of the fourth preset size, and the detection sample set of the third preset size is equivalent to the detection sample set of the fourth preset size.

[0039] The above embodiment relates to a terminal device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program so that the terminal device can execute the above target tracking method.

[0040] The above embodiment relates to a readable storage medium storing a computer program, which executes the above target tracking method when running on a processor.

[0041] The present invention uses a feature pyramid network to extract n template feature maps of different scales from a template frame; uses a feature pyramid network to extract n detection feature maps of different scales corresponding to the scales of the n template feature maps from a detection frame; uses an i-th region candidate subnetwork to determine a pre-selected number of candidate images on the detection feature map according to the i-th template feature map and the detection feature map with the same scale as the i-th template feature map, as well as the scores and positions corresponding to each candidate image; after the n region candidate subnetworks complete processing of the n template feature maps and the corresponding detection feature maps, the top m candidate images with the highest scores among all candidate images are determined as tracking targets according to the scores of each candidate image, and the positions corresponding to the tracking targets are used as target positions. The technical solution of the present invention, on the one hand, uses a feature pyramid network as a feature extraction layer of a tracking framework, effectively integrates low-level high-resolution information and high-level high-semantic information, can more accurately locate the target position, and the tracking performance on small target tracking objects is particularly outstanding; on the other hand, the tracking target is screened by an improved region candidate subnetwork to achieve accurate tracking of small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope of protection of the present invention. In each of the drawings, similar components are numbered similarly.

[0043] Figure 1 A schematic diagram of a target tracking method according to an embodiment of the present invention is shown;

[0044] Figure 2 A schematic diagram of the structure of a target tracking model according to an embodiment of the present invention is shown;

[0045] Figure 3 A schematic diagram of a process for determining the scores and positions of candidate images according to an embodiment of the present invention is shown;

[0046] Figure 4 A schematic diagram of the structure of a regional candidate sub-network model according to an embodiment of the present invention is shown;

[0047] Figure 5 A schematic diagram showing a flow chart of another target tracking method according to an embodiment of the present invention;

[0048] Figure 6 A schematic diagram of the structure of a target tracking device according to an embodiment of the present invention is shown;

[0049] Figure 7 A schematic diagram of a structure for determining the scores and positions of candidate images according to an embodiment of the present invention is shown.

[0050] Description of main component symbols:

[0051] 1-target tracking device; 100-template feature map acquisition module; 200-detection feature map acquisition module; 300-alternative image determination module; 400-tracking target determination module; 500-response value calculation module; 600-template frame update module; 310-template sample set acquisition unit; 320-detection sample set acquisition unit; 330-alternative image score calculation unit; 340-sample position determination unit; 350-alternative image position determination unit. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0053] The components of the embodiments of the present invention generally described and shown in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0054] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present invention, are intended only to indicate specific features, numbers, steps, operations, elements, components or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing items or adding the possibility of one or more features, numbers, steps, operations, elements, components or combinations of the foregoing items.

[0055] Furthermore, the terms “first”, “second”, “third”, etc. are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0056] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present invention belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present invention.

[0057] The present invention is a target tracking method proposed by improving three existing technologies, including a twin neural network structure, a feature pyramid network and a region candidate network.

[0058] The twin network structure refers to the main structure of the network consisting of two branches, the upper and lower branches, which share all the weights of the same network, just like twins. It is often used to solve classification problems with many or uncertain categories, but a small number of samples under each category. In the field of visual target tracking, the upper branch of the twin network is the template branch used to extract the appearance features of the template frame, and the lower branch is the detection branch. Its input is the candidate area for search captured on the current frame based on the tracking result of the previous frame. After passing through the same network, the feature map of the template branch is used to calculate the similarity with the feature maps of multiple candidate areas of the current frame, and the candidate area with the highest score is used as the tracking result of the current frame.

[0059] A feature pyramid usually refers to a single input image that can generate multiple images of different scales. The four vertices of these images of different scales are connected to construct an image pyramid similar to a real pyramid. The Feature Pyramid Network (FPN) used in the field of target tracking can add a scale dimension (or depth) to a two-dimensional image. Unlike traditional detection algorithms that only use top-level features for prediction, the Feature Pyramid Network can fuse features at different levels and also make independent predictions at different feature layers to obtain more robust semantic information. Through its own bottom-up and top-down lines, the Feature Pyramid Network makes full use of the high-resolution information of the lower layers and the high-semantic information of the higher layers. Especially for small targets, the Feature Pyramid Network also increases the resolution of the feature map, which can obtain more useful information about small targets.

[0060] The Region Proposal Network (RPN) is a network used to extract candidate frames, which first appeared in the Faster RCNN structure. It uses candidate frames (also called anchor frame technology), which are usually used to represent fixed reference frames in computer vision. In the target tracking task, the tracked target has characteristics such as uncertain category, uncertain position, and uncertain scale. The improved anchor frame technology can pre-set a set of fixed reference frames of different scales and positions to cover approximately all positions and scales. Each fixed reference frame is responsible for detecting targets whose intersection-over-union ratio is greater than a preset threshold. The region candidate network not only has good recognition effect, but also fast recognition speed.

[0061] Example 1

[0062] In this embodiment, see Figure 1 , showing a target tracking method comprising the following steps:

[0063] Step S100: extracting n template feature maps of different scales from the template frame using a feature pyramid network.

[0064] At the initial stage of target tracking, the first frame in the video frame can be used as a template frame, and the feature pyramid network (FPN) can be used to extract n template feature maps of different scales from the template frame. For example, see Figure 2 , the feature pyramid network (FPN) extracts 4 template feature maps of different scales from the template frame, including a template feature map of 32×32 pixel scale, a template feature map of 64×64 pixel scale, a template feature map of 128×128 pixel scale, and a template feature map of 256×256 pixel scale.

[0065] It should be understood that n is a positive integer and can be set to 3, 4, 5, 6, etc., and can be adjusted according to the actual effect of target tracking. The scale of the template feature map can also be flexibly set according to specific needs.

[0066] Step S200: extracting n detection feature maps of different scales corresponding to the scales of the n template feature maps from the detection frame using a feature pyramid network.

[0067] The detection frame is the current frame in which the tracking target needs to be obtained, and the feature pyramid network is used to extract n detection feature maps of different scales corresponding to the scales of the n template feature maps from the detection frame. Figure 2 , the feature pyramid network (FPN) extracts 4 detection feature maps of different scales from the detection frame, including 32×32 pixel scale detection feature map, 64×64 pixel scale detection feature map, 128×128 pixel scale detection feature map and 256×256 pixel scale detection feature map.

[0068] Step S300: using the i-th region candidate subnetwork to determine a pre-selected number of candidate images, as well as the scores and positions corresponding to each candidate image on the detection feature map according to the i-th template feature map and the detection feature map of the same scale as the i-th template feature map.

[0069] For example, see Figure 2When n is 4, it corresponds to 4 region candidate subnetworks (RPNs), the template feature map of 32×32 pixels and the detection feature map of 32×32 pixels are used as the input of the first region candidate subnetwork (RPN), the template feature map of 64×64 pixels and the detection feature map of 64×64 pixels are used as the input of the second region candidate subnetwork (RPN), the template feature map of 128×128 pixels and the detection feature map of 128×128 pixels are used as the input of the third region candidate subnetwork (RPN), and the template feature map of 256×256 pixels and the detection feature map of 256×256 pixels are used as the input of the fourth region candidate subnetwork (RPN).

[0070] The four region candidate subnetworks (RPNs) determine the pre-selected number of candidate images, as well as the scores and positions of each candidate image on the corresponding detection feature map. Three anchor boxes with aspect ratios can be pre-set on the detection feature map of each scale to cover the tracking targets that may exist in the detection feature map. The three aspect ratios include 1:2, 1:1 and 2:1. Correspondingly, the detection feature maps of the four scales include 12 preset anchor boxes.

[0071] The classification branch of the region candidate subnetwork (RPN) can be used to calculate the classification score of each anchor box, and the regression branch of the region candidate subnetwork (RPN) can be used to determine the regression position of each anchor box. It should be understood that each anchor box contains an alternative image, the regression position of each anchor box is the position of the corresponding alternative image, and the classification score of each anchor box is the score of the corresponding alternative image.

[0072] Step S400: After the n region candidate sub-networks have completed processing the n template feature maps and the corresponding detection feature maps, the top m candidate images with the highest scores among all the candidate images are determined as tracking targets according to the scores of the candidate images, and the positions corresponding to the tracking targets are used as the target positions.

[0073] According to the scores corresponding to the candidate images, the candidate images are sorted in order from high to low, and the first m candidate images with the highest scores are determined as tracking targets, where m is a preset positive integer.

[0074] The above i is less than or equal to n.

[0075] This embodiment uses a feature pyramid network to extract n template feature maps of different scales from a template frame; uses a feature pyramid network to extract n detection feature maps of different scales corresponding to the scales of the n template feature maps from a detection frame; uses the i-th region candidate subnetwork to determine a pre-selected number of candidate images on the detection feature map according to the i-th template feature map and the detection feature map with the same scale as the i-th template feature map, as well as the scores and positions corresponding to each candidate image; after the n region candidate subnetworks have processed the n template feature maps and the corresponding detection feature maps, the top m candidate images with the highest scores among all candidate images are determined as tracking targets according to the scores of each candidate image, and the positions corresponding to the tracking targets are used as the target positions. On the one hand, the technical solution of this embodiment uses the feature pyramid network as the feature extraction layer of the tracking framework, effectively integrating the high-resolution information of the low layer and the high-semantic information of the high layer, and can more accurately locate the target position, and the tracking performance on small target tracking objects is particularly outstanding; on the other hand, the tracking targets are screened by the improved region candidate subnetwork to achieve accurate tracking of small targets.

[0076] Example 2

[0077] The region candidate subnetwork, such as Figure 4 As shown, it contains a classification branch for distinguishing targets from backgrounds and a regression branch for bounding box regression.

[0078] For further information, see Figure 3 , step S300 of the above embodiment 1 includes the following steps:

[0079] Step S310: performing a convolution operation on the i-th template feature map to obtain a template sample set of a first preset size and a template sample set of a second preset size.

[0080] The first convolutional layer of the i-th region candidate subnetwork performs a convolution operation on the input i-th template feature map to obtain a template sample set of a first preset size and a template sample set of a second preset size.

[0081] Furthermore, a template sample set of a first preset size of 4×4×(2k×256) is obtained on the classification branch of the i-th region candidate subnetwork. The features of the template sample with a size of 4×4 have 2k variations in k different anchor frames. It should be understood that the 2k variations indicate that the image in each anchor frame may exist in two situations, either background or target, that is, two states of 0 or 1.

[0082] Furthermore, a template sample set of a second preset size of 4×4×(4k×256) is obtained on the regression branch of the i-th region candidate subnetwork. The features of the template sample represented as 4×4 pixels have 4k variations on k different anchor boxes. It should be understood that the 4k variations correspond to the width, height, horizontal coordinate and vertical coordinate of the position, and each anchor box is represented by the corresponding width, height, horizontal coordinate and vertical coordinate.

[0083] Step S320: performing a convolution operation on the detection feature map having the same scale as the i-th template feature map to obtain a detection sample set of a third preset size and a detection sample set of a fourth preset size.

[0084] The first convolution layer of the i-th region candidate subnetwork performs a convolution operation on an input detection feature map having the same scale as the i-th template feature map to obtain a detection sample set of a third preset size and a detection sample set of a fourth preset size.

[0085] Furthermore, a detection sample set of a third preset size of 20×20×256 is obtained on the classification branch of the i-th region candidate subnetwork. A detection sample set of the fourth preset size of 20×20×256 is obtained on the regression branch of the i-th region candidate subnetwork.

[0086] The 256 in the above scale represents the number of channels of the template sample. The feature dimension is expanded to 256 dimensions through the network training process of the feature pyramid.

[0087] Step S330: The classification branch of the i-th region candidate subnetwork calculates the score of each candidate image in the detection sample set of the third preset size according to the template sample set of the first preset size and the detection sample set of the third preset size.

[0088] The classification branch will give the classification score corresponding to each input detection sample, that is, the detailed score predicted as the target or background. The corresponding score can be expressed as ★Indicates related operations.

[0089] Step S340: the regression branch of the i-th region candidate subnetwork determines the position of each sample in the detection sample set of the fourth preset size according to the template sample set of the second preset size and the detection sample set of the fourth preset size.

[0090] The regression branch gives the position regression value of each detection sample This position regression value contains the horizontal coordinate, vertical coordinate, width and height, corresponding to d x , d y , d w and d h Four values, ★Indicates related operations.

[0091] Step S350: determining the positions corresponding to the candidate images according to the positions of the samples in the detection sample set of the fourth preset size, wherein the detection sample set of the third preset size is equivalent to the detection sample set of the fourth preset size.

[0092] The detection sample set of the third preset size and the detection sample set of the fourth preset size are sample sets of the same size, and the positions corresponding to the respective candidate images can be determined according to the positions of the respective samples in the detection sample sets of the four preset sizes.

[0093] The first m candidate classification output information And regression output information The location information of the m candidate locations with the highest scores can be obtained The specific calculation formula is as follows:

[0094]

[0095]

[0096]

[0097] in are the original center coordinates and length and width of the candidate box corresponding to the i-th candidate position, cls represents the classification branch and reg represents the regression branch. For each subscript: i∈[0,w), j∈[0,h), l∈[0,2k), p∈[0,k), each A is a vector set of output information.

[0098] Example 3

[0099] In this embodiment, see Figure 5 , showing that the target tracking method further includes the following steps after the above steps S100-S400:

[0100] Step S500: Determine the position response values ​​of the target positions corresponding to the first m candidate images.

[0101] The position response values ​​of the target positions corresponding to the first m candidate images can be calculated according to the following formulas:

[0102] in, represents the position response value, t * represents the target position corresponding to any one of the m candidate images, y(t * ) represents the target position t * t represents the interference position closest to the target position, y(t) represents the response result of the interference position t, Δ is a quadratic continuously differentiable function, t and t* The closer Δ(tt * ) approaches 0, t and t * The farther the distance, the greater the * ) approaches 1.

[0103] It should be understood that, according to the above formula, m position response values ​​corresponding to the m candidate positions can be determined.

[0104] Step S600: when the maximum position response value is greater than a preset response threshold, the candidate image corresponding to the maximum position response value is used as the template frame.

[0105] A candidate image whose corresponding position response value is greater than a preset response threshold is selected from the m candidate positions as a new template frame to continue to execute the above step S100.

[0106] The above implementation is based on the online update method of high-scoring sample feedback, which uses the high-scoring candidate samples in the tracking process as new template frames for subsequent detection tasks, effectively improving the accuracy and robustness of target tracking.

[0107] Furthermore, the target tracking model corresponding to the target tracking method is pre-trained using the training sample set until the error loss of the target tracking model is less than a preset error threshold; the error loss is calculated using the following loss function:

[0108] in, represents the error loss, β i represents the weight coefficient of the candidate sub-network of the i-th region, μ represents the attenuation parameter, represents the weighted response value of the n regional candidate sub-networks after weighting, and the weighted response value calculation formula is as follows:

[0109] Among them, y β (t * ) represents the target position t * The weighted response result of , t represents the interference position closest to the target position, y β (t) represents the weighted response result of the interference position t, Δ is a quadratic continuous differentiable function; the weighted response result calculation formula is as follows:

[0110] s i (t) represents the response result of the candidate sub-network of the i-th region, where the sum of the weighted coefficients is 1, that is,

[0111] The above loss function is used to calculate the error loss of the target tracking model corresponding to the target tracking method. When the error loss of the target tracking model is less than the preset error threshold, it indicates that the tracking quality of the target tracking model corresponding to the target tracking method meets the standard.

[0112] Example 4

[0113] In this embodiment, see Figure 6 , showing a target tracking device 1 including: a template feature map acquisition module 100, a detection feature map acquisition module 200, an alternative image determination module 300 and a tracking target determination module 400.

[0114] The template feature map acquisition module 100 is used to extract n template feature maps of different scales from the template frame using a feature pyramid network; the detection feature map acquisition module 200 is used to extract n detection feature maps of different scales corresponding to the scales of the n template feature maps from the detection frame using a feature pyramid network; the candidate image determination module 300 is used to determine a pre-selected number of candidate images, as well as the scores and positions corresponding to each candidate image on the detection feature map using the i-th region candidate subnetwork according to the i-th template feature map and the detection feature map of the same scale as the i-th template feature map; the tracking target determination module 400 is used to determine the top m candidate images with the highest scores among all the candidate images as tracking targets according to the scores of each candidate image after the n region candidate subnetworks have completed processing the n template feature maps and the corresponding detection feature maps, and the positions corresponding to the tracking targets are used as target positions.

[0115] For further information, see Figure 7 The candidate image determination module 300 includes: a template sample set acquisition unit 310, a detection sample set acquisition unit 320, a candidate image score calculation unit 330, a sample position determination unit 340 and a candidate image position determination unit 350.

[0116] The template sample set acquisition unit 310 is used to perform a convolution operation on the i-th template feature map to obtain a template sample set of a first preset size and a template sample set of a second preset size; the detection sample set acquisition unit 320 is used to perform a convolution operation on the detection feature map of the same scale as the i-th template feature map to obtain a detection sample set of a third preset size and a detection sample set of a fourth preset size; the candidate image score calculation unit 330 is used for the classification branch of the i-th region candidate subnetwork to calculate the score of each candidate image in the detection sample set of the third preset size according to the template sample set of the first preset size and the detection sample set of the third preset size; the sample position determination unit 340 is used for the regression branch of the i-th region candidate subnetwork to determine the position of each sample in the detection sample set of the fourth preset size according to the template sample set of the second preset size and the detection sample set of the fourth preset size; the candidate image position determination unit 350 is used to determine the position corresponding to each candidate image according to the position of each sample in the detection sample set of the fourth preset size, and the detection sample set of the third preset size is equivalent to the detection sample set of the fourth preset size.

[0117] A target tracking device 1 also includes: a response value calculation module 500, used to determine the position response value of the target position corresponding to the first m candidate images; a template frame updating module 600, used to use the candidate image corresponding to the maximum position response value as the template frame when the maximum position response value is greater than a preset response threshold.

[0118] The target tracking device 1 of this embodiment is used to execute the target tracking method described in the above embodiment through the coordinated use of the template feature map acquisition module 100, the detection feature map acquisition module 200, the alternative image determination module 300 and the tracking target determination module 400. The implementation scheme and beneficial effects involved in the above embodiment are also applicable in this embodiment and will not be repeated here.

[0119] The above embodiment relates to a terminal device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program so that the terminal device can execute the target tracking method described in the above embodiment.

[0120] The above embodiment relates to a readable storage medium storing a computer program. When the computer program is run on a processor, the target tracking method described in the above embodiment is executed.

[0121] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or the flow chart, and the combination of boxes in the structure diagram and / or the flow chart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0122] In addition, the functional modules or units in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0123] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0124] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A target tracking method, characterized in that: The method includes: Use the feature pyramid network to extract n template feature maps of different scales from the template frame; Extracting n detection feature maps of different scales corresponding to the scales of the n template feature maps from the detection frame using a feature pyramid network; Determine, using the i-th region candidate subnetwork according to the i-th template feature map and the detection feature map of the same scale as the i-th template feature map, a pre-selected number of candidate images, and the scores and positions corresponding to each candidate image on the detection feature map; After the n region candidate subnetworks have processed the n template feature maps and the corresponding detection feature maps, the top m candidate images with the highest scores are determined as tracking targets among all the candidate images according to the scores of the candidate images, and the positions corresponding to the tracking targets are used as the target positions; Pre-training a target tracking model corresponding to the target tracking method using a training sample set until an error loss of the target tracking model is less than a preset error threshold; The error loss is calculated using the following loss function: Among them, L t* (β) represents the error loss, β i represents the weight coefficient of the candidate sub-network of the i-th region, μ represents the attenuation parameter, Represents the weighted response value of the n region candidate sub-networks; The weighted response value calculation formula is as follows: Among them, y β (t * ) represents the target position t * The weighted response result of , t represents the interference position closest to the target position, y β (t) represents the weighted response result of the interference position t, Δ is a quadratic continuous differentiable function; The weighted response result calculation formula is as follows: s i (t) represents the response result of the candidate sub-network of the i-th region.

2. The target tracking method according to claim 1, characterized in that: The method of using the i-th region candidate subnetwork to determine a pre-selected number of candidate images on the detection feature map according to the i-th template feature map and the detection feature map having the same scale as the i-th template feature map, as well as the scores and positions corresponding to the candidate images, includes: Performing a convolution operation on the i-th template feature map to obtain a template sample set of a first preset size and a template sample set of a second preset size; Performing a convolution operation on a detection feature map having the same scale as the i-th template feature map to obtain a detection sample set of a third preset size and a detection sample set of a fourth preset size; The classification branch of the i-th region candidate subnetwork calculates the score of each candidate image in the detection sample set of the third preset size according to the template sample set of the first preset size and the detection sample set of the third preset size; The regression branch of the i-th region candidate subnetwork determines the position of each sample in the detection sample set of the fourth preset size according to the template sample set of the second preset size and the detection sample set of the fourth preset size; The positions corresponding to the candidate images are determined according to the positions of the samples in the detection sample set of the fourth preset size, and the detection sample set of the third preset size is equal to the detection sample set of the fourth preset size.

3. The target tracking method according to claim 1, characterized in that: Also includes: Determine the position response values ​​of the target positions corresponding to the first m candidate images; When the maximum position response value is greater than a preset position response threshold, the candidate image corresponding to the maximum position response value is used as the template frame.

4. The target tracking method according to claim 3, characterized in that: The position response value is calculated according to the following formula: Among them, ξ t* {y} represents the position response value, t * represents the target position, y(t * ) represents the target position t * , t represents the interference position closest to the target position, y(t) represents the response result of the interference position t, and Δ is a quadratic continuously differentiable function.

5. The target tracking method according to any one of claims 1 to 4, characterized in that: The n different scales include at least one of a 32×32 pixel scale, a 64×64 pixel scale, a 128×128 pixel scale, and a 256×256 pixel scale.

6. A target tracking device, characterized in that: The device is used to execute the target tracking method according to claim 1, comprising: The template feature map acquisition module is used to extract n template feature maps of different scales from the template frame using a feature pyramid network; A detection feature map acquisition module, used for extracting n detection feature maps of different scales corresponding to the scales of the n template feature maps from the detection frame by using a feature pyramid network; A candidate image determination module is used to determine a pre-selected number of candidate images, and a score and position corresponding to each candidate image on the detection feature map according to the i-th template feature map and the detection feature map with the same scale as the i-th template feature map using the i-th region candidate sub-network; The tracking target determination module is used to determine the top m candidate images with the highest scores among all the candidate images as tracking targets after the n region candidate sub-networks have processed the n template feature maps and the corresponding detection feature maps, based on the scores of the candidate images, and the positions corresponding to the tracking targets are used as the target positions.

7. The target tracking device according to claim 6, characterized in that: The candidate image determination module comprises: A template sample set acquisition unit, configured to perform a convolution operation on the i-th template feature map to acquire a template sample set of a first preset size and a template sample set of a second preset size; A detection sample set acquisition unit, used for performing a convolution operation on a detection feature map having the same scale as the i-th template feature map to acquire a detection sample set of a third preset size and a detection sample set of a fourth preset size; a candidate image score calculation unit, configured for the classification branch of the i-th region candidate subnetwork to calculate the score of each candidate image in the detection sample set of the third preset size according to the template sample set of the first preset size and the detection sample set of the third preset size; a sample position determination unit, configured for the regression branch of the i-th region candidate subnetwork to determine the position of each sample in the detection sample set of the fourth preset size according to the template sample set of the second preset size and the detection sample set of the fourth preset size; The candidate image position determination unit is used to determine the position corresponding to each candidate image according to the position of each sample in the detection sample set of the fourth preset size, and the detection sample set of the third preset size is equivalent to the detection sample set of the fourth preset size.

8. A terminal device, characterized in that: It comprises a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program so that the terminal device can execute the target tracking method according to any one of claims 1 to 5.

9. A readable storage medium, characterized in that: It stores a computer program, which executes the target tracking method according to any one of claims 1 to 5 when running on a processor.