A target tracking method, device and storage medium

Abstract

Embodiments of the present application provide a target tracking method and device and a storage medium. The method comprises: extracting deep features and shallow features from a template image and a to-be-searched image corresponding to a current frame respectively; performing feature fusion on the obtained features based on a preset fusion coefficient to obtain a fused feature map; performing target classification and target regression on the fused feature map to obtain a first score parameter of each pixel in the to-be-searched image and a plurality of candidate regions in which a to-be-tracked target is located; and determining a target region in which the to-be-tracked target is located in the to-be-searched image from the plurality of candidate regions according to a second score parameter of each pixel in the to-be-searched image obtained based on quality evaluation and a corresponding first score parameter. Through the above method, features containing different information at different levels are fused, the ability of model recognition and positioning is improved, and the tracking accuracy is improved by establishing a mechanism for evaluating the classification result.

Description

Technical Field

[0001] This application relates to the fields of image processing and computer vision technology, and in particular to a target tracking method, device and storage medium. Background Technology

[0002] With the continuous development of artificial intelligence (AI) technologies, target tracking, as an important branch of AI, has received increasing attention and plays a vital role in many fields, including video surveillance, autonomous driving, and medical imaging. Successful applications in these areas demonstrate the technology's significant practical value and promising future. However, video sequences are diverse; the shape of the target may change, rapid movement can cause blurring, altered lighting conditions can lead to flickering brightness, and the target may be occluded. These complexities make target tracking a challenging task.

[0003] In current single-target tracking algorithms based on Siamese networks, feature extraction is a crucial step in the algorithm model. Generally, neural networks are used to extract single features from the target to be tracked and subsequent frame images. However, single feature information often does not perform well in certain scenarios, and if only a single feature is used for target tracking, the tracking results will be biased. Summary of the Invention

[0004] This application provides a target tracking method, device, and storage medium. By extracting multi-level features and fusing them, the target tracking accuracy is improved.

[0005] In a first aspect, embodiments of this application provide a target tracking method, the method comprising:

[0006] Deep and shallow features are extracted from the template image and the search image corresponding to the current frame, respectively, to obtain a first feature map of multiple levels corresponding to the template image and a second feature map of multiple levels corresponding to the search image; the template image is obtained by extracting the target to be tracked from the reference frame of the video, and the search image is determined based on the region where the target to be tracked was located in the previous frame;

[0007] Based on the preset fusion coefficients, feature fusion is performed on each first feature map and each second feature map to obtain a fused feature map;

[0008] Based on the fused feature map, target classification and target regression are performed to obtain the first scoring parameter of each pixel in the image to be searched, as well as multiple candidate regions where the target to be tracked is located; the first scoring parameter represents the first probability that the corresponding pixel is located in the target region where the target to be tracked is located;

[0009] Based on the second scoring parameter and the corresponding first scoring parameter of each pixel in the search image obtained based on quality assessment, the target region where the target to be tracked is located in the search image is determined from the plurality of candidate regions; the second scoring parameter represents the second probability that the corresponding pixel is located in the target region where the target to be tracked is located.

[0010] In the above embodiments, features at multiple levels are extracted from the template image and the alternative search image, and then these features are fused together. The fused feature map is then used for subsequent target tracking. Since the fused feature map, based on multiple levels of feature maps, contains both shallow features such as color, edges, and corners of the image, and deep features such as semantics, it is suitable for target tracking in most scenarios. Furthermore, a quality assessment method is introduced to further improve the target tracking accuracy.

[0011] In one possible implementation, the step of fusing features between each first feature map and each second feature map based on a preset fusion coefficient to obtain a fused feature map includes:

[0012] Based on a preset first fusion coefficient, feature fusion is performed on each first feature map and each second fusion feature map to obtain a first fusion feature map for target classification.

[0013] Based on a preset second fusion coefficient, feature fusion is performed on each first feature map and each second fusion feature map to obtain a second fusion feature map for target regression.

[0014] In the above embodiments, since target classification is for distinguishing the target from the background, and target regression is for accurately locating the area where the target is located, the fusion coefficients used for target classification and target regression are different.

[0015] In one possible implementation, the step of fusing features between the first feature maps and the second fusion feature maps based on a preset first fusion coefficient to obtain a first fusion feature map for target classification includes:

[0016] For each first feature map and each second feature map at the same level as each first feature map, the first feature map and the second feature map are convolved to obtain a first convolutional feature map;

[0017] The first convolutional feature maps of each level are weighted and summed according to the preset first fusion coefficient to obtain the first fusion feature map used for target classification.

[0018] In one possible implementation, the step of fusing features between the first feature maps and the second fused feature maps based on a preset second fusion coefficient to obtain a second fused feature map for target regression includes:

[0019] For each first feature map and each second feature map at the same level as each first feature map, the first feature map and the second feature map are convolved to obtain a second convolved feature map;

[0020] The second convolutional feature maps of each level are weighted and summed according to the second fusion coefficient to obtain the second fusion feature map used for target regression.

[0021] In the above embodiments, based on different tasks, different fusion coefficients are used to fuse features at different levels. The resulting first fused feature map and second feature map not only contain high-level features such as semantic information, but also low-level features such as color and texture information, thereby improving the accuracy of target tracking.

[0022] In one possible implementation, the step of extracting deep and shallow features from the template image and the search image respectively to obtain a first feature map of multiple levels corresponding to the template image and a second feature map of multiple levels corresponding to the search image includes:

[0023] Deep features and shallow features are extracted from the template image and the image to be searched using a multi-layered convolutional network; at least one of the number of convolutional layers and the number of convolutional kernels in the different layers of the convolutional network is different.

[0024] The outputs of at least two layers of the convolutional network are used as at least two first feature maps corresponding to the template image, and the outputs of at least two layers of the convolutional network are used as at least two second feature maps corresponding to the image to be searched.

[0025] In the above embodiments, a feature extraction network is constructed using convolutional layers. Shallow and deep features can be obtained simply by using the network outputs of different levels of this feature extraction network. The network structure is simple and does not require the template and the image to be searched to be input into the shallow and deep feature extraction networks respectively for feature extraction.

[0026] In one possible implementation, determining the target region in the search image where the target to be tracked is located from the plurality of candidate regions, based on the second scoring parameter of each pixel in the search image obtained based on quality assessment and the corresponding first scoring parameter, includes:

[0027] Based on the product of the second scoring parameter and the corresponding first scoring parameter of each pixel in the image to be searched obtained from the quality assessment, the total probability that each pixel in the image to be searched is located in the target region where the target to be tracked is located is determined.

[0028] The region with the highest total probability among the multiple candidate regions is selected as the target region where the target to be tracked is located in the search region.

[0029] In the above embodiments, by establishing a quality assessment mechanism, the probability of each pixel in the image to be searched being located in the target region of the target to be tracked can be obtained more accurately, thereby improving the accuracy of target tracking. In addition, in the process of determining the target region, it is based on the results of target classification and quality assessment, without introducing prior data, thereby improving the generalization ability of the entire target tracking algorithm.

[0030] In one possible implementation, target regression is performed based on the fused feature map to obtain multiple candidate regions where the target to be tracked is located, including:

[0031] For each pixel in the image to be searched, the fused feature map is input into the regression network to obtain a set of candidate distance values ​​corresponding to the pixel. The set of candidate distance values ​​is used to describe the distance between the pixel coordinates and the boundaries of the candidate regions corresponding to the pixel target.

[0032] Based on the set of candidate distance values, the candidate region corresponding to the pixel is determined.

[0033] In one possible implementation, determining the candidate region corresponding to the pixel based on the set of candidate distance values ​​includes:

[0034] The difference between the ordinate of the pixel coordinate and the first distance is used as the ordinate of the first corner position of the candidate region corresponding to the pixel coordinate. The first distance is used to characterize the distance between the pixel coordinate and the first boundary of the corresponding candidate region.

[0035] The difference between the x-coordinate of the pixel coordinate and the second distance is used as the x-coordinate of the first corner position of the candidate region corresponding to the pixel coordinate. The second distance is used to characterize the distance between the pixel coordinate and the second boundary of the corresponding candidate region.

[0036] The sum of the ordinate of the pixel coordinate and the third distance is used as the ordinate of the second corner position of the candidate region corresponding to the pixel coordinate. The third distance is used to characterize the distance between the pixel coordinate and the third boundary of the corresponding candidate region.

[0037] The sum of the x-coordinate of the pixel coordinate and the fourth distance is used as the x-coordinate of the second corner position of the candidate region corresponding to the pixel coordinate. The fourth distance is used to characterize the distance between the pixel coordinate and the fourth boundary of the corresponding candidate region.

[0038] In the above embodiments, the candidate region is determined based on the coordinates of each pixel in the image to be searched, without introducing a preset anchor (prior data), thereby improving the generalization ability of the entire target tracking algorithm.

[0039] Secondly, embodiments of this application provide a target tracking device, the device comprising:

[0040] The feature extraction module is used to extract deep features and shallow features from the template image and the search image corresponding to the current frame, respectively, to obtain a first feature map of multiple levels corresponding to the template image and a second feature map of multiple levels corresponding to the search image; the template image is obtained by extracting the target to be tracked from the reference frame of the video, and the search image is determined based on the region where the target to be tracked was located in the previous frame;

[0041] The feature fusion module is used to fuse each first feature map and each second feature map based on preset fusion coefficients to obtain a fused feature map;

[0042] The classification and regression module is used to perform target classification and target regression based on the fused feature map to obtain the first scoring parameter of each pixel in the image to be searched, as well as multiple candidate regions where the target to be tracked is located; the first scoring parameter represents the first probability that the corresponding pixel is located in the target region where the target to be tracked is located;

[0043] The target region determination module is used to determine the target region in the search image where the target to be tracked is located from the plurality of candidate regions based on the second scoring parameter of each pixel in the search image obtained based on quality assessment and the corresponding first scoring parameter. The second scoring parameter represents the second probability that the corresponding pixel is located in the target region where the target to be tracked is located.

[0044] Thirdly, embodiments of this application provide a target tracking device, the device comprising:

[0045] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described in the first aspect above.

[0046] Fourthly, embodiments of this application provide a computer storage medium storing a computer program for causing a computer to perform the method described in the first aspect above. Attached Figure Description

[0047] Figure 1 This is a schematic diagram illustrating an application scenario of a target tracking method provided in an embodiment of this application.

[0048] Figure 2 This is a schematic flowchart of a target tracking method provided in an embodiment of this application;

[0049] Figure 3 This is a schematic diagram of the structure of a feature extraction network provided in an embodiment of this application;

[0050] Figure 4 This is a schematic diagram of the algorithm structure of a target tracking method provided in an embodiment of this application;

[0051] Figure 5 A schematic diagram of a candidate region provided in an embodiment of this application;

[0052] Figure 6 This is a schematic diagram illustrating a specific process of a target tracking method provided in an embodiment of this application;

[0053] Figure 7 A schematic diagram of a target tracking device provided in an embodiment of this application;

[0054] Figure 8 This is a schematic diagram of a target tracking device provided in an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0056] like Figure 1 The diagram illustrates an application scenario of a target tracking method provided in this application. This scenario includes: a server 101, a database 102, a reference frame image 103_1, and at least one subsequent frame image (shown as the first frame image 103_2 and the first frame image 103_3 after the reference frame). The server 101 performs target tracking based on the reference frame image 103_1 and at least one subsequent frame image; it extracts the target to be tracked from the reference frame image 103_3 and, using this target as a reference, determines the region where the target is located in each subsequent frame image; the database 102 stores the data and programs required for executing the defect detection method.

[0057] In current single-target tracking algorithms based on Siamese networks, feature extraction is a crucial step in the algorithm model. However, existing feature extraction methods often fail to adequately represent the target using single-level deep features, and low-level features tend to express only simple characteristics. As the network depth increases, the abstraction level of deep features becomes higher, and the semantic information of high-level features becomes richer. If only low-level features or only high-level features are used for target tracking, the tracking results may be biased in certain scenarios due to the reliance on a single feature. To address these issues, this application provides a target tracking method, such as... Figure 2 As shown, the method includes:

[0058] S201: Extract deep features and shallow features from the template image and the search image corresponding to the current frame respectively to obtain a first feature map of multiple levels corresponding to the template image and a second feature map of multiple levels corresponding to the search image.

[0059] Before feature extraction, the reference frame image and the current frame image need to be processed. A template image is obtained by extracting the target to be tracked from the reference frame of the video. The template image is fixed. In this embodiment, if the initial frame does not contain the target to be tracked, or the target to be tracked in the initial frame is not clear enough, other frames can be selected for target extraction. The current frame image is cropped based on the target region in the previous frame image to obtain a search region that is larger than the target region in the previous frame image.

[0060] In this embodiment of the application, deep features and shallow features are extracted from the template image and the image to be searched using multiple layers of convolutional networks; at least one of the number of convolutional layers and the number of convolutional kernels in the different layers of the convolutional networks is different.

[0061] The outputs of at least two layers of the convolutional network in the plurality of layers are used as at least two first feature maps corresponding to the template image, and the outputs of at least two layers of the convolutional network in the plurality of layers are used as at least two second feature maps corresponding to the image to be searched. In this embodiment of the application, the number of layers of the feature extraction network and the parameters of each layer of the network (size of convolutional kernel, number of convolutional kernels, stride, etc.) are not specifically limited, ensuring that at least one shallow feature (pixel information, such as the color, texture, edge, corner, position and details of the image) and at least one deep information (abstract information, such as semantic information) are output.

[0062] like Figure 3 The diagram shown is a structural diagram of a feature extraction network (residual network-50) provided in an embodiment of this application. It includes a network structure with 5 layers, wherein the 3rd layer network, the 4th layer network, and the 5th layer network output feature maps respectively.

[0063] The first layer consists of one convolutional layer and one pooling layer, both with a stride of 2. The second layer has nine convolutional layers, grouped in sets of three with the same structure, and each convolutional layer has a stride of 1. The third layer has twelve convolutional layers, grouped in sets of three, with the stride of 1 for the second convolutional layer in the first group, and the stride of the remaining convolutional layers being 1. The fourth layer has eighteen convolutional layers, grouped in sets of three with the same structure, and each convolutional layer has a stride of 1. The fifth layer has nine convolutional layers, grouped in sets of three with the same structure, and each convolutional layer has a stride of 1. The total stride of the feature extraction network is 8.

[0064] Figure 3 The network shown can extract both deep and shallow features using only one network structure while maintaining a simple network structure.

[0065] S202: Based on the preset fusion coefficient, perform feature fusion on each first feature map and each second feature map to obtain a fused feature map.

[0066] In one possible implementation, feature fusion is performed on each first feature map and each second fusion feature map based on a preset first fusion coefficient to obtain a first fusion feature map for target classification. The specific process is as follows: Figure 4 As shown.

[0067] The first convolutional feature map is determined using Formula 1. The first convolutional feature maps corresponding to each layer of the network are weighted and summed according to the preset first fusion coefficient to obtain the first fusion feature map used for target classification, as shown in Formula 2.

[0068]

[0069]

[0070] Among them, C i This represents the first convolutional feature map corresponding to the i-th layer of the network. This represents the first feature map (i-th) output after features are extracted from the network at the i-th layer. This represents the i-th second feature map output after features are extracted from the i-th layer of the network; C total Represents the first fused feature map, γ i This represents the fusion coefficient corresponding to the i-th layer of the network used for target classification. For different layers of the network, the corresponding fusion coefficients can be set to the same or different values. For example, if semantic information is more important for the template image, the first fusion coefficient corresponding to the deep feature extraction network is set to be greater than the first fusion coefficient corresponding to the shallow feature extraction network, and the sum of all the first fusion coefficients is 1.

[0071] like Figure 4 As shown, based on a preset second fusion coefficient, feature fusion is performed on each first feature map and each second fusion feature map to obtain a second fusion feature map for target regression.

[0072] Formula 3 is used to determine the second convolutional feature map. The second convolutional feature maps corresponding to each layer of the network are weighted and summed according to the preset second fusion coefficient to obtain the second fusion feature map used for target regression, as shown in Formula 4.

[0073]

[0074]

[0075] Among them, B i This represents the second convolutional feature map corresponding to the i-th layer of the network. This represents the i-th second feature map output after features are extracted by the network at the i-th layer. This represents the i-th second feature map output after features are extracted from the i-th layer of the network; B total Represents the second fused feature map, λ i This represents the fusion coefficient corresponding to the i-th layer of the network used for target regression. For different layers of the network, the corresponding fusion coefficients can be set to the same or different values. For example, if color and edge information are more important for the image to be searched, the second fusion coefficient corresponding to the shallow feature extraction network is set to be greater than the second fusion coefficient corresponding to the deep feature extraction network, and the sum of all the second fusion coefficients is 1.

[0076] Figure 4The network structure for extracting features from the template image is the same as the network structure for extracting features from the search image. Figure 3 Taking the network structure shown as an example, the outputs of the 3rd, 4th, and 5th layer network structures are feature maps from shallow features to deep features.

[0077] S203: Based on the fused feature map, perform target classification and target regression to obtain the first score parameters of each pixel in the image to be searched, as well as multiple candidate regions where the target to be tracked is located.

[0078] like Figure 4 As shown, after feature fusion, there are two task branches: classification and regression. The purpose of target classification is to distinguish the target to be tracked from the background, and the purpose of target regression is to determine the region of the target to be tracked in the image to be searched.

[0079] (1) In the classification branches.

[0080] The first feature fusion map is input into the classification network to obtain the first scoring parameter of each pixel in the image to be searched, wherein the first scoring parameter represents the first probability that the corresponding pixel is located in the target region where the target to be tracked is located.

[0081] To improve the network's operating efficiency, the feature extraction step downsamples the template image and the image to be searched according to the total step size of the network (set to 8 in this embodiment, but can also be set to other values). Therefore, the element coordinates corresponding to the results output by the classification network are restored to the pixel coordinates in the search area according to Formula 5.

[0082] Formula 5: x' = s / 2 + xs, y' = s / 2 + ys

[0083] Here, (x',y') represents the coordinates of a pixel in the image to be searched, and (x,y) represents the corresponding coordinates in the output of the classification network. Since the size of the output of the classification network differs from the search area, the first scoring parameter corresponding to (x,y) in the image to be searched corresponds to a region (total step size * total step size) of multiple pixels centered at (x',y'). For example, with a total step size of 8, if (x,y) is (0,0), then (x',y') = (4,4). Therefore, in the image to be searched, the 8*8 pixels surrounding (4,4) are all part of the candidate region.

[0084] The output of the classification network is a w*h first rating parameter matrix (between 0 and 1), such as... Given a 3x3 first scoring parameter matrix, where the maximum score is 0.95 and its coordinates in the first scoring parameter matrix are (1,1) (with the position of "0.15" in the lower left corner as the origin), the coordinates of the corresponding pixel in the image to be searched (taking a total step size of 8 as an example) are (8 / 2+8, 8 / 2+8) = (12,12). Therefore, the 8x8 pixels centered at (12,12) are all in the candidate region.

[0085] (2) In the regression branch.

[0086] The second fused feature map is input into the regression network to obtain a set of candidate distance values ​​(l, t, r, b) corresponding to the pixel. This set of candidate distance values ​​describes the distance between the pixel coordinates and the boundaries of the candidate region corresponding to the pixel target. l represents the distance between the pixel coordinates and the left boundary of the candidate region corresponding to the pixel target; t represents the distance between the pixel coordinates and the upper boundary of the candidate region corresponding to the pixel target; r represents the distance between the pixel coordinates and the right boundary of the candidate region corresponding to the pixel target; and b represents the distance between the pixel coordinates and the lower boundary of the candidate region corresponding to the pixel target.

[0087] The specific implementation method for determining the candidate region corresponding to the pixel using (l,t,r,b) is as follows:

[0088] The difference between the ordinate of the pixel coordinate and the first distance is used as the ordinate of the first corner position of the candidate region corresponding to the pixel coordinate. The first distance is used to characterize the distance between the pixel coordinate and the first boundary of the corresponding candidate region. That is, calculated by formula 6, y0' is the ordinate of the upper left corner position of the candidate region, y' is the ordinate of the pixel in the search region, and t reg This represents the distance value output by the regression network.

[0089] y′0=y′-t reg Formula 6

[0090] The difference between the x-coordinate of the pixel coordinate and the second distance is used as the x-coordinate of the first corner position of the candidate region corresponding to the pixel coordinate. The second distance is used to characterize the distance between the pixel coordinate and the second boundary of the corresponding candidate region. That is, calculated by formula 7, x0' is the x-coordinate of the upper left corner position of the candidate region, x' is the x-coordinate of the pixel in the search region, and t reg This represents the distance value output by the regression network.

[0091] x′0=x′-l reg Formula 7

[0092] The sum of the ordinate of the pixel coordinate and the third distance is used as the ordinate of the second corner position of the candidate region corresponding to the pixel coordinate. The third distance is used to characterize the distance between the pixel coordinate and the third boundary of the corresponding candidate region. That is, calculated using Formula 8, where y1' is the ordinate of the lower right corner position of the candidate region, y' is the ordinate of the pixel in the search region, and b... reg This represents the distance value output by the regression network.

[0093] y′1=y′+b reg Formula 8

[0094] The sum of the x-coordinate of the pixel coordinate and the fourth distance is used as the x-coordinate of the second corner position of the candidate region corresponding to the pixel coordinate. The fourth distance is used to characterize the distance between the pixel coordinate and the fourth boundary of the corresponding candidate region. That is, calculated by formula 9, x1' is the x-coordinate of the lower right corner position of the candidate region, x' is the x-coordinate of the pixel in the search region, and r reg This represents the distance value output by the regression network.

[0095] x′1=x′+r reg Formula 9

[0096] Among them, such as Figure 5 The candidate region shown has the first and second boundaries perpendicular to each other, the second and fourth boundaries parallel to each other, the third and fourth boundaries perpendicular to each other, and the first and third boundaries parallel to each other.

[0097] The above implementation method can finally obtain the coordinates (x0', y0') of the upper left corner of the candidate region in the search region, and the coordinates (x1', y1') of the lower right corner of the candidate region in the search region.

[0098] In the above target classification and target regression process, no anchors (prior boxes) are set in advance, that is, no multiple rectangles of different sizes are set in advance in the search area. Instead, the candidate area is determined directly based on the position of the pixel, which improves the generalization ability of the regression network and the classification network.

[0099] S204: Based on the second scoring parameter and the corresponding first scoring parameter of each pixel in the image to be searched obtained based on quality assessment, determine the target region in the image to be tracked from the plurality of candidate regions.

[0100] Using Formula 10, a set of distance values ​​output by the regression network is used to evaluate the quality of each pixel in the search image to obtain the second scoring parameter.

[0101]

[0102] The second scoring parameter represents the second probability that the corresponding pixel is located in the target area of ​​the target to be tracked.

[0103] Based on the product of the second scoring parameter and the corresponding first scoring parameter of each pixel in the image to be searched obtained from the quality assessment, the total probability that each pixel in the image to be searched is located in the target region where the target to be tracked is located is determined.

[0104] For example, for a certain pixel, if its first scoring parameter is 0.8 and its second scoring parameter is 0.9, then the total probability that the pixel is located in the target area of ​​the target to be tracked is 0.8 * 0.9 = 0.72.

[0105] The region with the highest total probability among the multiple candidate regions is selected as the target region where the target to be tracked is located in the search region.

[0106] Each pixel in the image to be searched corresponds to a total probability and a candidate region. If a certain pixel has the highest total probability, then its corresponding candidate region is the target region.

[0107] In addition, if the image to be searched obtained based on the current frame is not the last frame (the next frame also includes the target to be tracked), then the next frame image continues to proceed from S201 to S204; if the next frame does not include the target to be tracked, then S201 to S204 are stopped.

[0108] The following is through Figure 6 The overall process of a target tracking method provided in the embodiments of this application is described.

[0109] First, deep and shallow features are extracted from the template image and the search image corresponding to the current frame, respectively, to obtain a first feature map of multiple levels corresponding to the template image and a second feature map of multiple levels corresponding to the search image, and the corresponding features are fused. Then, target classification and target regression are performed based on the fused feature maps to obtain the first scoring parameters of each pixel in the search image and multiple candidate regions where the target to be tracked is located. In addition, based on the second scoring parameters of each pixel in the search image obtained based on quality assessment and the corresponding first scoring parameters, the target region where the target to be tracked is located in the search image is determined from the multiple candidate regions. Finally, it is determined whether the next frame includes the target to be tracked. If it does, the above process is performed on the next frame; if it does not, the tracking result is output.

[0110] In this embodiment, features containing different information at different levels are fused through feature fusion and input into the classification and regression branches respectively to improve the model's recognition and localization capabilities. By establishing a mechanism to evaluate the classification results, the scores of rectangular boxes far from the center position are suppressed, thereby improving tracking accuracy. Moreover, for existing anchor-based tracking algorithms, the setting of anchor boxes introduces prior knowledge about the data distribution, which affects the model's generalization ability. Furthermore, the introduction of multiple anchor boxes brings unnecessary computational burden. This embodiment does not set anchor boxes and directly predicts the candidate region corresponding to the pixel coordinates, reducing the impact of prior knowledge on the tracking results.

[0111] Based on the same inventive concept, this application provides a target tracking device, such as... Figure 7 As shown, the device 700 includes:

[0112] The feature extraction module 701 is used to extract deep features and shallow features from the template image and the search image corresponding to the current frame, respectively, to obtain a first feature map of multiple levels corresponding to the template image and a second feature map of multiple levels corresponding to the search image; the template image is obtained by extracting the target to be tracked from the reference frame of the video, and the search image is determined based on the region where the target to be tracked is located in the previous frame;

[0113] The feature fusion module 702 is used to perform feature fusion on each first feature map and each second feature map based on preset fusion coefficients to obtain a fused feature map;

[0114] The classification and regression module 703 is used to perform target classification and target regression based on the fused feature map to obtain the first scoring parameter of each pixel in the image to be searched, and multiple candidate regions where the target to be tracked is located; the first scoring parameter represents the first probability that the corresponding pixel is located in the target region where the target to be tracked is located;

[0115] The target region determination module 704 is used to determine the target region in the search image from the plurality of candidate regions based on the second scoring parameter and the corresponding first scoring parameter of each pixel in the search image obtained based on quality assessment; the second scoring parameter represents the second probability that the corresponding pixel is located in the target region where the target is located.

[0116] In one possible implementation, the feature fusion module 702 is used to perform feature fusion on each first feature map and each second feature map based on a preset fusion coefficient to obtain a fused feature map, including:

[0117] Based on a preset first fusion coefficient, feature fusion is performed on each first feature map and each second fusion feature map to obtain a first fusion feature map for target classification.

[0118] Based on a preset second fusion coefficient, feature fusion is performed on each first feature map and each second fusion feature map to obtain a second fusion feature map for target regression.

[0119] In one possible implementation, the feature fusion module 702 is used to perform feature fusion on the first feature maps and the second fusion feature maps based on a preset first fusion coefficient to obtain a first fusion feature map for target classification, including:

[0120] For each first feature map and each second feature map at the same level as each first feature map, the first feature map and the second feature map are convolved to obtain a first convolutional feature map;

[0121] The first convolutional feature maps of each level are weighted and summed according to the preset first fusion coefficient to obtain the first fusion feature map used for target classification.

[0122] In one possible implementation, the feature fusion module 702 is used to perform feature fusion on each of the first feature maps and each of the second fused feature maps based on a preset second fusion coefficient, to obtain a second fused feature map for target regression, including:

[0123] For each first feature map and each second feature map at the same level as each first feature map, the first feature map and the second feature map are convolved to obtain a second convolved feature map;

[0124] The second convolutional feature maps of each level are weighted and summed according to the second fusion coefficient to obtain the second fusion feature map used for target regression.

[0125] In one possible implementation, the feature extraction module 701 is used to extract deep features and shallow features from the template image and the image to be searched, respectively, to obtain a first feature map of multiple levels corresponding to the template image and a second feature map of multiple levels corresponding to the image to be searched, including:

[0126] Deep features and shallow features are extracted from the template image and the image to be searched using a multi-layered convolutional network; at least one of the number of convolutional layers and the number of convolutional kernels in the different layers of the convolutional network is different.

[0127] The outputs of at least two layers of the convolutional network are used as at least two first feature maps corresponding to the template image, and the outputs of at least two layers of the convolutional network are used as at least two second feature maps corresponding to the image to be searched.

[0128] In one possible implementation, the target region determination module 704 is used to determine the target region in the search image from the plurality of candidate regions based on the second scoring parameter and the corresponding first scoring parameter of each pixel in the search image obtained based on quality assessment, including:

[0129] Based on the product of the second scoring parameter and the corresponding first scoring parameter of each pixel in the image to be searched obtained from the quality assessment, the total probability that each pixel in the image to be searched is located in the target region where the target to be tracked is located is determined.

[0130] The region with the highest total probability among the multiple candidate regions is selected as the target region where the target to be tracked is located in the search region.

[0131] In one possible implementation, the classification regression module 703 is used to perform target regression based on the fused feature map to obtain multiple candidate regions where the target to be tracked is located, including:

[0132] For each pixel in the image to be searched, the fused feature map is input into the regression network to obtain a set of candidate distance values ​​corresponding to the pixel. The set of candidate distance values ​​is used to describe the distance between the pixel coordinates and the boundaries of the candidate regions corresponding to the pixel target.

[0133] Based on the set of candidate distance values, the candidate region corresponding to the pixel is determined.

[0134] In one possible implementation, the classification regression module 703 is used to determine the candidate region corresponding to the pixel based on the set of candidate distance values, including:

[0135] The difference between the ordinate of the pixel coordinate and the first distance is used as the ordinate of the first corner position of the candidate region corresponding to the pixel coordinate. The first distance is used to characterize the distance between the pixel coordinate and the first boundary of the corresponding candidate region.

[0136] The difference between the x-coordinate of the pixel coordinate and the second distance is used as the x-coordinate of the first corner position of the candidate region corresponding to the pixel coordinate. The second distance is used to characterize the distance between the pixel coordinate and the second boundary of the corresponding candidate region.

[0137] The sum of the ordinate of the pixel coordinate and the third distance is used as the ordinate of the second corner position of the candidate region corresponding to the pixel coordinate. The third distance is used to characterize the distance between the pixel coordinate and the third boundary of the corresponding candidate region.

[0138] The sum of the x-coordinate of the pixel coordinate and the fourth distance is used as the x-coordinate of the second corner position of the candidate region corresponding to the pixel coordinate. The fourth distance is used to characterize the distance between the pixel coordinate and the fourth boundary of the corresponding candidate region.

[0139] Based on the same inventive concept, this application also provides a target tracking device, the device including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the target tracking methods in the above embodiments.

[0140] The following reference Figure 8 This application describes a target tracking device 80 according to one embodiment of the present application. Figure 8 The target tracking device 80 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0141] like Figure 8 As shown, the target tracking device 80 is presented in the form of a general-purpose target tracking device. The components of the target tracking device 80 may include, but are not limited to: at least one processor 81, at least one memory 82, and a bus 83 connecting different system components (including memory 82 and processor 81).

[0142] The processor 81 is used to read and execute instructions from the memory 82, so that the at least one processor can execute the target tracking method provided in the above embodiments.

[0143] Bus 83 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0144] The memory 82 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 821 and / or cache memory 822, and may further include read-only memory (ROM) 823.

[0145] The memory 82 may also include a program / utility 825 having a set (at least one) of program modules 824, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0146] The target tracking device 80 can also communicate with one or more external devices 84 (e.g., keyboard, pointing device, etc.), one or more devices that enable a user to interact with the target tracking device 80, and / or any device that enables the target tracking device 80 to communicate with one or more other target tracking devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 85. Furthermore, the target tracking device 80 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 86. As shown, network adapter 86 communicates with other modules used in the target tracking device 80 via bus 83. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the target tracking device 80, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0147] In some possible implementations, various aspects of the target tracking method provided in this application can also be implemented in the form of a program product, which includes program code that, when the program product is run on a computer device, causes the computer device to perform the steps of a target tracking method according to various exemplary embodiments of this application as described above.

[0148] In addition, this application also provides a computer-readable storage medium storing a computer program for causing a computer to perform the method described in any of the above embodiments.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0152] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A target tracking method, characterized in that, The method includes: Deep and shallow features are extracted from the template image and the search image corresponding to the current frame, respectively, to obtain a first feature map of multiple levels corresponding to the template image and a second feature map of multiple levels corresponding to the search image. The template image is obtained by extracting the target to be tracked from the reference frame of the video, and the search image is determined based on the region where the target to be tracked was located in the previous frame. Both the first feature map and the second feature map of multiple levels contain shallow feature maps containing pixel information and deep feature maps containing abstract information. Based on the preset fusion coefficient, feature fusion is performed on each first feature map and each second feature map to obtain a first fused feature map and a second fused feature map; The first fused feature map is input into the classification network to obtain the first scoring parameters of each pixel in the image to be searched. During the extraction of deep and shallow features of the template image and the image to be searched corresponding to the current frame, downsampling is performed according to a preset step size. After the first fused feature map is input into the classification network, the first scoring parameter matrix is ​​obtained. The coordinates of each element in the first scoring parameter matrix are restored to the coordinates of the pixels in the image to be searched based on the preset step size. The first scoring parameters of each pixel in the image to be searched are determined based on the first scoring parameter matrix. The second fused feature map is input into the regression network to obtain a set of candidate distance values. Based on the set of candidate distance values, multiple candidate regions for target tracking and the second scoring parameters of each pixel in the image to be searched are determined. Based on the second scoring parameter and the corresponding first scoring parameter, the target region where the target to be tracked is located in the image to be searched is determined from the plurality of candidate regions; the second scoring parameter represents the second probability that the corresponding pixel is located in the target region where the target to be tracked is located.

2. The method according to claim 1, characterized in that, The process of fusing features from each first feature map and each second feature map based on a preset fusion coefficient to obtain a fused feature map includes: Based on a preset first fusion coefficient, feature fusion is performed on each first feature map and each second feature map to obtain a first fused feature map for target classification. Based on a preset second fusion coefficient, feature fusion is performed on each first feature map and each second feature map to obtain a second fused feature map for target regression.

3. The method according to claim 2, characterized in that, The step of fusing features between the first feature maps and the second feature maps based on a preset first fusion coefficient to obtain a first fused feature map for target classification includes: For each first feature map and each second feature map at the same level as each first feature map, the first feature map and the second feature map are convolved to obtain a first convolutional feature map; The first convolutional feature maps of each level are weighted and summed according to the preset first fusion coefficient to obtain the first fusion feature map used for target classification.

4. The method according to claim 2, characterized in that, The step of fusing features between the first feature maps and the second feature maps based on a preset second fusion coefficient to obtain a second fused feature map for target regression includes: For each first feature map and each second feature map at the same level as each first feature map, the first feature map and the second feature map are convolved to obtain a second convolved feature map; The second convolutional feature maps of each level are weighted and summed according to the second fusion coefficient to obtain the second fusion feature map used for target regression.

5. The method according to claim 1, characterized in that, The step of extracting deep and shallow features from the template image and the search image respectively to obtain a first feature map of multiple levels corresponding to the template image and a second feature map of multiple levels corresponding to the search image includes: Deep features and shallow features are extracted from the template image and the image to be searched using a multi-layered convolutional network; at least one of the number of convolutional layers and the number of convolutional kernels is different in the different layers of the convolutional network. The outputs of at least two layers of the convolutional network are used as at least two first feature maps corresponding to the template image, and the outputs of at least two layers of the convolutional network are used as at least two second feature maps corresponding to the image to be searched.

6. The method according to claim 1, characterized in that, Based on the second scoring parameter and the corresponding first scoring parameter, the target region where the target to be tracked is located in the search image is determined from the plurality of candidate regions, including: Based on the product of the second scoring parameter and the corresponding first scoring parameter, the total probability that each pixel in the image to be searched is located in the target region where the target to be tracked is located is determined; The region with the highest total probability among the multiple candidate regions is taken as the target region where the target to be tracked is located.

7. The method according to claim 1, characterized in that, Based on the set of candidate distance values, the candidate region corresponding to the pixel is determined, including: The difference between the ordinate of the pixel and the first distance is used as the ordinate of the first corner position of the candidate region corresponding to the pixel. The first distance is used to characterize the distance between the pixel and the first boundary of the corresponding candidate region. The difference between the x-coordinate of the pixel and the second distance is used as the x-coordinate of the first corner position of the candidate region corresponding to the pixel. The second distance is used to characterize the distance between the pixel and the second boundary of the corresponding candidate region. The sum of the ordinate of the pixel and the third distance is used as the ordinate of the second corner position of the candidate region corresponding to the pixel. The third distance is used to characterize the distance between the pixel and the third boundary of the corresponding candidate region. The sum of the pixel's x-coordinate and the fourth distance is used as the x-coordinate of the second corner position of the candidate region corresponding to the pixel. The fourth distance is used to characterize the distance between the pixel and the fourth boundary of the corresponding candidate region.

8. A target tracking device, characterized in that, The device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-7.

9. A computer storage medium, characterized in that, The computer storage medium stores a computer program that enables the computer to perform the method as described in any one of claims 1-7.

Images