Single-target tracking method and device, electronic equipment and computer readable storage medium

Abstract

Embodiments of the present application provide a single-target tracking method, device, electronic equipment and computer readable storage medium. The method comprises: obtaining a to-be-processed video, the to-be-processed video containing a to-be-tracked target; determining a static template image, a dynamic template image and a search region image according to the to-be-processed video; obtaining a similarity score of the to-be-tracked target and the dynamic template image through a trained similarity comparison network based on the dynamic template image and the search region image; updating the dynamic template according to the numerical value of the similarity score; and tracking the to-be-tracked target based on the static template image, the updated dynamic template image and the search region image. In this way, the problem of tracking failure caused by changes in object shape, size, color, etc. in the prior art can be alleviated, and a scheme for quickly and automatically updating the dynamic template after tracking failure is provided.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of target tracking, and in particular, to a single target tracking method and device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] Real-time tracking of any single object is a basic requirement for many practical applications, such as autonomous driving, monitoring, augmented reality, robots and automatic tracking missiles. Accurate and fast tracking has great significance for many practical applications in civilian or military fields such as autonomous driving and automatic targeting missiles. However, in practical application scenarios, the appearance characteristics of the target, such as shape, size, color and texture, may change during the movement of the target; the target may be occluded by other objects or disappear for a period of time in the video sequence, which will cause the tracker to fail to correctly locate the target position, thereby causing tracking failure; the lighting conditions in the video sequence may change over time and with changes in the scene, which will affect the appearance characteristics of the target, thereby making it difficult for the tracker to accurately track the target; the camera will change the position and scale of the target in the image during movement, thereby making it difficult for the tracker to locate the target position; there will usually be a lot of background interference in the video sequence, such as trees, cars, pedestrians, etc., which will make the tracker easily confused between the target and the background, thereby causing tracking failure; therefore, it is very challenging to accurately track the target at different time points and in different scenes.

[0003] In addition to the above problems, single target tracking needs to be completed under the condition of real-time, which means that the tracker needs to accurately locate and track the target within a short period of time, while also meeting the limitations of computing resources and bandwidth, etc., so the large number of parameters of the model also limits the practical application of single target tracking.

[0004] In summary, single target tracking has the problems of difficulty in obtaining image detail information due to image quality problems and uneven target size, and difficulty in target association due to similar object interference, target dense movement and camera jitter, resulting in tracking failure. SUMMARY

[0005] In a first aspect of the present application, a single target tracking method is provided. The method comprises:

[0006] obtaining a static template image, a dynamic template image and a search region image based on a to-be-processed video, the to-be-processed video containing a to-be-tracked target;

[0007] The static template image is a rectangular area image obtained by expanding the minimum bounding rectangle of the target to be tracked in the first frame of the video to be processed by a first preset expansion coefficient; the dynamic template image is a rectangular area image obtained by expanding the minimum bounding rectangle of the target to be tracked in a preset number of images before the current frame of the video to be processed by a second preset expansion coefficient; the search area image is a rectangular area image obtained by expanding the minimum bounding rectangle of the target to be tracked in the current frame of the video to be processed by a third preset expansion coefficient;

[0008] Calculate the similarity score between the minimum bounding rectangle area image of the target to be tracked in the search area image and each frame image in the dynamic template image;

[0009] If the similarity score between the minimum bounding rectangle area image of the target to be tracked in the search area image and a frame image in the dynamic template image is greater than 0, the minimum bounding rectangle area image of the target to be tracked in the search area image is used to replace the frame image in the dynamic template image to update the dynamic template image;

[0010] Tracking a target based on the static template image, the updated dynamic template image, and the search area image. In one possible implementation, calculating a similarity score between the minimum bounding rectangle area image of the target in the search area image and each frame image in the dynamic template image includes:

[0011] Obtain a first RGB image of three channels of the minimum circumscribed rectangular area image of the target to be tracked in the search area image and a second RGB image of three channels of each frame image in the dynamic template image;

[0012] Superimposing the first RGB image and the second RGB image of each frame along the channel direction to obtain a preset number of six-channel matrices;

[0013] Based on each six-channel matrix, a preset number of vector expansion feature maps are obtained through the trained ResNet18 model;

[0014] Based on the feature map of each vector, the similarity score of the minimum bounding rectangle area image of the target to be tracked in the search area image and each frame image in the dynamic template image is obtained through the trained similarity comparison network.

[0015] In one possible implementation, if the similarity scores between the minimum bounding rectangle area image of the target to be tracked in the search area image and each frame image in the dynamic template image are both less than 0, the minimum bounding rectangle area image of the target to be tracked in the search area image of the frame image in the last replaced dynamic template image is retained;

[0016] Dividing the search area image with a similarity score less than 0 into a plurality of rectangular areas according to the size of the minimum circumscribed rectangular area image of the target to be tracked in the search area image of the frame image in the replaced dynamic template image;

[0017] Calculating a similarity score between each rectangular area and the minimum circumscribed rectangular area image of the target to be tracked in the search area image of the frame image in the replaced dynamic template image;

[0018] If the similarity score between a certain rectangular area and the minimum circumscribed rectangular area image of the target to be tracked in the search area image of the frame image in the replaced dynamic template image is greater than 0, the rectangular area is used as the minimum circumscribed rectangular area image of the target to be tracked in the search area image to replace the frame image in the dynamic template image to update the dynamic template image.

[0019] In one possible implementation, tracking the target to be tracked based on the static template image, the updated dynamic template image, and the search area image includes:

[0020] The static template image, the updated dynamic template image and the search area image are passed through the trained RepVGG-A0 model to obtain the static template image depth feature, the dynamic template image depth feature and the search area image depth feature, wherein the image depth feature is a feature map obtained after a multi-layer neural network;

[0021] According to the depth features of the static template image, the depth features of the dynamic template image, and the depth features of the search area image, an image depth interactive feature is obtained through the trained attention mechanism. The image depth interactive feature is a correlation feature map obtained by cross-attention of the depth features of the static template image, the depth features of the dynamic template image, and the depth features of the search area image;

[0022] Based on the deep interactive features of the image, the coordinate information of the minimum bounding rectangle of the target to be tracked in the search area image is obtained through the trained prediction head network model.

[0023] In one possible implementation, based on the image depth interactive features, the trained prediction head network model is used to obtain the coordinate information of the minimum bounding rectangle of the target to be tracked in the search area image, including:

[0024] Based on the deep interactive features of the image, a two-channel feature map is obtained through the trained prediction head network model;

[0025] Based on the two-channel feature maps, the upper left corner coordinates and the lower right corner coordinates of the minimum bounding rectangle of the target to be tracked are obtained in a cascade manner as the coordinate information.

[0026] In a second aspect of the present application, a single target tracking device is provided. The device comprises:

[0027] An image acquisition module, configured to obtain a static template image, a dynamic template image, and a search area image based on a video to be processed, wherein the video to be processed includes a target to be tracked;

[0028] The static template image is a rectangular area image obtained by expanding the minimum bounding rectangle of the target to be tracked in the first frame of the video to be processed by a first preset expansion coefficient; the dynamic template image is a rectangular area image obtained by expanding the minimum bounding rectangle of the target to be tracked in a preset number of images before the current frame of the video to be processed by a second preset expansion coefficient; the search area image is a rectangular area image obtained by expanding the minimum bounding rectangle of the target to be tracked in the current frame of the video to be processed by a third preset expansion coefficient;

[0029] An image analysis module is used to calculate the similarity score between the minimum bounding rectangle area image of the target to be tracked in the search area image and each frame image in the dynamic template image;

[0030] a template updating module configured to replace a frame image in the dynamic template image with the minimum circumscribed rectangular region image of the target to be tracked in the search region image, so as to update the dynamic template image, if the similarity score between the minimum circumscribed rectangular region image of the target to be tracked in the search region image and a frame image in the dynamic template image is greater than 0;

[0031] The target tracking module tracks the target to be tracked based on the static template image, the updated dynamic template image, and the search area image. In one possible implementation, the image analysis module is specifically used to:

[0032] Obtain a first RGB image of three channels of the minimum circumscribed rectangular area image of the target to be tracked in the search area image and a second RGB image of three channels of each frame image in the dynamic template image;

[0033] Superimposing the first RGB image and the second RGB image of each frame along the channel direction to obtain a preset number of six-channel matrices;

[0034] Based on each six-channel matrix, a preset number of vector expansion feature maps are obtained through the trained ResNet18 model;

[0035] Based on the feature map of each vector, the similarity score of the minimum bounding rectangle area image of the target to be tracked in the search area image and each frame image in the dynamic template image is obtained through the trained similarity comparison network.

[0036] In a possible implementation, the template updating module is further configured to:

[0037] If the similarity scores of the minimum bounding rectangle area image of the target to be tracked in the search area image and each frame image in the dynamic template image are both less than 0, then the minimum bounding rectangle area image of the target to be tracked in the search area image of the frame image in the last replaced dynamic template image is retained;

[0038] Dividing the search area image with a similarity score less than 0 into a plurality of rectangular areas according to the size of the minimum circumscribed rectangular area image of the target to be tracked in the search area image of the frame image in the replaced dynamic template image;

[0039] Calculating a similarity score between each rectangular area and the minimum circumscribed rectangular area image of the target to be tracked in the search area image of the frame image in the replaced dynamic template image;

[0040] If the similarity score between a certain rectangular area and the minimum circumscribed rectangular area image of the target to be tracked in the search area image of the frame image in the replaced dynamic template image is greater than 0, the rectangular area is used as the minimum circumscribed rectangular area image of the target to be tracked in the search area image to replace the frame image in the dynamic template image to update the dynamic template image.

[0041] In a third aspect of the present application, an electronic device is provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the program.

[0042] In a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the method as described above is implemented.

[0043] It should be understood that the contents described in the Summary of the Invention are not intended to limit the key or important features of the embodiments of the present application, nor are they intended to limit the scope of the present application. Other features of the present application will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and other features, advantages and aspects of the embodiments of the present application will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0045] Figure 1 A schematic diagram of a single target tracking method is shown;

[0046] Figure 2A schematic diagram of a model training method for calculating similarity comparison scores is shown;

[0047] Figure 3 A flow chart showing a method for calculating a similarity comparison score is shown;

[0048] Figure 4 A flow chart of a model training method for tracking a target to be tracked is shown;

[0049] Figure 5 A flow chart showing a method for tracking a target to be tracked;

[0050] Figure 6 shows a block diagram of a single target tracking device according to an embodiment of the present application;

[0051] Figure 7 A schematic diagram of the structure of a terminal device or server suitable for implementing an embodiment of the present application is shown. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0053] In order to facilitate the understanding of the embodiments of the present application, some terms involved in the embodiments of the present application are first explained.

[0054] Single-target tracking involves continuously tracking a specific target in a video. The goal is to determine the target's position across different frames of the video. Single-target tracking tasks typically consist of two phases: The first phase is target initialization, which involves determining the target's initial position and size in the first frame of the video sequence. The second phase is target tracking, which involves tracking and locating the target in subsequent frames using information from previous frames.

[0055] The purpose of the present invention is to provide a single target tracking method, device, electronic device and computer-readable storage medium to alleviate the problems existing in the prior art, such as the difficulty of the model in obtaining image detail information due to excessive computational complexity, image quality problems and uneven target size, as well as the difficulty of target association due to interference from similar objects, dense target motion and camera shake, which leads to tracking failure.

[0056] Next, the system architecture involved in the embodiments of the present application is introduced. It should be noted that the system architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. It is known to those skilled in the art that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0057] See also Figure 2 ,The model training method for calculating the similarity comparison score includes the following steps:

[0058] S201, obtain a training data set.

[0059] The training dataset includes videos containing targets and target labels. The targets include airplanes in the sky, cars in motion, pedestrians in a shopping mall, electric vehicles in motion, and dogs running. The target labels include target categories.

[0060] Videos of different scenes, different time periods, and different weather conditions can be collected through trackers, etc. When the videos are collected, they need to be processed to obtain video frames as training images.

[0061] S202: A similarity comparison network model is obtained by training according to the training data set.

[0062] In this application example, we used the Got10k, LaSOT, TrackingNet, and COCO2017 public datasets. These datasets are organized into videos, with each video broken down into multiple images. A folder contains the images from a single video. The datasets encompass a wide range of scenes and objects. Got10k and TrackingNet are short-term datasets, LaSOT is a long-term dataset, and COCO2017 is a detection dataset.

[0063] During training, a minimum bounding rectangular area image including the target to be tracked is randomly selected from a folder and expanded by a preset expansion coefficient. The image obtained by expanding the rectangular area is cropped as the search area image. Then, a minimum bounding rectangular area image including the target to be tracked is randomly selected within 200 frames from the search area image and expanded by a preset expansion coefficient. The image obtained by expanding the rectangular area is cropped as the static template image. Then, a minimum bounding rectangular area image including the target to be tracked is randomly selected within 20 frames from the search area image and expanded by a preset expansion coefficient. The image obtained by expanding the rectangular area is cropped as the dynamic template image.

[0064] Each Epoch randomly takes multiple inputs consisting of search area images, static template images, and dynamic template images.

[0065] The calculation formula for the minimum bounding rectangle image size of the target to be tracked based on the expansion coefficient is as follows:

[0066] X1=x1-b*W

[0067] Y1=y1-b*H

[0068] X2=x2+b*W

[0069] Y2=y2+b*H

[0070] (x1, y1) - the coordinates of the upper left corner of the image of the minimum bounding rectangle of the target to be tracked;

[0071] (x2, y2) - the coordinates of the lower right corner of the image of the minimum bounding rectangle of the target to be tracked;

[0072] W - the minimum circumscribed rectangular area image width of the target to be tracked;

[0073] H - the image height of the minimum circumscribed rectangular area of ​​the target to be tracked;

[0074] (X1, Y1) - the coordinates of the upper left corner of the image of the minimum bounding rectangle of the target to be tracked after expansion;

[0075] (X2, Y2) - the coordinates of the lower right corner of the image of the minimum bounding rectangle of the target to be tracked after expansion;

[0076] b--Expansion coefficient.

[0077] In a preferred embodiment, each epoch randomly selects 60,000 inputs consisting of the search area images, static template images, and dynamic template images. Training is performed for 500 epochs using the Adamw optimizer with a learning rate of 0.0001. The learning rate is reduced to 0.00001 at the 400th epoch.

[0078] See also Figure 3 ,The method for calculating the similarity comparison score includes the following steps:

[0079] S301, obtaining a static template image, a dynamic template image, and a search area image based on a video to be processed, where the video to be processed includes a target to be tracked;

[0080] Among them, the static template image is a rectangular area image obtained by expanding the minimum bounding rectangle of the target to be tracked in the first frame image containing the target to be tracked in the video to be processed by a first preset expansion coefficient; the dynamic template image is a rectangular area image obtained by expanding the minimum bounding rectangle of the target to be tracked in the preset number of images before the current frame containing the target to be tracked in the video to be processed by a second preset expansion coefficient; the search area image is a rectangular area image obtained by expanding the minimum bounding rectangle of the target to be tracked in the current frame image containing the target to be tracked in the video to be processed by a third preset expansion coefficient.

[0081] S302, calculating the similarity score between the minimum bounding rectangle area image of the target to be tracked in the search area image and each frame image in the dynamic template image, specifically including:

[0082] S302a, obtaining a first RGB image of three channels of the minimum circumscribed rectangular area image of the target to be tracked in the search area image and a second RGB image of three channels of each frame image in the dynamic template image;

[0083] S302b, superimposing the first RGB image and each frame of the second RGB image along a channel direction to obtain a preset number of six-channel matrices;

[0084] S302c, based on each six-channel matrix, obtain a preset number of vector expansion feature maps through the trained ResNet18 model;

[0085] In order to reduce the number of model parameters and computational complexity, the number of channels of the ResNet18 network is halved, retaining only three layers: layer1, layer2, and layer3. In addition, to meet the requirements of the number of input channels, the number of model input channels is changed to 6.

[0086] The trimmed model is 360KB, and the number of parameters and computational complexity are negligible.

[0087] The following formula is used to obtain the matrix superimposed into six channels along the channel direction:

[0088] Xinput = cat([img1, img2])

[0089] img1 - the first input image

[0090] img2 - the second input image

[0091] S302d, based on the feature map of each vector, the similarity score between the minimum bounding rectangle area image of the target to be tracked in the search area image and each frame image in the dynamic template image is obtained through the trained similarity comparison network.

[0092] The similarity comparison network formula is as follows:

[0093] feature=Conv(Xinput)

[0094] feature=flatten(feature)

[0095] out=fc(feature)

[0096] feature--extracted features

[0097] out--similarity score

[0098] Update method of dynamic template image:

[0099] On the one hand, if the similarity score between the minimum bounding rectangle area image of the target to be tracked in the search area image and a frame image in the dynamic template image is greater than 0, the minimum bounding rectangle area image of the target to be tracked in the search area image is used to replace the frame image in the dynamic template image to update the dynamic template image;

[0100] Among them, for each frame of tracking results, a minimum bounding rectangle area image of the target to be tracked is cropped, and then the similarity score of each minimum bounding rectangle area image of the target to be tracked and the current dynamic template is calculated respectively. After calculating 20 similarity scores, the target image with the largest similarity score is taken. If the similarity score is greater than 0, the cropped target to be tracked corresponding to this frame is expanded as a new dynamic template.

[0101] The application is as follows:

[0102] target_imgs=[img1, img2,..., img20],

[0103] similarities=net(target_imgs),

[0104] score=max(similaritys),

[0105] if score>0:

[0106] On the other hand, if the similarity scores of the minimum bounding rectangle area image of the target to be tracked in the search area image and each frame image in the dynamic template image are less than 0, then the minimum bounding rectangle area image of the target to be tracked in the search area image of the frame image in the last replaced dynamic template image is retained;

[0107] According to the size of the minimum circumscribed rectangular area image of the target to be tracked in the search area image of the frame image in the replaced dynamic template image, the search area image with a similarity score less than 0 is divided into multiple rectangular areas.

[0108] Applying the above similarity comparison network formula, calculating the similarity score between each of the rectangular areas and the minimum circumscribed rectangular area image of the target to be tracked in the search area image of the frame image in the replaced dynamic template image;

[0109] If the similarity score between a certain rectangular area and the minimum circumscribed rectangular area image of the target to be tracked in the search area image of the frame image in the replaced dynamic template image is greater than 0, then the rectangular area is used as the minimum circumscribed rectangular area image of the target to be tracked in the search area image to replace the frame image in the dynamic template image, and then the dynamic template image should be updated by the above application.

[0110] refer to Figure 4 , the model training method for tracking the target to be tracked:

[0111] S401, obtain a training data set.

[0112] The training dataset includes videos containing targets and target labels. The targets include airplanes in the sky, cars in motion, pedestrians in a shopping mall, electric vehicles in motion, and dogs running. The target labels include target categories.

[0113] Videos of different scenes, different time periods, and different weather conditions can be collected through trackers, etc. When the videos are collected, they need to be processed to obtain video frames as training images.

[0114] S402: Train the RepVGG-A0 model, attention mechanism, and prediction head network model based on the training data set.

[0115] See also Figure 5 , the method for tracking the target to be tracked includes the following steps:

[0116] S501, the static template image, the updated dynamic template image, and the search area image are subjected to the trained RepVGG-A0 model to obtain the depth features of the static template image, the depth features of the dynamic template image, and the depth features of the search area image, wherein the image depth features are feature maps obtained after passing through a multi-layer neural network;

[0117] The deep features of an image refer to the feature maps obtained after passing through multiple layers of neural networks (convolutional layers). In layman's terms, they are the feature maps obtained by a very large (called very deep in artificial intelligence) network. The terms "large" and "small" are relative.

[0118] In the embodiment of the present application, the acquired images are objects that appear in real scenes such as airplanes, cars, pedestrians, electric vehicles, and dogs, and are combined with the RepVGG-A0 model. The image depth feature maps output by the RepVGG-A0 model are recorded as feature0, feature1, and feature2;

[0119] In the RepVGG-A0 model, the specific structure of each ordinary convolution module BaseConv is as follows:

[0120] fout=BaseConv(fin)=Relu(Norm(Conv(fin)));

[0121] Among them, Relu--Relu activation function;

[0122] Norm--batch normalization layer FrozenBatchNorm2d;

[0123] Conv--two-dimensional convolution operation Conv2d;

[0124] S502, obtaining an image depth interactive feature through a trained attention mechanism based on the static template image depth feature, the dynamic template image depth feature, and the search area image depth feature, wherein the image depth interactive feature is a correlation feature map obtained by cross-attention of the static template image depth feature, the dynamic template image depth feature, and the search area image depth feature;

[0125] After the static template image, dynamic template image, and search area image obtain their respective depth features, in order to achieve effective target tracking, it is necessary to cross-attention the depth features of the static template image and the depth features of the search area image. At the same time, the depth features of the dynamic template image and the depth features of the search area image are cross-attentioned to strengthen the features of the effective area and weaken the features of the invalid area, and obtain an enhanced weighted feature map;

[0126] The texture aggregation module is used on the enhanced weighted feature map in a cascade manner to further enhance and aggregate the target details.

[0127] In the embodiment of the present application, the image depth features are decomposed and combined into q, k, and v; q, k, and v are applied to perform a cross-attention operation on the last level of the image depth features to obtain interactive features of the static template image, the dynamic template image, and the search area image:

[0128] Combined with the image depth feature map output by RepVGG-A0, the obtained image depth feature maps feature0, feature1, and feature2 are redistributed using the attention module to obtain feature0', feature1', and feature2'. Through this attention module, interactive features between the static template image depth feature map, the dynamic template image depth feature map, and the search area image depth feature map can be obtained;

[0129] Among them, the calculation method of the cross attention module is as follows:

[0130] feature0'=flatten(feature0),

[0131] feature1'=flatten(feature1),

[0132] feature2'=flatten(feature2),

[0133] q=fc(feature2'),

[0134] k=fc(cat([feature0',feature1'])),

[0135] v=fc(cat([feature0',feature1'])),

[0136] Attention=softmax()v,

[0137] feature0--deep feature map of static template

[0138] feature1--deep feature map of dynamic template

[0139] feature2--deep feature map of search image

[0140] Flatten - Expand a multidimensional array into a vector

[0141] feature0'--deep feature map of the expanded static template

[0142] feature1'--deep feature map of the expanded dynamic template

[0143] feature2'--deep feature map of the expanded search image

[0144] fc--fully connected layer

[0145] cat--connect vector

[0146] Softmax--normalized exponential function

[0147] Attention--attention feature map

[0148] Among them, the steps of obtaining the deep interactive features of the processed image are:

[0149] S502a, using a multi-head cross attention mechanism to perform cross attention on the image depth features, strengthen the features of the valid area, weaken the features of the invalid area, and obtain an enhanced weighted feature map;

[0150] S502b, based on the enhanced weighted feature map, a texture aggregation module is used in a cascade manner to further enhance and aggregate target details.

[0151] S503, based on the image depth interactive feature, obtain the coordinate information of the minimum bounding rectangle of the target to be tracked in the search area image through the trained prediction head network model;

[0152] S503a, based on the image depth interactive features, a two-channel feature map is obtained through the trained prediction head network model;

[0153] S503b: Based on the two-channel feature maps, the coordinates of the upper left corner and the lower right corner of the minimum bounding rectangle of the target to be tracked are obtained in a cascade manner as the coordinate information.

[0154] In one possible implementation, the image depth feature maps feature0, feature1, and feature2 are subjected to a cross-attention module to obtain interactive features among the static template image depth features, the dynamic template image depth features, and the search image depth features. The interactive features are then subjected to feature signal conversion in a cascade manner to obtain a two-channel feature map and the coordinates of the upper left corner and the lower right corner of the target to be tracked.

[0155] wherein, the two-channel feature maps are expanded to obtain two vectors, a softmax operation is performed on the vectors respectively, and then the mathematical expectations are calculated respectively;

[0156] Among them, the mathematical expectation is the coordinate value of the tracking target;

[0157] Among them, when calculating the mathematical expectation, the random variable is fixed from 8 to 312, and a 20*20 matrix is ​​obtained by repeating 20 numbers with an interval of 16 20 times, and the feature map obtained by the prediction head network is used as the probability value.

[0158] Among them, the calculation method of characteristic signal conversion is as follows:

[0159] P1'~Pn'=Softmax(P1~Pn)

[0160] x,y=X1*P1+X2*P2+...+Xn*Pn

[0161] X1~Xn--fixed number

[0162] P1~Pn--The corresponding values ​​in the feature map output by the prediction head network

[0163] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all optional embodiments, and the actions and modules involved are not necessarily required by this application.

[0164] The above is an introduction to the method embodiment. The following is a device embodiment to further illustrate the solution described in this application.

[0165] Figure 6 FIG. 5 shows a block diagram of a single target tracking device 500 according to an embodiment of the present application. Figure 6 As shown, the apparatus 600 includes:

[0166] An image acquisition module 601 is configured to obtain a static template image, a dynamic template image, and a search area image based on a video to be processed, wherein the video to be processed includes a target to be tracked;

[0167] The static template image is a rectangular area image obtained by expanding the minimum bounding rectangle of the target to be tracked in the first frame of the video to be processed by a first preset expansion coefficient; the dynamic template image is a rectangular area image obtained by expanding the minimum bounding rectangle of the target to be tracked in a preset number of images before the current frame of the video to be processed by a second preset expansion coefficient; the search area image is a rectangular area image obtained by expanding the minimum bounding rectangle of the target to be tracked in the current frame of the video to be processed by a third preset expansion coefficient;

[0168] An image analysis module 602 is configured to calculate a similarity score between the minimum bounding rectangle area image of the target to be tracked in the search area image and each frame image in the dynamic template image;

[0169] a template updating module 603 configured to replace a frame image in the dynamic template image with the minimum bounding rectangle region image of the target to be tracked in the search region image, so as to update the dynamic template image, if the similarity score between the minimum bounding rectangle region image of the target to be tracked in the search region image and a frame image in the dynamic template image is greater than 0;

[0170] The target tracking module 604 tracks the target to be tracked based on the static template image, the updated dynamic template image and the search area image.

[0171] The image analysis module is specifically used for:

[0172] Obtain a first RGB image of three channels of the minimum circumscribed rectangular area image of the target to be tracked in the search area image and a second RGB image of three channels of each frame image in the dynamic template image;

[0173] Superimposing the first RGB image and the second RGB image of each frame along the channel direction to obtain a preset number of six-channel matrices;

[0174] Based on each six-channel matrix, a preset number of vector expansion feature maps are obtained through the trained ResNet18 model;

[0175] Based on the feature map of each vector, the similarity score of the minimum bounding rectangle area image of the target to be tracked in the search area image and each frame image in the dynamic template image is obtained through the trained similarity comparison network.

[0176] The template update module is also used to:

[0177] If the similarity scores of the minimum bounding rectangle area image of the target to be tracked in the search area image and each frame image in the dynamic template image are both less than 0, then the minimum bounding rectangle area image of the target to be tracked in the search area image of the frame image in the last replaced dynamic template image is retained;

[0178] According to the size of the minimum circumscribed rectangular area image of the target to be tracked in the search area image of the frame image in the replaced dynamic template image, the search area image with a similarity score less than 0 is divided into multiple rectangular areas.

[0179] Calculating a similarity score between each rectangular area and the minimum circumscribed rectangular area image of the target to be tracked in the search area image of the frame image in the replaced dynamic template image;

[0180] If the similarity score between a certain rectangular area and the minimum bounding rectangular area image of the target to be tracked in the search area image of the frame image in the replaced dynamic template image is greater than 0, the frame image in the dynamic template image is replaced with the rectangular area as the minimum bounding rectangular area image of the target to be tracked in the search area image, thereby updating the dynamic template image. Those skilled in the art will clearly understand that for ease and brevity of description, the specific working process of the described modules can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0181] An electronic device is provided in an embodiment of the present application, such as Figure 7 As shown, Figure 7 The electronic device 700 shown includes a processor 701 and a memory 703. The processor 701 and the memory 703 are connected, for example, via a bus 7002. Optionally, the electronic device 700 may further include a transceiver 704. It should be noted that in actual applications, the number of transceivers 704 is not limited to one, and the structure of the electronic device 700 does not constitute a limitation on the embodiments of the present application.

[0182] The processor 701 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor 701 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0183] The bus 702 may include a path for transmitting information between the above components. The bus 702 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 702 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0184] The memory 703 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0185] The memory 703 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 701. The processor 701 is used to execute the application code stored in the memory 703 to implement the content shown in the above method embodiment.

[0186] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0187] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding contents of the aforementioned method embodiment.

[0188] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0189] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A single target tracking method, characterized in that: include: Obtaining a static template image, a dynamic template image, and a search area image based on a video to be processed, wherein the video to be processed includes a target to be tracked; The static template image is a rectangular area image obtained by expanding the minimum bounding rectangle of the target to be tracked in the first frame of the video to be processed by a first preset expansion coefficient; the dynamic template image is a rectangular area image obtained by expanding the minimum bounding rectangle of the target to be tracked in a preset number of images before the current frame of the video to be processed by a second preset expansion coefficient; the search area image is a rectangular area image obtained by expanding the minimum bounding rectangle of the target to be tracked in the current frame of the video to be processed by a third preset expansion coefficient; Calculate the similarity score between the minimum bounding rectangle area image of the target to be tracked in the search area image and each frame image in the dynamic template image; If the similarity score between the minimum bounding rectangle area image of the target to be tracked in the search area image and a frame image in the dynamic template image is greater than 0, the minimum bounding rectangle area image of the target to be tracked in the search area image is used to replace the frame image in the dynamic template image to update the dynamic template image; The target to be tracked is tracked based on the static template image, the updated dynamic template image, and the search area image.

2. The method according to claim 1, characterized in that The calculating of the similarity score between the minimum bounding rectangle area image of the target to be tracked in the search area image and each frame image in the dynamic template image includes: Obtain a first RGB image of three channels of the minimum circumscribed rectangular area image of the target to be tracked in the search area image and a second RGB image of three channels of each frame image in the dynamic template image; Superimposing the first RGB image and the second RGB image of each frame along the channel direction to obtain a preset number of six-channel matrices; Based on each six-channel matrix, a preset number of vector expansion feature maps are obtained through the trained ResNet18 model; Based on the feature map of each vector, the similarity score of the minimum bounding rectangle area image of the target to be tracked in the search area image and each frame image in the dynamic template image is obtained through the trained similarity comparison network.

3. The method according to claim 1, characterized in that Also includes: If the similarity scores of the minimum bounding rectangle area image of the target to be tracked in the search area image and each frame image in the dynamic template image are both less than 0, then the minimum bounding rectangle area image of the target to be tracked in the search area image of the frame image in the last replaced dynamic template image is retained; Dividing the search area image with a similarity score less than 0 into a plurality of rectangular areas according to the size of the minimum circumscribed rectangular area image of the target to be tracked in the search area image of the frame image in the replaced dynamic template image; Calculating a similarity score between each rectangular area and the minimum circumscribed rectangular area image of the target to be tracked in the search area image of the frame image in the replaced dynamic template image; If the similarity score between a certain rectangular area and the minimum circumscribed rectangular area image of the target to be tracked in the search area image of the frame image in the replaced dynamic template image is greater than 0, the rectangular area is used as the minimum circumscribed rectangular area image of the target to be tracked in the search area image to replace the frame image in the dynamic template image to update the dynamic template image.

4. The method according to claim 1, wherein Tracking the target to be tracked based on the static template image, the updated dynamic template image, and the search area image includes: The static template image, the updated dynamic template image and the search area image are passed through the trained RepVGG-A0 model to obtain the static template image depth feature, the dynamic template image depth feature and the search area image depth feature, wherein the image depth feature is a feature map obtained after a multi-layer neural network; According to the depth features of the static template image, the depth features of the dynamic template image, and the depth features of the search area image, an image depth interactive feature is obtained through the trained attention mechanism. The image depth interactive feature is a correlation feature map obtained by cross-attention of the depth features of the static template image, the depth features of the dynamic template image, and the depth features of the search area image; Based on the deep interactive features of the image, the coordinate information of the minimum bounding rectangle of the target to be tracked in the search area image is obtained through the trained prediction head network model.

5. The method according to claim 4, characterized in that Based on the deep interactive features of the image, the trained prediction head network model is used to obtain the coordinate information of the minimum bounding rectangle of the target to be tracked in the search area image, including: Based on the deep interactive features of the image, a two-channel feature map is obtained through the trained prediction head network model; Based on the two-channel feature maps, the upper left corner coordinates and the lower right corner coordinates of the minimum bounding rectangle of the target to be tracked are obtained in a cascade manner as the coordinate information.

6. A single target tracking device, characterized in that: include: An image acquisition module, configured to obtain a static template image, a dynamic template image, and a search area image based on a video to be processed, wherein the video to be processed includes a target to be tracked; The static template image is a rectangular area image obtained by expanding the minimum bounding rectangle of the target to be tracked in the first frame of the video to be processed by a first preset expansion coefficient; the dynamic template image is a rectangular area image obtained by expanding the minimum bounding rectangle of the target to be tracked in a preset number of images before the current frame of the video to be processed by a second preset expansion coefficient; the search area image is a rectangular area image obtained by expanding the minimum bounding rectangle of the target to be tracked in the current frame of the video to be processed by a third preset expansion coefficient; An image analysis module is used to calculate the similarity score between the minimum bounding rectangle area image of the target to be tracked in the search area image and each frame image in the dynamic template image; a template updating module configured to replace a frame image in the dynamic template image with the minimum circumscribed rectangular region image of the target to be tracked in the search region image, so as to update the dynamic template image, if the similarity score between the minimum circumscribed rectangular region image of the target to be tracked in the search region image and a frame image in the dynamic template image is greater than 0; The target tracking module tracks the target to be tracked based on the static template image, the updated dynamic template image and the search area image.

7. The device according to claim 6, characterized in that The image analysis module is specifically used for: Obtain a first RGB image of three channels of the minimum circumscribed rectangular area image of the target to be tracked in the search area image and a second RGB image of three channels of each frame image in the dynamic template image; Superimposing the first RGB image and the second RGB image of each frame along the channel direction to obtain a preset number of six-channel matrices; Based on each six-channel matrix, a preset number of vector expansion feature maps are obtained through the trained ResNet18 model; Based on the feature map of each vector, the similarity score of the minimum bounding rectangle area image of the target to be tracked in the search area image and each frame image in the dynamic template image is obtained through the trained similarity comparison network.

8. The device according to claim 6, characterized in that The template update module is also used to: If the similarity scores of the minimum bounding rectangle area image of the target to be tracked in the search area image and each frame image in the dynamic template image are both less than 0, then the minimum bounding rectangle area image of the target to be tracked in the search area image of the frame image in the last replaced dynamic template image is retained; Dividing the search area image with a similarity score less than 0 into a plurality of rectangular areas according to the size of the minimum circumscribed rectangular area image of the target to be tracked in the search area image of the frame image in the replaced dynamic template image; Calculating a similarity score between each rectangular area and the minimum circumscribed rectangular area image of the target to be tracked in the search area image of the frame image in the replaced dynamic template image; If the similarity score between a certain rectangular area and the minimum circumscribed rectangular area image of the target to be tracked in the search area image of the frame image in the replaced dynamic template image is greater than 0, the rectangular area is used as the minimum circumscribed rectangular area image of the target to be tracked in the search area image to replace the frame image in the dynamic template image to update the dynamic template image.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images