Target tracking method and device, electronic device and storage medium

Abstract

The present invention provides an image-based target tracking method, a target tracking device, an electronic device, and a non-volatile computer-readable storage medium. The method includes convolving a template image and a search image using a preset target tracking model to extract a first feature of the template image and a second feature of the search image, wherein the search image and the template image contain the same target object; performing linear operations on the first and second features to obtain third and fourth features, respectively; generating a first feature vector based on the first and third features, and generating a second feature vector based on the second and fourth features; and outputting target tracking information corresponding to the search image based on the first and second feature vectors. By performing linear operations on the features extracted by convolution, more features can be obtained and the accuracy of the extracted features can be improved, thereby ensuring the accuracy of the target tracking information.

Description

Technical Field

[0001] The present invention relates to the field of target tracking technology, and in particular to an image-based target tracking method, a target tracking device, an electronic device, and a non-volatile computer-readable storage medium. Background Art

[0002] Currently, target tracking requires the target in the initial frame of a video as a template input, and subsequent frames as search input. Strategies such as deep learning are then used to estimate the target's position. However, deep learning models typically perform convolution calculations to extract features to ensure accuracy. This limits the number of features that can be extracted, thus affecting the accuracy of subsequent target tracking. Summary of the Invention

[0003] Embodiments of the present invention provide an image-based target tracking method, an image-based target tracking device, an electronic device, and a non-volatile computer-readable storage medium.

[0004] The target tracking method of an embodiment of the present invention includes convolving a template image and a search image respectively through a preset target tracking model to extract a first feature of the template image and a second feature of the search image, wherein the search image and the template image contain the same target object; performing linear operations on the first feature and the second feature respectively to obtain a third feature and a fourth feature; generating a first feature vector based on the first feature and the third feature, and generating a second feature vector based on the second feature and the fourth feature; and outputting target tracking information corresponding to the search image based on the first feature vector and the second feature vector.

[0005] In some embodiments, the target tracking method also includes convolving the updated image through the target tracking model to extract the fifth feature of the updated image, and performing a linear operation on the fifth feature to obtain a sixth feature, the continuous multi-frame images include the template image and the search image, the shooting time of the template image is earlier than the shooting time of the search image, and the updated image is any frame image between the template image and the search image in the continuous multi-frame images; generating a third feature vector based on the fifth feature and the sixth feature; outputting target tracking information based on the first feature vector and the second feature vector includes: weighted fusion of the first feature vector and the third feature vector through the preset weight of the first feature vector and the preset weight of the third feature vector to generate a fourth feature vector; outputting the target tracking information based on the fourth feature vector and the second feature vector.

[0006] In some embodiments, based on a preset anchor-free target detection algorithm, the fourth feature vector and the second feature vector are processed and split separately to obtain multiple feature vectors after splitting; the multiple feature vectors after splitting are fused to obtain multiple fused feature vectors; and the target tracking information is output based on the multiple fused feature vectors.

[0007] In some embodiments, the target tracking information includes the type of the target object, the center position of the target object, and the distance between the center position and the target frame. The preset anchor-free target detection algorithm processes the fourth eigenvector and the second eigenvector to output the target tracking information, including: splitting the fourth eigenvector to obtain a fifth eigenvector and a sixth eigenvector, and splitting the second eigenvector to obtain a seventh eigenvector and an eighth eigenvector, the fifth eigenvector and the seventh eigenvector both including information for detecting the classification and the center position, the sixth eigenvector and the eighth eigenvector both including information for detecting the distance between the center position and the target frame; generating a ninth eigenvector based on the fifth eigenvector and the seventh eigenvector, and generating a tenth eigenvector based on the sixth eigenvector and the eighth eigenvector; outputting the type of the target object and the center position of the target object based on the ninth eigenvector; and outputting the distance between the center position and the target frame based on the tenth eigenvector.

[0008] In some embodiments, the target tracking method also includes obtaining a preset target tracking model; obtaining the preset target tracking model includes: obtaining a training set, the training set including multiple image groups, each image group including a training template image and a training search image, the training set also including label information corresponding one-to-one to the image groups, the training search image and the training template image containing the same target object; convolving the training template image and the training search image through a tracking model to extract a first feature of the training template image and a second feature of the training search image; performing linear operations on the first feature and the second feature to obtain a third feature and a fourth feature, respectively; generating a first feature vector based on the first feature and the third feature, and generating a second feature vector based on the second feature and the fourth feature; and outputting target tracking information based on the first feature vector and the second feature vector; calculating a loss value based on the target tracking information and the label information based on a preset loss function; adjusting the tracking model according to the loss value until the tracking model converges to obtain the target tracking model.

[0009] In some embodiments, the image group also includes a training update image, the continuous multi-frame image includes the training template image and the training search image, the shooting time of the training template image is earlier than the shooting time of the training search image, and the training update image is any frame image between the training template image and the training search image in the continuous multi-frame image, and the target tracking method also includes: convolving the training update image through a tracking model to extract a fifth feature of the training update image, and performing a linear operation on the fifth feature to obtain a sixth feature; generating a third feature vector based on the fifth feature and the sixth feature; outputting target tracking information based on the first feature vector and the second feature vector includes: weightedly fusing the first feature vector and the third feature vector through a preset weight to generate a fourth feature vector; outputting the target tracking information based on the fourth feature vector and the second feature vector; adjusting the tracking model according to the loss value until the tracking model converges to obtain the target tracking model, includes: adjusting the preset weights and preset model parameters according to the loss value until the tracking model converges to obtain the target tracking model.

[0010] In some embodiments, the training set includes positive samples and negative samples, and in the image group corresponding to the positive samples, the target object contained in the template image is the same as the target object contained in the search image; in the image group corresponding to the negative samples, the target object contained in the template image is different from the target object contained in the search image.

[0011] The image-based target tracking device of an embodiment of the present invention includes a feature extraction module and a post-processing module. The feature extraction module is used to convolve a template image and a search image respectively using a preset target tracking model to extract a first feature of the template image and a second feature of the search image, wherein the search image and the template image contain the same target object; perform linear operations on the first feature and the second feature respectively to obtain a third feature and a fourth feature; generate a first feature vector based on the first feature and the third feature, and generate a second feature vector based on the second feature and the fourth feature; and the post-processing module is used to output target tracking information corresponding to the search image based on the first feature vector and the second feature vector.

[0012] An electronic device according to an embodiment of the present invention includes a processor configured to execute the target tracking method according to any one of the above embodiments.

[0013] An embodiment of the present invention provides a non-volatile computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the target tracking method of any of the above embodiments is implemented.

[0014] The target tracking method, target tracking device, electronic device and non-volatile computer-readable storage medium of the present invention respectively convolute the template image and the search image through the target tracking model to obtain the first feature and the second feature, and then perform linear calculation on the first feature and the second feature to obtain the third feature and the fourth feature, thereby obtaining more features and improving the accuracy of the extracted features. Thereafter, a first feature vector is generated based on the first feature and the third feature, and a second feature vector is generated based on the second feature and the fourth feature, thereby outputting target tracking information based on the first feature vector and the second feature vector, which can ensure the accuracy of the target tracking information. In addition, since the extracted features are linearly calculated, the first feature and the second feature that the target tracking model needs to extract are reduced. Therefore, the target tracking model only needs to use fewer convolution kernels for feature extraction, and the computing power consumed during feature extraction is less, so that the target tracking model can be applied to devices with less computing power.

[0015] Additional aspects and advantages of the embodiments of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments with reference to the following drawings, in which:

[0017] Figure 1 is a flowchart of a target tracking method according to certain embodiments of the present invention;

[0018] Figure 2 is a schematic diagram of the principle of a target tracking method according to certain embodiments of the present invention;

[0019] Figure 3 is a schematic diagram of feature extraction of a convolutional neural network according to certain embodiments of the present invention;

[0020] Figure 4 is a schematic diagram of feature extraction of a template tracking model according to certain embodiments of the present invention;

[0021] Figure 5 is a flowchart of a target tracking method according to certain embodiments of the present invention;

[0022] Figure 6 is a flowchart of a target tracking method according to certain embodiments of the present invention;

[0023] Figure 7 is a flowchart of a target tracking method according to certain embodiments of the present invention;

[0024] Figure 8 is a flowchart of a target tracking method according to certain embodiments of the present invention;

[0025] Figure 9 is a flowchart of a target tracking method according to certain embodiments of the present invention;

[0026] Figure 10 is a flowchart of a target tracking method according to certain embodiments of the present invention;

[0027] Figure 11 is a schematic diagram of a module of an image-based target tracking device according to certain embodiments of the present invention;

[0028] Figure 12 is a schematic plan view of an electronic device according to some embodiments of the present invention; and

[0029] Figure 13 The figure is a schematic diagram of the interaction between a non-volatile computer-readable storage medium and a processor according to certain embodiments of the present invention. DETAILED DESCRIPTION

[0030] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present invention, and should not be understood as limiting the embodiments of the present invention.

[0031] First, the nouns appearing in the present invention are explained below:

[0032] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0033] Deep learning (DL): A branch of machine learning, it is an algorithm that attempts to achieve high-level abstraction of data using multiple processing layers with complex structures or multiple nonlinear transformations. Deep learning learns the inherent patterns and representational hierarchies of training sample data. The information gained from this learning process is highly helpful in interpreting data such as text, images, and sound. The ultimate goal of deep learning is to enable machines to acquire human-like analytical learning capabilities and recognize data such as text, images, and sound. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition that far surpass previous related technologies.

[0034] Siamese Network: For two inputs, input1 and input2, they enter neural networks network1 and network2, respectively. The final loss calculation evaluates the similarity of the two network's posterior vectors, that is, the similarity of the two inputs. Network1 and network2 can be the same, such as both using convolutional neural networks (CNNs). They can also be different networks, such as one using a CNN and the other using a long short-term memory (LSTM), with weights shared between network1 and network2.

[0035] Convolutional neural networks (CNNs) are a type of feedforward neural network (FNNs) with a deep structure that incorporates convolutional computations. They are a representative algorithm for deep learning. Convolutional neural networks possess the ability to learn representations and perform translation-invariant classification of input information based on their hierarchical structure, hence the term "translation-invariant artificial neural network."

[0036] See also Figure 1 and Figure 2 , the image-based target tracking method according to an embodiment of the present invention includes:

[0037] Step 011: Convolve the template image and the search image respectively through a preset target tracking model to extract a first feature of the template image and a second feature of the search image, wherein the search image and the template image contain the same target object.

[0038] Specifically, when tracking a target, feature extraction must be performed on both the template image and the search image, both of which contain the same target object. Generally, the search image is larger than the template image to prevent the search image from omitting the target object or only containing a portion of it, which would affect tracking performance.

[0039] For example, the template image is typically the first frame of a continuous multi-frame image collection. Using a preset target recognition algorithm, the target object in the first frame can be identified, and then an image region containing the target object (e.g., an image region twice the width and height of the target object) is captured to serve as the template image. When tracking the target object in subsequent frames, a search image is determined. Based on the position of the target object in the template image, the approximate position of the target object in the search image is determined. A larger search image is then captured (e.g., an image region four times the width and height of the target object is captured as the search image), thereby tracking the target object in the search image.

[0040] By convolving the template image and the search image with the target tracking model, the first feature of the template image and the second feature of the search image can be extracted. The target tracking model of the present invention is a twin network, which is composed of two branch networks (such as Figure 2 The net1 and net2 in

[15] extract the first feature of the template image and the second feature of the search image respectively.

[0041] Step 012: Perform a linear operation on the first feature and the second feature to obtain the third feature and the fourth feature respectively.

[0042] A normal convolution kernel in a deep neural network is split into two parts. The first part involves normal convolutions, but the total number of them is strictly controlled. Given the intrinsic features of the first part, a series of simple linear operations are then applied to generate more features.

[0043] See also Figure 3 , respectively, are the convolution process of conventional CNN and the convolution process of the backbone branch (backbone) of the target tracking model of the present invention. Conventional CNN will use all convolution kernels to perform convolution to obtain the first eigenvector and the second eigenvector. Figure 3 In , the features of the template image M1 are convolved to generate a first feature vector P1, and the search image M2 is convolved to generate a second feature vector P2.

[0044] See also Figure 4The target tracking model of the present invention first performs a normal convolution with half the convolution kernel to extract a feature (such as the first or second feature). This feature is then subjected to a linear operation to obtain a new feature (such as the third or fourth feature). Finally, the two parts are combined to obtain the first or second eigenvector. The linear operation can be a 3x3 or 5x5 depth-level convolution process, which is computationally efficient.

[0045] like Figure 4 In the method, the template image P1 is convolved to obtain the first feature, and then the first feature is linearly processed to obtain the third feature, and then the first feature and the third feature are synthesized to generate the first feature vector P1; the template image P2 is convolved to obtain the second feature, and then the second feature is linearly processed to obtain the fourth feature, and then the second feature and the fourth feature are synthesized to generate the second feature vector P2.

[0046] In this way, the number of convolution kernels required for feature extraction is reduced by half, which can reduce the amount of convolution calculation without changing the size of the final output feature.

[0047] After extracting the first feature and the second feature through the target tracking model, a linear operation is performed on the first feature to obtain the third feature, and then a linear operation is performed on the second feature to obtain the third feature.

[0048] Step 013: Generate a first feature vector based on the first feature and the third feature, and generate a second feature vector based on the second feature and the fourth feature.

[0049] Specifically, after obtaining the first, second, third, and fourth features, feature merging can be performed to ensure that the size of the output features is substantially the same as the size of the features extracted by conventional CNN. The first feature and the third feature generated by the linear operation of the first feature are merged to obtain the first feature vector, and the second feature and the fourth feature generated by the linear operation of the second feature are merged to obtain the second feature vector.

[0050] Step 014: Output target tracking information corresponding to the search image based on the first eigenvector and the second eigenvector.

[0051] Specifically, after completing feature extraction to obtain a first eigenvector and a second eigenvector with higher precision, the target tracking information corresponding to the search image can be output based on the first eigenvector and the second eigenvector. For example, the first eigenvector and the second eigenvector are input into a preset target tracking algorithm (such as an anchor free algorithm) to output the position of the target object.

[0052] The target tracking method of the present invention uses a target tracking model to convolve the template image and the search image respectively to obtain a first feature and a second feature, and then linearly calculates the first feature and the second feature to obtain a third feature and a fourth feature, thereby obtaining more features and improving the accuracy of the extracted features. After that, a first feature vector is generated based on the first feature and the third feature, and a second feature vector is generated based on the second feature and the fourth feature, thereby outputting target tracking information based on the first feature vector and the second feature vector, which can ensure the accuracy of the target tracking information. In addition, since the extracted features are linearly calculated, the first feature and the second feature that the target tracking model needs to extract are reduced. Therefore, the target tracking model only needs to use fewer convolution kernels for feature extraction, and the computing power consumed when performing feature extraction is less, so that the target tracking model can be applied to devices with less computing power (such as embedded devices).

[0053] See also Figure 5 , the target tracking method also includes:

[0054] Step 015: Convolving the update image with multiple convolution kernels to extract a fifth feature of the update image, and performing a linear operation on the fifth feature to obtain a sixth feature, wherein the continuous multi-frame image includes a template image and a search image, the template image is captured earlier than the search image, and the update image is any frame image between the template image and the search image in the continuous multi-frame image;

[0055] Step 016: Generate a third eigenvector based on the fifth and sixth features.

[0056] Specifically, because the template image is relatively fixed, the target object being tracked may change significantly during the target tracking process. Continuing to use the template image for target tracking will affect the accuracy of target tracking. Therefore, the target tracking model of the present invention can add a branch network to extract features from the updated image.

[0057] The update image can be any frame between the template image (e.g., the first frame in the collected continuous multi-frame image) and the search image. If the search image is the search image of the current frame, the update image is generated based on any frame between the template image (e.g., the first frame in the collected continuous multi-frame image) and the current frame. If the previous frame of the search image is used as the update image, the update image can be generated by cropping the search image corresponding to the previous frame to capture an image area twice the width and height of the target object.

[0058] The changes in the target object in the updated image at this time (such as light, shape changes, etc.) are basically the same as the changes in the target object in the current frame. By extracting features from the updated image and combining them with the features extracted from the updated image, feature tracking can be performed more accurately.

[0059] In this way, the updated image is convolved with the target tracking model to obtain the fifth feature of the updated image, and the fifth feature is linearly processed to obtain the sixth feature. Finally, the fifth and sixth features are combined to obtain the third feature vector corresponding to the updated image.

[0060] Please refer again Figure 5 , step 014 includes:

[0061] Step 0141: weightedly fusing the first eigenvector and the third eigenvector using the preset weight of the first eigenvector and the preset weight of the third eigenvector to generate a fourth eigenvector;

[0062] Step 0142: Output target tracking information based on the fourth eigenvector and the second eigenvector.

[0063] Specifically, please combine Figure 2 When combined with the third eigenvector P3 extracted from the updated image for target tracking, the features of the template image can also improve target tracking accuracy. Therefore, the first eigenvector P1 extracted from the template image and the third eigenvector P3 extracted from the updated image can be fused. For example, the first eigenvector P1 and the third eigenvector P3 are weightedly fused using the preset weights of the first and third eigenvectors to generate a fourth eigenvector P4. Target tracking information can be output based on the fused fourth eigenvector P4 and the second eigenvector P2.

[0064] For example, the preset weight can be determined based on the number of image frames between the updated image and the template image. The larger the number of frames, the farther the acquisition time of the updated image is from the acquisition time of the template image, and the higher the probability that the target object has changed. At this time, the updated image should be given a larger weight. That is to say, the larger the number of image frames between the updated image and the template image, the greater the weight of the third eigenvector P3 corresponding to the updated image.

[0065] In this way, target tracking is performed by combining the first eigenvector P1 of the template image and the third eigenvector P3 of the updated image in a weighted fusion manner. Even if the target object changes, it can be tracked accurately, thereby improving the accuracy of target tracking.

[0066] See also Figure 6 The target tracking information includes the type of the target object, the center position of the target object, and the distance between the center position and the target frame. Step 0142 includes:

[0067] Step 01421: Based on the preset anchor-free target detection algorithm, the type of the target object and the center position of the target object are output according to the part of the fourth eigenvector used to detect the classification and the center position and the part of the second eigenvector used to detect the classification and the center position, and the distance between the center position and the target frame is output according to the part of the fourth eigenvector used to detect the distance between the center position and the target frame and the part of the second eigenvector used to detect the distance between the center position and the target frame.

[0068] Specifically, the current twin networks all track target objects based on anchor frames. For example, the anchor-based algorithm generates prediction frames based on several preset anchor frames of fixed scales and aspect ratios. When the data sets (training sets for training) are different, the hyperparameters of the model need to be redesigned, and the universality is poor. The present invention adopts an anchor-free target detection algorithm (such as an anchor-free algorithm), which can generate prediction frames based on points, thereby realizing anchor-free target tracking. It can adapt to the same hyperparameters for different data sets without redesign, and has strong universality.

[0069] Target tracking information includes the target object type, center position, and distance from the center position to the target bounding box. The fused fourth eigenvector contains the components used for classification and center position detection, as well as the distance detection from the center position to the target bounding box. Similarly, the second eigenvector also contains the components used for classification and center position detection, as well as the distance detection from the center position to the target bounding box.

[0070] Therefore, based on the anchor-free target detection algorithm, the type of the target object and the center position of the target object can be output through the part of the fourth eigenvector used to detect the classification and center position and the part of the second eigenvector used to detect the classification and center position; the distance between the center position and the target frame can be output through the part of the fourth eigenvector used to detect the distance between the center position and the target frame and the part of the second eigenvector used to detect the distance between the center position and the target frame.

[0071] See also Figure 7 , step 01421 includes:

[0072] Step 01422: Split the fourth eigenvector to obtain the fifth eigenvector and the sixth eigenvector, and split the second eigenvector to obtain the seventh eigenvector and the eighth eigenvector. The fifth eigenvector and the seventh eigenvector both include information for detecting classification and center position, and the sixth eigenvector and the eighth eigenvector both include information for detecting the center position and the distance detection of the target frame.

[0073] Step 01423: Generate a ninth eigenvector based on the fifth eigenvector and the seventh eigenvector, and generate a tenth eigenvector based on the sixth eigenvector and the eighth eigenvector;

[0074] Step 01424: Output the type of the target object and the center position of the target object according to the ninth eigenvector; and

[0075] Step 01425: Based on the tenth eigenvector, output the distance between the center position and the target box.

[0076] Specifically,

[0077] Please combine Figure 2 , through feature splitting, split the fourth eigenvector P4 to obtain the fifth eigenvector P5 for classification and center position detection and the sixth eigenvector P6 for detecting the distance between the center position and the target frame, and split the second eigenvector P2 to obtain the seventh eigenvector P7 for classification and center position detection and the eighth eigenvector P8 for detecting the distance between the center position and the target frame. In other words, the fifth eigenvector and the seventh eigenvector both include information for detecting classification and center position, and the sixth eigenvector and the eighth eigenvector both include information for detecting the distance between the center position and the target frame.

[0078] Then, the fifth eigenvector P5 and the seventh eigenvector P7 used for classification and center position detection are cross-correlated to generate the ninth eigenvector P9; the sixth eigenvector P6 and the eighth eigenvector P8 used for distance detection between the center position and the target frame are cross-correlated to generate the tenth eigenvector P10.

[0079] Finally, the preset anchor-free feature detection algorithm can process the ninth eigenvector P9 to obtain the type and center position of the target object, and process the tenth eigenvector P10 to obtain the distance between the center position of the target object and the target frame. In this way, the position of the target object (that is, the position of the target frame determined by the center position and the distance between the center position and the target frame) and type can be determined to achieve tracking of the target object.

[0080] In an example, if the fourth eigenvector P4 is 6*6*256 and the second eigenvector P2 is 22*22*256, the fifth eigenvector P5 used for classification and center position detection in the fourth eigenvector P4 is 4*4*2k*256, and the seventh eigenvector P7 used for classification and center position detection in the second eigenvector P2 is 20*20*256. When the fifth eigenvector P5 and the seventh eigenvector P7 are cross-correlated, the 4*4*2k*256 can be split into 2k 4*4*256s, and cross-correlated with 20*20*256 respectively (in fact, it is a convolution operation, 4*4 is the size of the convolution kernel), and the result is the ninth eigenvector P9, which is 17*17*2k.

[0081] Similarly, the sixth eigenvector P6 in the fourth eigenvector P4 used for detecting the distance between the center position and the target frame is 4*4*4k*256, and the eighth eigenvector P8 in the second eigenvector P2 used for detecting the distance between the center position and the target frame is 20*20*256. When the sixth eigenvector P6 and the eighth eigenvector P8 are cross-correlated, 4*4*4k*256 can be split into 4k 4*4*256s, and cross-correlated with 20*20*256 respectively (in fact, it is a convolution operation, 4*4 is the size of the convolution kernel), and the result is the tenth eigenvector P10, which is 17*17*2k.

[0082] See also Figure 8 and Figure 9 , the target tracking method also includes:

[0083] 017: Get the preset target tracking model;

[0084] Specifically, the target tracking model needs to be trained in advance.

[0085] Step 017 includes:

[0086] Step 0171: Obtain a training set, where the training set includes multiple image groups, each image group includes a training template image and a training search image, and the training set also includes label information corresponding to the image groups. The training search image and the training template image contain the same target object.

[0087] Step 0172: Convolve the training template image and the training search image using the tracking model to extract a first feature of the training template image and a second feature of the training search image;

[0088] Step 0173: Perform a linear operation on the first feature and the second feature to obtain the third feature and the fourth feature respectively;

[0089] Step 0174: Generate a first feature vector based on the first feature and the third feature, and generate a second feature vector based on the second feature and the fourth feature; and

[0090] Step 0175: Output target tracking information according to the first eigenvector and the second eigenvector;

[0091] Step 0176: Based on a preset loss function, a loss value is calculated according to the target tracking information and label information;

[0092] Step 0177: Adjust the tracking model according to the loss value until the tracking model converges to obtain the target tracking model.

[0093] Specifically, the target tracking model needs to be pre-trained. During training, a training set can be obtained first. As the name suggests, the training set is used to train the model. In order to reduce generalization error, we need to continuously train the target tracking model with the training set to better approximate the real data.

[0094] The training set includes multiple image groups, each image group includes a training template image and a training search image, and the training set also includes label information corresponding to the image groups one by one, and the label information includes real tracking information corresponding to the image groups.

[0095] The training set includes positive samples and negative samples. In the image group corresponding to the positive samples, the target object contained in the training template image is the same as the target object contained in the training search image. In the image group corresponding to the negative samples, the target object contained in the training template image is different from the target object contained in the training search image. In this way, by training with a training set that contains both positive and negative samples, the training effect of the target tracking model can be improved.

[0096] Furthermore, due to the limited number of samples currently available, when obtaining a training set, only the training template images can be acquired. Data augmentation can then be performed based on these images to obtain training search images. This can include deforming the target object in the training template images or changing the color information of the training template images. This eliminates the need to acquire a training search image for each training template image during actual target tracking, and allows a sufficient number of samples to be obtained even with a limited number of samples.

[0097] Then input the training set into the tracking model for training. During the training process, it is also necessary to perform feature extraction on the training template image and the training search image of the image group. The feature extraction method is the same as step 011 and will not be repeated here.

[0098] Similarly, for detailed explanations of Step 0173, Step 0174, and Step 0175, please refer to the descriptions of Step 012, Step 013, and Step 014, respectively.

[0099] After obtaining the target tracking information, the loss value can be calculated based on the preset loss function according to the label information and the target tracking information. It can be understood that the greater the difference between the target tracking information and the label information, the greater the loss value, indicating that the detection accuracy of the target tracking information is lower. At this time, the hyperparameters in the tracking model can be adjusted according to the loss value, thereby continuously reducing the loss value and converging the loss value. If after multiple rounds of training with the training set, the loss value no longer jumps but remains basically unchanged, then the tracking model can be considered to have converged, thereby obtaining a converged target tracking model. In this way, the training set can be used to train the target tracking model.

[0100] After obtaining a converged target tracking model, you can test it against the test set to verify the effectiveness of the training. The test set is used to test the model's accuracy. By applying the test set to the target tracking model trained with the training set, a model score is obtained. The test set is used to evaluate the generalization ability of the target tracking model but should not be used as a basis for algorithmic choices such as parameter tuning and feature selection. The role of the test set is reflected in the testing process.

[0101] During the training process, the validation set can be used to assist with parameter adjustment, feature selection, and other algorithm-related decisions. The validation set can be used to determine whether the training of the target tracking model is progressing in a negative direction. The validation set's role is reflected in the training process. For example, by observing the change in loss between the training and validation sets, it can be determined whether the model is overfitting. If so, training can be stopped promptly, and the model structure and hyperparameters can be adjusted, significantly saving time.

[0102] See also Figure 10 The image group further includes a training update image, the continuous multiple frames of images include a training template image and a training search image, the shooting time of the training template image is earlier than the shooting time of the training search image, and the training update image is any frame image between the training template image and the training search image in the continuous multiple frames of images; the target tracking method further includes:

[0103] Step 0178: Convolve the training update image through the tracking model to extract the fifth feature of the training update image, and perform a linear operation on the fifth feature to obtain the sixth feature;

[0104] Step 0179: Generate a third eigenvector based on the fifth and sixth features;

[0105] Step 0175: Output target tracking information based on the first eigenvector and the second eigenvector, including:

[0106] Step 01751: weightedly fuse the first eigenvector and the third eigenvector using a preset weight value to generate a fourth eigenvector;

[0107] Step 01752: Output target tracking information based on the fourth eigenvector and the second eigenvector;

[0108] Step 0177: Adjust the tracking model according to the loss value until the tracking model converges to obtain the target tracking model, including:

[0109] Step 01771: Adjust the preset weights and preset model parameters according to the loss value until the tracking model converges to obtain the target tracking model.

[0110] Specifically, for the explanation of step 0178 and step 0179, please refer to the description of step 015 and step 016 respectively. For the explanation of step 01751 and step 01752, please refer to step 0141 and step 0142 respectively. No further details will be given here.

[0111] When adjusting the tracking model according to the loss value, the preset weights for weighted fusion of the first eigenvector of the training template image and the third eigenvector of the training update image and the preset model parameters (i.e., the hyperparameters of the tracking model) can be adjusted simultaneously, thereby realizing the training of the tracking model including three branch networks to obtain the target tracking model.

[0112] In order to better implement the target tracking method of the embodiment of the present invention, the embodiment of the present invention also provides an image-based target tracking device 10. Figure 11 , the target tracking device 10 may include:

[0113] A feature extraction module 11 is configured to convolve the template image and the search image using a preset target tracking model to extract a first feature of the template image and a second feature of the search image, wherein the search image and the template image contain the same target object; perform linear operations on the first feature and the second feature to obtain a third feature and a fourth feature; generate a first feature vector based on the first feature and the third feature, and generate a second feature vector based on the second feature and the fourth feature;

[0114] The post-processing module 12 is configured to output target tracking information corresponding to the search image based on the first eigenvector and the second eigenvector.

[0115] The feature extraction module 11 is also used to convolve the updated image through multiple convolution kernels to extract the fifth feature of the updated image, and perform linear operation on the fifth feature to obtain the sixth feature. The updated image is generated based on a frame image between the template image and the current frame in the continuous multi-frame image, the template image is any frame in the continuous multi-frame image, and the current frame is the last frame in the continuous multi-frame image; based on the fifth feature and the sixth feature, a third feature vector is generated; and through a preset weight, the first feature vector and the third feature vector are weightedly fused to generate a fourth feature vector; the post-processing module 12 is specifically further used to weightedly fuse the first feature vector and the third feature vector through the preset weight of the first feature vector and the preset weight of the third feature vector to generate a fourth feature vector; based on the fourth feature vector and the second feature vector, target tracking information is output.

[0116] The post-processing module 12 is specifically further used to output the type of the target object and the center position of the target object based on the preset anchor-free frame target detection algorithm according to the part of the fourth eigenvector used to detect the classification and the center position and the part of the second eigenvector used to detect the classification and the center position, and output the distance between the center position and the target frame according to the part of the fourth eigenvector used to detect the distance between the center position and the target frame and the part of the second eigenvector used to detect the distance between the center position and the target frame.

[0117] The target tracking device 10 also includes a feature processing module 13, which is used to split the fourth eigenvector to obtain the fifth eigenvector and the sixth eigenvector, and split the second eigenvector to obtain the seventh eigenvector and the eighth eigenvector. The fifth eigenvector and the seventh eigenvector both include information for detecting the classification and the center position, and the sixth eigenvector and the eighth eigenvector both include information for detecting the distance between the center position and the target frame; based on the fifth eigenvector and the seventh eigenvector, a ninth eigenvector is generated, and based on the sixth eigenvector and the eighth eigenvector, a tenth eigenvector is generated; the post-processing module 12 is also used to output the type of the target object and the center position of the target object based on the ninth eigenvector; and based on the tenth eigenvector, output the distance between the center position and the target frame.

[0118] The target tracking device 10 further includes

[0119] An acquisition module 14 is used to acquire a preset target tracking model;

[0120] The acquisition module 14 is specifically used for:

[0121] Obtaining a training set, where the training set includes multiple image groups, each image group includes a training template image and a training search image, and the training set also includes label information corresponding to the image groups, where the training search image and the training template image contain the same target object;

[0122] Convolving the training template image and the training search image through the tracking model to extract a first feature of the training template image and a second feature of the training search image;

[0123] Performing a linear operation on the first feature and the second feature to obtain a third feature and a fourth feature respectively;

[0124] Generate a first feature vector based on the first feature and the third feature, and generate a second feature vector based on the second feature and the fourth feature; and

[0125] outputting target tracking information according to the first eigenvector and the second eigenvector;

[0126] Based on the preset loss function, the loss value is calculated according to the target tracking information and label information;

[0127] The tracking model is adjusted according to the loss value until the tracking model converges to obtain the target tracking model.

[0128] The acquisition module 14 is specifically used for:

[0129] Convolving the training update image through the tracking model to extract the fifth feature of the training update image, and performing a linear operation on the fifth feature to obtain the sixth feature;

[0130] Generate a third eigenvector based on the fifth and sixth features;

[0131] By presetting weights, weighted fusion of the first eigenvector and the third eigenvector is performed to generate a fourth eigenvector;

[0132] outputting target tracking information according to the fourth eigenvector and the second eigenvector;

[0133] The preset weights and preset model parameters are adjusted according to the loss value until the tracking model converges to obtain the target tracking model.

[0134] Each module in the target tracking device 10 can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0135] See also Figure 12 The electronic device 100 according to the embodiment of the present invention includes a processor 30. The processor 30 is configured to execute the target tracking method according to any one of the above embodiments, which will not be described in detail for the sake of brevity.

[0136] The electronic device 100 may be an embedded device, a mobile phone, a smart phone, a personal digital assistant (PDA), a tablet computer and a video game device, a portable terminal (such as a laptop), a server, or a larger device (such as a desktop computer and a television).

[0137] See also Figure 13 The embodiment of the present invention further provides a computer-readable storage medium 300 on which a computer program 310 is stored. When the computer program 310 is executed by the processor 30, the steps of the target tracking method of any of the above-mentioned embodiments are implemented. For the sake of brevity, they are not repeated here.

[0138] It will be understood that computer program 310 includes computer program code. The computer program code may be in source code form, object code form, executable file, or some intermediate form. Computer-readable storage media may include any entity or device capable of carrying computer program code, recording media, USB flash drives, mobile hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media.

[0139] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are mutually inconsistent.

[0140] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes additional implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0141] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. An image-based target tracking method, characterized in that: include: Convolving a template image and a search image respectively using a preset target tracking model to extract a first feature of the template image and a second feature of the search image, wherein the search image and the template image contain the same target object; Performing linear operations on the first feature and the second feature respectively to obtain a third feature and a fourth feature; generating a first feature vector based on the first feature and the third feature, and generating a second feature vector based on the second feature and the fourth feature; and Output target tracking information corresponding to the search image according to the first feature vector and the second feature vector.

2. The target tracking method according to claim 1, characterized in that Also includes: performing convolution on the update image using the target tracking model to extract a fifth feature of the update image, and performing a linear operation on the fifth feature to obtain a sixth feature, wherein the continuous multi-frame image includes the template image and the search image, the template image is captured earlier than the search image, and the update image is any frame image between the template image and the search image in the continuous multi-frame image; generating a third feature vector according to the fifth feature and the sixth feature; Outputting target tracking information according to the first feature vector and the second feature vector includes: weightedly fusing the first eigenvector and the third eigenvector using a preset weight of the first eigenvector and a preset weight of the third eigenvector to generate a fourth eigenvector; The target tracking information is output according to the fourth eigenvector and the second eigenvector.

3. The target tracking method according to claim 2, characterized in that The target tracking information includes a type of the target object, a center position of the target object, and a distance between the center position and a target frame. Outputting the target tracking information according to the fourth eigenvector and the second eigenvector includes: Based on a preset anchor-free target detection algorithm, the type of the target object and the center position of the target object are output according to the part of the fourth eigenvector used to detect the classification and the center position and the part of the second eigenvector used to detect the classification and the center position, and the distance of the center position from the target frame is output according to the part of the fourth eigenvector used to detect the distance between the center position and the target frame and the part of the second eigenvector used to detect the distance between the center position and the target frame.

4. The target tracking method according to claim 3, characterized in that: The preset anchor-free target detection algorithm processes the fourth eigenvector and the second eigenvector to output the target tracking information, including: Splitting the fourth eigenvector to obtain a fifth eigenvector and a sixth eigenvector, and splitting the second eigenvector to obtain a seventh eigenvector and an eighth eigenvector, wherein the fifth eigenvector and the seventh eigenvector both include information for detecting the classification and the center position, and the sixth eigenvector and the eighth eigenvector both include information for detecting the distance between the center position and the target frame; generating a ninth eigenvector based on the fifth and seventh eigenvectors, and generating a tenth eigenvector based on the sixth and eighth eigenvectors; outputting the type of the target object and the center position of the target object according to the ninth eigenvector; and According to the tenth eigenvector, the distance between the center position and the target frame is output.

5. The target tracking method according to claim 1, wherein: Also includes: Obtaining the preset target tracking model; The obtaining of the preset target tracking model includes: Acquire a training set, the training set comprising a plurality of image groups, each of the image groups comprising a training template image and a training search image, the training set further comprising label information corresponding one-to-one to the image groups, the training search image and the training template image containing the same target object; Convolving the training template image and the training search image through a tracking model to extract a first feature of the training template image and a second feature of the training search image; Performing a linear operation on the first feature and the second feature to obtain a third feature and a fourth feature, respectively; generating a first feature vector based on the first feature and the third feature, and generating a second feature vector based on the second feature and the fourth feature; and Outputting target tracking information according to the first feature vector and the second feature vector; Based on a preset loss function, a loss value is calculated according to the target tracking information and the label information; The tracking model is adjusted according to the loss value until the tracking model converges to obtain the target tracking model.

6. The target tracking method according to claim 5, characterized in that: The image group further includes a training update image, the continuous multiple frames of images include the training template image and the training search image, the shooting time of the training template image is earlier than the shooting time of the training search image, and the training update image is any frame image between the training template image and the training search image in the continuous multiple frames of images; The target tracking method further includes: performing convolution on the training update image using a tracking model to extract a fifth feature of the training update image, and performing a linear operation on the fifth feature to obtain a sixth feature; generating a third feature vector according to the fifth feature and the sixth feature; Outputting target tracking information according to the first feature vector and the second feature vector includes: By using a preset weight value, weightedly fusing the first eigenvector and the third eigenvector to generate a fourth eigenvector; outputting the target tracking information according to the fourth eigenvector and the second eigenvector; The step of adjusting the tracking model according to the loss value until the tracking model converges to obtain the target tracking model includes: The preset weights and preset model parameters are adjusted according to the loss value until the tracking model converges to obtain the target tracking model.

7. The target tracking method according to claim 5, characterized in that: The training set includes positive samples and negative samples. In the image group corresponding to the positive samples, the target object contained in the template image is the same as the target object contained in the search image; in the image group corresponding to the negative samples, the target object contained in the template image is different from the target object contained in the search image.

8. An image-based target tracking device, characterized in that: include: a feature extraction module, configured to convolve a template image and a search image using a preset target tracking model to extract a first feature of the template image and a second feature of the search image, wherein the search image and the template image contain the same target object; perform linear operations on the first feature and the second feature to obtain a third feature and a fourth feature; generate a first feature vector based on the first feature and the third feature, and generate a second feature vector based on the second feature and the fourth feature; A post-processing module is used to output target tracking information corresponding to the search image based on the first feature vector and the second feature vector.

9. An electronic device, characterized in that: The device comprises a processor, wherein the processor is used to execute the target tracking method according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium for a computer program, characterized in that: When the computer program is executed by one or more processors, the target tracking method according to any one of claims 1 to 7 is implemented.

Images