Target tracking methods, devices, electronic devices, and readable storage media

Abstract

This invention provides a target tracking method, apparatus, electronic device, and readable storage medium. The method includes: acquiring a first image pair of the target object; performing multi-level feature extraction on the first image pair to obtain feature information of the first image pair at at least two scales; fusing the feature information at at least two scales using a deep aggregation network to obtain a feature extraction result of the first image pair; fusing a first feature vector and a second feature vector based on a multi-head attention mechanism to obtain a fused feature vector of the first image pair; and performing prediction processing on the fused feature vector to obtain a position prediction result of the target object. Through this method, multi-level image features can be extracted, and more skip connections are made between low-level and high-level features of the image, thereby extracting richer and deeper feature information, which greatly improves the accuracy and efficiency of target tracking.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more particularly to a target tracking method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Single-target tracking is an important task in computer vision, aiming to track a specified target throughout a video stream. It has significant applications in motion analysis, vehicle navigation, and autonomous driving.

[0003] In related technologies, single-target tracking methods generally involve a local linear matching process to track a single target.

[0004] However, the above methods focus too much on the characteristics of the tracked object itself, and the information richness is too low, resulting in a low accuracy in predicting the location of the tracked object when performing target tracking. Summary of the Invention

[0005] This invention provides a target tracking method, apparatus, electronic device, and readable storage medium, which can effectively reduce the computational complexity of target tracking, extract multi-level image features, and enable more jump connections between low-level and high-level features of the image, thereby extracting richer and deeper feature information and greatly improving the accuracy of target tracking.

[0006] In a first aspect, embodiments of the present invention provide a target tracking method, the method comprising:

[0007] A first image pair of the tracked object is acquired, the first image pair including a reference image and a search region image, the first image pair being used to predict the position of the tracked object in the next frame of the search region image;

[0008] Multi-level feature extraction is performed on the first image pair to obtain feature information of the first image pair at at least two scales;

[0009] A deep aggregation network is used to fuse the feature information at at least two scales to obtain the feature extraction result of the first image pair. The feature extraction result includes a first feature vector of the reference image and a second feature vector of the search region image.

[0010] Based on the multi-head attention mechanism, the first feature vector and the second feature vector are fused to obtain the fused feature vector of the first image pair.

[0011] The fused feature vector is subjected to prediction processing to obtain the position prediction result of the tracked object.

[0012] Optionally, acquiring the first image pair of the tracked object includes:

[0013] Acquire the initial and target images of the tracked object;

[0014] The initial image is cropped to obtain a reference image including the tracked object;

[0015] The target image is cropped to obtain the search area image of the tracked object;

[0016] The proportion of the tracked object in the image of the search area meets a preset ratio requirement.

[0017] Optionally, the step of performing multi-level feature extraction on the first image pair to obtain feature information of the first image pair at at least two scales includes:

[0018] The reference image and the search region image are downsampled at at least two scales to obtain a first sampled image of the reference image at each scale and a second sampled image of the search region image at each scale.

[0019] The residual network is used to extract features from the first sampled image and the second sampled image to obtain a first feature map of the reference image at each scale and a second feature map of the search region image at each scale.

[0020] Optionally, the step of using a deep aggregation network to perform feature fusion on the feature information at the at least two scales to obtain the feature extraction result of the first image pair includes:

[0021] The first feature map is processed by deformable convolution using a deep aggregation network to obtain the third feature map of the reference image;

[0022] The second feature map is processed by deformable convolution using a deep aggregation network to obtain the fourth feature map of the search region image;

[0023] The third feature map is convolved to obtain the fifth feature map, wherein the number of channels in the fifth feature map is a preset number;

[0024] The fourth feature map is convolved to obtain the sixth feature map, wherein the number of channels in the sixth feature map is a preset number.

[0025] The fifth and sixth feature maps are flattened to obtain the first feature vector of the reference image and the second feature vector of the search region image.

[0026] Optionally, the feature fusion of the first feature vector and the second feature vector based on the multi-head attention mechanism to obtain the fused feature vector of the first image pair includes:

[0027] The first feature vector is encoded based on a multi-head self-attention mechanism to obtain the first encoded vector of the first feature vector.

[0028] The second feature vector is encoded based on a multi-head self-attention mechanism to obtain the second encoded vector of the second feature vector.

[0029] The first and second encoded vectors are fused and decoded based on a multi-head cross-attention mechanism to obtain a fused feature vector of the first and second feature vectors.

[0030] Optionally, the step of fusing and decoding the first and second encoded vectors based on a multi-head cross-attention mechanism to obtain a fused feature vector of the first and second feature vectors includes:

[0031] Spatial position encoding is performed on the first encoding vector and the second encoding vector to obtain a third encoding vector and a fourth encoding vector. The third encoding vector includes the spatial position encoding of the first encoding vector and the first encoding vector, and the fourth encoding vector includes the spatial position encoding of the second encoding vector and the second encoding vector.

[0032] Using the second encoding vector, the third encoding vector, and the fourth encoding vector as input, a multi-head cross-attention network is used to calculate the first fused feature vector of the first feature vector and the second feature vector;

[0033] The first fused feature vector is processed using a feedforward neural network in residual form to obtain the second fused feature vector;

[0034] The fusion feature vector of the first feature vector and the second feature vector is calculated based on the first fusion feature vector and the first fusion feature vector.

[0035] Optionally, the step of performing prediction processing on the fused feature vector to obtain the location prediction result of the tracked object includes:

[0036] The fused feature vector is processed using a multilayer perceptron to obtain the position prediction result of the tracked object.

[0037] Optionally, the location prediction result includes a predicted bounding box and the foreground probability corresponding to the predicted bounding box, and the method further includes:

[0038] The edge penalty score for each predicted bounding box is calculated using the Hanning window function.

[0039] For each predicted bounding box, the edge penalty score and foreground probability are weighted and calculated to obtain the prediction score for each predicted bounding box.

[0040] Based on the prediction score, the final position prediction result of the tracked object is determined as the prediction bounding box with the highest prediction score.

[0041] Optionally, the frame containing the predicted location of the tracked object is the next frame of the search area image, and the method further includes:

[0042] Based on the location prediction results, the motion trajectory of the tracked object in the video stream is predicted.

[0043] Secondly, embodiments of the present invention provide a target tracking device, the device comprising:

[0044] The image pair acquisition module is used to acquire a first image pair of the tracked object, the first image pair including a reference image and a search area image, and the first image pair is used to predict the position of the tracked object in the next frame of the search area image;

[0045] A multi-level feature extraction module is used to perform multi-level feature extraction on the first image pair to obtain feature information of the first image pair at at least two scales.

[0046] The deep aggregation module is used to perform feature fusion on feature information at at least two scales using a deep aggregation network to obtain the feature extraction result of the first image pair. The feature extraction result includes a first feature vector of the reference image and a second feature vector of the search region image.

[0047] The feature fusion module is used to fuse the first feature vector and the second feature vector based on a multi-head attention mechanism to obtain the fused feature vector of the first image pair.

[0048] The prediction module is used to perform prediction processing on the fused feature vector to obtain the position prediction result of the tracked object.

[0049] Thirdly, embodiments of the present invention provide an electronic device, including: a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the target tracking method as described in the first aspect.

[0050] Fourthly, embodiments of the present invention provide a readable storage medium storing computer instructions, which, when executed by a processor, implement the target tracking method as described in the first aspect.

[0051] The embodiments of the present invention have the following advantages:

[0052] In summary, this application provides an embodiment of a target tracking method. After acquiring a first image pair of the target object, multi-level feature extraction is performed on the first image pair to obtain feature information of the first image pair at at least two scales. Then, a deep aggregation network is used to fuse the feature information at the at least two scales to obtain the feature extraction result of the first image pair. Furthermore, a multi-head attention mechanism is used to fuse the first feature vector and the second feature vector to obtain the fused feature vector of the first image pair. Finally, the fused feature vector is used for prediction processing to obtain the position prediction result of the target object. This method effectively reduces the computational complexity of target tracking, extracts multi-level image features, and enables more skip connections between low-level and high-level features of the image, thereby extracting richer and deeper feature information and significantly improving the accuracy and efficiency of target tracking. Attached Figure Description

[0053] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart of an embodiment of the target tracking method of the present invention is shown;

[0055] Figure 2 An architecture diagram of a deep aggregation network according to an embodiment of this application is shown;

[0056] Figure 3 A flowchart of an encoding network based on a multi-head self-attention mechanism according to an embodiment of this application is shown;

[0057] Figure 4 A flowchart of a fusion decoding network based on multi-head cross-attention according to an embodiment of this application is shown;

[0058] Figure 5 A flowchart of another embodiment of the target tracking method of the present invention is shown;

[0059] Figure 6 A structural block diagram of an embodiment of the target tracking device of the present invention is shown;

[0060] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Furthermore, it should be noted that the various data acquisition processes in the embodiments of the present invention are all carried out in compliance with the relevant data protection laws and policies of the country where the invention is located, and with the authorization of the owner of the corresponding device.

[0062] Reference Figure 1 The flowchart illustrates an embodiment of the target tracking method of the present invention, which may specifically include the following steps:

[0063] Step 101: Obtain a first image pair of the tracked object. The first image pair includes a reference image and a search area image. The first image pair is used to predict the position of the tracked object in the next frame of the search area image.

[0064] When predicting the position of a tracked object in a given frame, the initial frame of the video stream containing that frame can be selected as a reference frame. The tracked object is identified within this reference frame, generating a reference image including the tracked object, which serves as a reference for subsequent position prediction. The previous frame is also needed as a search frame, where the tracked object is identified, generating a search region image including the tracked object. This allows for consideration of the tracked object's continuous movements during position prediction. Thus, a first image pair consisting of the search image and the reference image is used to predict the tracked object's position in the next frame of the search region image. The combined use of the reference image and the search region image fully considers the individual characteristics of the tracked object and its relationships between preceding and following frames, avoiding situations where only local optima exist during position prediction.

[0065] In one optional embodiment of this application, step 101, obtaining the first image pair of the tracked object, includes:

[0066] Step S11: Obtain the initial image and target image of the object being tracked;

[0067] Step S12: Crop the initial image to obtain a reference image including the tracked object;

[0068] Step S13: Crop the target image to obtain the search area image of the tracked object;

[0069] The proportion of the tracked object in the image of the search area meets a preset ratio requirement.

[0070] In this context, the initial image refers to the first frame in the video stream containing the tracked object, and the target image refers to the frame preceding the frame where the tracked object's position is predicted. To ensure sufficient capture of the tracked object's features in the images, both the initial and target images need to be cropped to prevent the tracked object's proportion from being too large or too small, which would result in insufficient feature extraction. Furthermore, to ensure that the search area image encompasses the tracked object's range in the next frame, the proportion of the tracked object in the search area image must meet a certain ratio requirement, i.e., a preset ratio requirement. This preset ratio requirement can be determined based on the specific tracking results and is not limited in this application.

[0071] As an example, the template image can be taken from the first frame of the original video image, expanded by twice the side length centered on the tracked object, and cropped to 64x64 to include the tracked object and its surrounding boundaries. The search area image can be taken from the previous frame image, expanded by four times the side length centered on the tracked object, and cropped to 128x128 to include the possible range of movement of the tracked object.

[0072] Step 102: Perform multi-level feature extraction on the first image pair to obtain feature information of the first image pair at at least two scales.

[0073] When performing feature extraction on an image, the feature levels extracted from images at different scales are also different. Therefore, after downsampling the first image pair at at least two scales, feature maps are then extracted separately, such as residual networks, convolutional neural networks, etc. The specific feature extraction method is not limited in this application, thereby obtaining the feature information of the first image pair at at least two scales.

[0074] In one optional embodiment of this application, step 102, which involves multi-level feature extraction of the first image pair to obtain feature information of the first image pair at at least two scales, may include:

[0075] Step S21: Downsample the reference image and the search region image at at least two scales to obtain a first sampled image of the reference image at each scale and a second sampled image of the search region image at each scale;

[0076] Step S22: Use a residual network to extract features from the first sampled image and the second sampled image to obtain a first feature map of the reference image at each scale and a second feature map of the search region image at each scale.

[0077] Different reference images and search region images are downsampled to obtain images at each sampling scale of the reference image (i.e., the first sampled image) and images at each scale of the search region image (i.e., the second sampled image). After downsampling, a residual network is used to extract features from the first and second sampled images to obtain feature maps of the reference image at each scale (i.e., the first feature map) and feature maps of the search region image at each scale (i.e., the second feature map).

[0078] As an example, the first image pair can be downsampled at resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32, respectively, to obtain four first-sampled images of the reference image at scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32, and four second-sampled images of the search region image at scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32. Then, ResNet34 is used to extract features from the four first-sampled images and the four second-sampled images, resulting in four first feature maps of the reference image at scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32, and four second feature maps of the search image at scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32. Enriching the feature information of the first image pair at multiple scales helps to more accurately predict the location of the tracked object.

[0079] Step 103: Use a deep aggregation network to perform feature fusion on the feature information at at least two scales to obtain the feature extraction result of the first image pair. The feature extraction result includes a first feature vector of the reference image and a second feature vector of the search region image.

[0080] After extracting feature information from the first image pair at at least two scales, a Deep Layer Aggregation (DLA) network can be used to fuse the feature information from each image at at least two scales, resulting in a first feature vector fused from the reference image features and a second feature vector fused from the search region image features. This allows for dynamic adaptation of the receptive field based on the scale and pose of the tracked object, thereby enabling more skip connections between low-level and high-level features of the first image pair and reducing the computational complexity of target tracking during feature fusion extraction. When upsampling and fusing feature information at each scale, deformable convolution or other convolutional processing methods can be used; however, this application does not limit the specific upsampling fusion method.

[0081] In an optional embodiment of this application, based on obtaining the first feature map of the reference image at each scale and the second feature map of the search region image at each scale in steps S21-S22, step 103, which involves using a deep aggregation network to perform feature fusion on the feature information at the at least two scales to obtain the feature extraction result of the first image pair, may include:

[0082] Step S31: Perform deformable convolution processing on the first feature map using a deep aggregation network to obtain the third feature map of the reference image;

[0083] Step S32: Perform deformable convolution processing on the second feature map using a deep aggregation network to obtain the fourth feature map of the search region image;

[0084] Step S33: Perform convolution processing on the third feature map to obtain the fifth feature map, wherein the number of channels in the fifth feature map is a preset number;

[0085] Step S34: Perform convolution processing on the fourth feature map to obtain the sixth feature map, wherein the number of channels in the sixth feature map is a preset number;

[0086] Step S35: Flatten the fifth feature map and the sixth feature map respectively to obtain the first feature vector of the reference image and the second feature vector of the search region image.

[0087] Deformable convolution refers to learning an offset based on a parallel network, causing the convolutional kernel to shift its sampling points on the input feature map, focusing on the region or target of interest. Deep aggregation networks can perform arbitrary feature fusion on feature maps at different scales, helping to create more skip connections between low-level and high-level features, strengthening the correlation between features at different scales, and uncovering deeper feature information of the image of the tracked object. This helps to improve the network for predicting the location of the tracked object, enriches parameter information, and improves the accuracy of target tracking.

[0088] Regarding the upsampling and fusion of feature maps at different scales through deformable convolution described in steps S31 and S32, exemplarily, refer to... Figure 2The diagram illustrates the architecture of a deep aggregation network according to an embodiment of this application. In this embodiment, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 represent feature maps extracted from the reference image and search region image by ResNet34 at these four scales. Each blank square represents a feature map at a different stage of deep aggregation. The first blank square corresponding to each scale represents the feature map extracted from the image at that scale. The upward and diagonal arrows indicate upsampling fusion of the two feature maps represented by the arrowhead and tail squares based on deformable convolution. In this embodiment, the downsampling stride of the deep aggregation network is 4.

[0089] For feature maps of the same image, in the first half of the deep aggregation network, feature fusion is performed at each adjacent scale. For example, the feature map at the 1 / 8 scale is fused with the feature map at the 1 / 16 scale, and the feature map at the 1 / 4 scale is fused with the feature maps at both the 1 / 8 and 1 / 16 scales. In the second half of the deep aggregation network, the feature maps at each scale fused in the first half are directly fused with the feature map at the 1 / 4 scale to obtain the third and fourth feature maps, achieving a skip connection between low-level and high-level features. Then, the feature extraction result is obtained by proceeding with the methods described in steps S33-S35.

[0090] For the convolution processing described in steps S33 and S34, a 1x1 conv process can be performed. That is, multiple 1x1 conv processes are used to reduce the number of channels in the third and fourth feature maps obtained after feature fusion, so that the number of channels in the reduced feature maps is the preset number. The flattening process described in step S35 transforms the multidimensional feature maps into one-dimensional feature vectors, thereby obtaining the first feature vector of the reference image and the second feature vector of the search region image.

[0091] As an example, the third and fourth feature maps are each processed with 256 1x1 convolutions, resulting in fifth and sixth feature maps with 256 channels each. These fifth and sixth feature maps are then flattened to obtain one-dimensional feature vectors. Figure 2 Taking the feature extraction example shown, the downsampling stride of the entire deep aggregation network is 4. Then the size of the feature map represented by the first and second feature vectors of the final output is 256*(H / 4)*(W / 4), where H and W represent the size of the reference image and the search region image input to the residual network, respectively.

[0092] Step 104: Based on the multi-head attention mechanism, perform feature fusion on the first feature vector and the second feature vector to obtain the fused feature vector of the first image pair.

[0093] To further strengthen the connection between the features of the reference image and the features of the search region image, the first feature vector and the second feature vector are fused based on a multi-head self-attention mechanism. The specific steps may include encoding processing based on multi-head self-attention and decoding fusion processing based on multi-head self-attention.

[0094] In an optional embodiment of this application, step 104, which involves fusing the first and second feature vectors based on a multi-head attention mechanism to obtain the fused feature vector of the first image pair, may include:

[0095] Step S41: Encode the first feature vector based on the multi-head self-attention mechanism to obtain the first encoded vector of the first feature vector;

[0096] Step S42: Encode the second feature vector based on the multi-head self-attention mechanism to obtain the second encoded vector of the second feature vector;

[0097] Step S43: Perform fusion decoding on the first encoding vector and the second encoding vector based on the multi-head cross-attention mechanism to obtain the fused feature vector of the first feature vector and the second feature vector.

[0098] Specifically, refer to Figure 3 The flowchart shown is of an encoding network based on a multi-head self-attention mechanism according to an embodiment of this application. In the flowchart, X represents the input, that is, the first feature vector or the second feature vector to be encoded, and Xenc represents the output of the encoding network, that is, the first encoding vector or the second encoding vector.

[0099] During the encoding process, according to... Figure 3 The architecture shown encodes the first feature vector and the second feature vector respectively. This embodiment takes the encoding of the first feature vector as an example, introducing spatial position encoding and using the sine function to generate the spatial position encoding P of the first feature vector. x The spatial location code is added to the first feature vector, and the first feature vector with added spatial location code, namely X+P, is used as the reference. x Q and K are two inputs to the multi-head self-attention network, and the first feature vector X, which is not encoded with spatial location, is used as the input V of the multi-head self-attention network.

[0100] After inputting into the multi-head self-attention network, the Multi-head Attention of the multi-head self-attention network is calculated according to the following formula:

[0101]

[0102] H i =Attention(QW i Q ,KW i K VW i V (2)

[0103] Multihead(Q,K,V)=Concat(H1,…,H i W O (3)

[0104] in, The scaling factor represents the scaling factor used when calculating the attention score using the scaled dot-product attention method. Specifically, it can be the number of columns in the Q and K matrices, i.e., the vector dimension; Hi represents the attention calculation result of the i-th head in multi-head attention, and W... i Q W i K and W i V These are the transformation matrices corresponding to Q, K, and V, respectively, which can be preset or obtained through pre-training; Concat() represents the connection function between multiple attention heads, W O This represents the parameter matrix used to compute Multihead(Q,K,V); Multihead(Q,K,V) represents the attention computation result of the multi-head self-attention network, i.e., Multi-head Attention.

[0105] Finally, the output Xenc of the encoding network, i.e., the first encoded vector, can be calculated using the following formula:

[0106] X enc =X+Multihead(X+P x ,X+P x (4)

[0107] The above encoding process is illustrated using the first feature vector as an example. The same process is applied to the second feature vector. Figure 3 The encoding process shown can be referred to the above calculation process, and will not be elaborated upon in this application.

[0108] After encoding, a fusion decoding process based on multi-head cross-attention is required. Therefore, in an optional embodiment of this application, step S43, which involves fusing and decoding the first and second encoded vectors based on a multi-head cross-attention mechanism to obtain a fused feature vector of the first and second feature vectors, may include:

[0109] Step S51: Perform spatial position encoding processing on the first encoding vector and the second encoding vector to obtain a third encoding vector and a fourth encoding vector. The third encoding vector includes the spatial position encoding of the first encoding vector and the first encoding vector, and the fourth encoding vector includes the spatial position encoding of the second encoding vector and the second encoding vector.

[0110] Step S52: Using the second encoding vector, the third encoding vector, and the fourth encoding vector as input, a multi-head cross-attention network is used to calculate the first fused feature vector of the first feature vector and the second feature vector;

[0111] Step S53: Process the first fused feature vector using a feedforward neural network in residual form to obtain the second fused feature vector;

[0112] Step S54: Calculate the fusion feature vector of the first feature vector and the second feature vector based on the first fusion feature vector and the first fusion feature vector.

[0113] When fusing and decoding encoded vectors, the Multihead cross-attention mechanism can be used. Specifically, refer to... Figure 4 The flowchart shown is a fusion decoding network based on multi-head cross-attention according to an embodiment of this application, where Xq represents the network after which Xq is processed. Figure 3 The first encoded vector of the first feature vector in the output Xenc obtained by the encoding network shown is Xkv, which represents the first encoded vector after processing the first feature vector. Figure 3 The second encoded vector is the second feature vector in the output Xenc calculated by the encoding network shown. The feedforward neural network in the figure is a residual form feedforward neural network, where add&norm represents the residual connection and normalization process.

[0114] Similar to the spatial location encoding described above, spatial location encoding is also introduced in step S51 when fusing and decoding the first and second encoding vectors. The sin function is used to generate the spatial location encodings Pq and Pkv of the first encoding vector Xq and the second encoding vector Xkv.

[0115] Corresponding to step S52, the first encoding vector with added spatial location encoding, i.e., the third encoding vector Xq+Pq, the second encoding vector without added spatial location encoding, and the second encoding vector with added spatial location encoding, i.e., the fourth encoding vector Xkv+Pkv, are used as inputs to the multi-head cross-attention network in the fusion decoding network. The calculation of Multi-head Attention in this multi-head self-attention network is similar to the aforementioned formulas (1)-(3), and will not be repeated here in the embodiments of this application.

[0116] Based on the output of the multi-head cross-attention network, the output of the regression processing is calculated according to the following formula:

[0117]

[0118] This represents the output after regression processing, i.e., the first fused feature vector, Multihead(X). q +P q ,X kv +P kv ,X kv ) represents the output of the multi-head cross-attention network.

[0119] To enhance the fitting ability of feature fusion, corresponding to step S53, the first fused feature vector is input into the feedforward neural network in the form of residuals and processed according to the following formula:

[0120] FFN(x)=max (0,xW1+b1)W2+b2, (6)

[0121] Where x represents the input of the feedforward neural network, and W1, W2 and b1, b2 are the parameters of the feedforward neural network, which can be obtained through pre-training.

[0122] Finally, corresponding to step S54, residual connection and standardization are performed according to the following formula to obtain the fused feature vector of the first and second feature vectors:

[0123]

[0124] in, X represents the output of the feedforward neural network after inputting the first fused feature vector, i.e., the second fused feature vector; dec This represents the fused feature vector of the first and second feature vectors.

[0125] Step 105: Perform prediction processing on the fused feature vector to obtain the position prediction result of the tracked object.

[0126] After obtaining the fused feature vector of the first image pair, the fused feature vector can be used for prediction processing to obtain the prediction result of the tracked object. Specific prediction methods can include multilayer perceptron (MLP) prediction, decision tree prediction, etc. This application does not limit this and can select according to the actual situation.

[0127] In one optional embodiment of this application, step 105, which involves performing prediction processing on the fused feature vector to obtain the position prediction result of the tracked object, may include:

[0128] Step S61: Use a multilayer perceptron to perform prediction processing on the fused feature vector to obtain the position prediction result of the tracked object.

[0129] Specifically, based on the fused feature vector, a multilayer perceptron is used to predict the normalized bounding box and foreground probability of the tracked object in the search region image through two prediction branches. The prediction branches include classification and regression branches, both of which can be MLPs with 256 hidden units.

[0130] In one optional embodiment of this application, the location prediction result includes a predicted bounding box and a foreground probability corresponding to the predicted bounding box, and the method may further include:

[0131] Step S71: Calculate the edge penalty score for each predicted bounding box using the Hanning window function;

[0132] Step S72: For each predicted bounding box, the edge penalty score and foreground probability are weighted and calculated to obtain the prediction score of each predicted bounding box;

[0133] Step S73: Based on the prediction score, determine the final position prediction result of the tracked object as the prediction bounding box with the highest prediction score.

[0134] Specifically, the edge penalty score for each predicted bounding box can be obtained using the window penalty through a 32x32 Hanning window. w And calculate the prediction score for each prediction bounding box according to the following formula:

[0135] score = (1 - w) × score w +w×score h (8)

[0136] Among them, score h represents the foreground probability corresponding to a certain predicted bounding box; score represents the prediction score of the predicted bounding box; w is the score in formula (8).h and score h The weighting coefficients can be preset.

[0137] After obtaining the prediction score of each predicted bounding box, the predicted bounding box with the highest prediction score is determined as the final position prediction result of the tracked object. Through the above method, the predicted bounding boxes that are far away from the tracked object in the search area image are suppressed, thus ensuring the continuity of the motion trajectory of the tracked object in consecutive frames.

[0138] In one optional embodiment of this application, the frame containing the predicted location of the tracked object is the next frame of the search area image, i.e., the current frame. The method may further include:

[0139] Based on the location prediction results, the motion trajectory of the tracked object in the video stream is predicted.

[0140] After obtaining the position prediction result of the tracked object in the current frame through the above method, the video stream frames can be continuously inferred based on the position prediction result to achieve single target tracking of the video stream.

[0141] Reference Figure 5 The flowchart shown is from another embodiment of the target tracking method of this application. The reference patch and search region patch are obtained according to the methods described in steps S11-S13 above. Multi-level features of the image are extracted and fused using a deep fusion backbone, i.e., a deep aggregation network and a residual network, according to the methods described in steps S21-S22 and S31-S34 above, to generate a multi-level fused feature map. Then, the features are processed according to the method described in step S35 above. The map is flattened to obtain the feature vector of the reference patch (i.e., the reference vector) and the feature vector of the search region patch (i.e., the search region vector). Following the methods described in steps S41-S42 and S51-S54, feature fusion based on multi-head attention (self-attention and cross-attention) is performed to obtain the fused feature vector of the reference patch and the search region patch. Following the method described in step S61, regression prediction and classification prediction are performed on the tracked object in the current frame based on the fused feature vector to obtain the position prediction result. Finally, following the methods described in steps S71-S73, a Hamming window is used to penalize the position prediction result, and the predicted bounding box with the highest prediction score is taken as the target prediction result of the tracked object.

[0142] In summary, this application provides an embodiment of a target tracking method. After acquiring a first image pair of the target object, multi-level feature extraction is performed on the first image pair to obtain feature information of the first image pair at at least two scales. Then, a deep aggregation network is used to fuse the feature information at the at least two scales to obtain the feature extraction result of the first image pair. Furthermore, a multi-head attention mechanism is used to fuse the first feature vector and the second feature vector to obtain the fused feature vector of the first image pair. Finally, the fused feature vector is used for prediction processing to obtain the position prediction result of the target object. This method effectively reduces the computational complexity of target tracking, extracts multi-level image features, and enables more skip connections between low-level and high-level features of the image, thereby extracting richer and deeper feature information and significantly improving the accuracy and efficiency of target tracking.

[0143] Reference Figure 6 The diagram illustrates a structural block diagram of an embodiment of a target tracking device according to the present invention. The device 200 may include:

[0144] Image pair acquisition module 201 is used to acquire a first image pair of the tracked object, the first image pair including a reference image and a search area image, the first image pair being used to predict the position of the tracked object in the next frame of the search area image;

[0145] The multi-level feature extraction module 202 is used to perform multi-level feature extraction on the first image pair to obtain feature information of the first image pair at at least two scales.

[0146] The deep aggregation module 203 is used to perform feature fusion on the feature information at the at least two scales using a deep aggregation network to obtain the feature extraction result of the first image pair. The feature extraction result includes a first feature vector of the reference image and a second feature vector of the search region image.

[0147] The feature fusion module 204 is used to fuse the first feature vector and the second feature vector based on a multi-head attention mechanism to obtain the fused feature vector of the first image pair.

[0148] The prediction module 205 is used to perform prediction processing on the fused feature vector to obtain the position prediction result of the tracked object.

[0149] Optionally, the image pair acquisition module may include:

[0150] The image acquisition submodule is used to acquire the initial image and target image of the tracked object;

[0151] The first cropping submodule is used to crop the initial image to obtain a reference image including the tracking object;

[0152] The second cropping submodule is used to crop the target image to obtain the search area image of the tracked object;

[0153] The proportion of the tracked object in the image of the search area meets a preset ratio requirement.

[0154] Optionally, the multi-level feature extraction module includes:

[0155] The downsampling submodule is used to downsample the reference image and the search region image at at least two scales to obtain a first sampled image of the reference image at each scale and a second sampled image of the search region image at each scale.

[0156] The residual network submodule is used to extract features from the first sampled image and the second sampled image using a residual network to obtain a first feature map of the reference image at each scale and a second feature map of the search region image at each scale.

[0157] Optionally, the deep aggregation module includes:

[0158] The first upsampling fusion submodule is used to perform deformable convolution processing on the first feature map using a deep aggregation network to obtain the third feature map of the reference image;

[0159] The second upsampling fusion submodule is used to perform deformable convolution processing on the second feature map using a deep aggregation network to obtain the fourth feature map of the search region image;

[0160] The first channel reduction submodule is used to perform convolution processing on the third feature map to obtain the fifth feature map, wherein the number of channels in the fifth feature map is a preset number;

[0161] The second channel reduction submodule is used to perform convolution processing on the fourth feature map to obtain the sixth feature map, wherein the number of channels in the sixth feature map is a preset number.

[0162] The flattening submodule is used to flatten the fifth feature map and the sixth feature map respectively to obtain the first feature vector of the reference image and the second feature vector of the search region image.

[0163] Optionally, the feature fusion module includes:

[0164] The first encoding submodule is used to encode the first feature vector based on a multi-head self-attention mechanism to obtain the first encoded vector of the first feature vector;

[0165] The first encoding submodule is used to encode the second feature vector based on a multi-head self-attention mechanism to obtain the second encoding vector of the second feature vector;

[0166] The fusion decoding submodule is used to perform fusion decoding processing on the first encoding vector and the second encoding vector based on a multi-head cross-attention mechanism to obtain a fused feature vector of the first feature vector and the second feature vector.

[0167] Optionally, the fusion decoding submodule includes:

[0168] A spatial location encoding submodule is used to perform spatial location encoding processing on the first encoding vector and the second encoding vector to obtain a third encoding vector and a fourth encoding vector. The third encoding vector includes the spatial location encoding of the first encoding vector and the first encoding vector, and the fourth encoding vector includes the spatial location encoding of the second encoding vector and the second encoding vector.

[0169] The multi-head cross-attention submodule is used to take the second encoding vector, the third encoding vector, and the fourth encoding vector as input, and use the multi-head cross-attention network to calculate the first fused feature vector of the first feature vector and the second feature vector;

[0170] The feedforward neural network submodule is used to process the first fused feature vector using a feedforward neural network in the form of a residual to obtain a second fused feature vector;

[0171] The fusion feature vector generation submodule is used to calculate the fusion feature vector of the first feature vector and the second feature vector based on the first fusion feature vector and the first fusion feature vector.

[0172] Optionally, the prediction module includes:

[0173] The multilayer perceptron submodule is used to perform prediction processing on the fused feature vector using a multilayer perceptron to obtain the position prediction result of the tracked object.

[0174] Optionally, the location prediction result includes a predicted bounding box and the foreground probability corresponding to the predicted bounding box, and the device may further include:

[0175] The Hanning window module is used to calculate the edge penalty score for each predicted bounding box using the Hanning window function;

[0176] The prediction score calculation module is used to perform a weighted calculation on the edge penalty score and foreground probability for each predicted bounding box to obtain the prediction score for each predicted bounding box.

[0177] The final location prediction result determination module is used to determine the final location prediction result of the tracked object as the prediction bounding box with the highest prediction score based on the prediction score.

[0178] Optionally, the frame containing the predicted location of the tracked object is the next frame of the search area image, and the device may further include:

[0179] Based on the location prediction results, the motion trajectory of the tracked object in the video stream is predicted.

[0180] Reference Figure 7 The electronic device 600 may include one or more of the following components: a processing component 602, a memory 604, a power supply component 606, a multimedia component 608, an audio component 610, an input / output (I / O) interface 612, a sensor component 614, and a communication component 616.

[0181] Processing component 602 typically controls the overall operation of electronic device 600, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the target tracking method described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0182] Memory 604 is configured to store various types of data to support the operation of electronic device 600. Examples of this data include instructions for any application or method operating on electronic device 600, contact data, phonebook data, messages, pictures, videos, etc. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0183] Power supply component 606 provides power to various components of electronic device 600. Power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 600.

[0184] Multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0185] Audio component 610 is configured to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) configured to receive external audio signals when electronic device 600 is in an operating mode, such as call mode, recording mode, and voice information processing mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0186] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0187] Sensor assembly 614 includes one or more sensors for providing state assessments of various aspects of electronic device 600. For example, sensor assembly 614 may detect the on / off state of electronic device 600, the relative positioning of components such as the display and keypad of device 600, changes in position of electronic device 600 or a component of electronic device 600, the presence or absence of user contact with electronic device 600, orientation or acceleration / deceleration of electronic device 600, and temperature changes of electronic device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0188] Communication component 616 is configured to facilitate wired or wireless communication between electronic device 600 and other devices. Electronic device 600 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency information processing (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0189] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the target tracking method described above.

[0190] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, which can be executed by a processor 620 of an electronic device 600 to complete the target tracking method described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0191] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device (server or terminal), enables the processor to perform the target tracking method described above.

[0192] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0193] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0194] The above provides a detailed description of the target tracking method, apparatus, electronic device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A target tracking method, characterized in that, The method includes: A first image pair of the tracked object is acquired, the first image pair including a reference image and a search region image, the first image pair being used to predict the position of the tracked object in the next frame of the search region image; Multi-level feature extraction is performed on the first image pair to obtain feature information of the first image pair at at least two scales; A deep aggregation network is used to fuse the feature information at at least two scales to obtain the feature extraction result of the first image pair. The feature extraction result includes a first feature vector of the reference image and a second feature vector of the search region image. Based on the multi-head attention mechanism, the first feature vector and the second feature vector are fused to obtain the fused feature vector of the first image pair. The fused feature vector is subjected to prediction processing to obtain the position prediction result of the tracked object; The step of performing multi-level feature extraction on the first image pair to obtain feature information of the first image pair at at least two scales includes: The reference image and the search region image are downsampled at at least two scales to obtain a first sampled image of the reference image at each scale and a second sampled image of the search region image at each scale. The residual network is used to extract features from the first sampled image and the second sampled image to obtain a first feature map of the reference image at each scale and a second feature map of the search region image at each scale; The location prediction result includes a predicted bounding box and the foreground probability corresponding to the predicted bounding box; the method further includes: The edge penalty score for each predicted bounding box is calculated using the Hanning window function. For each predicted bounding box, the edge penalty score and foreground probability are weighted and calculated to obtain the prediction score for each predicted bounding box. Based on the prediction score, the final position prediction result of the tracked object is determined as the prediction bounding box with the highest prediction score.

2. The method according to claim 1, characterized in that, The acquisition of the first image pair of the tracked object includes: Acquire the initial and target images of the tracked object; The initial image is cropped to obtain a reference image including the tracked object; The target image is cropped to obtain the search area image of the tracked object; The proportion of the tracked object in the image of the search area meets a preset ratio requirement.

3. The method according to claim 1, characterized in that, The step of using a deep aggregation network to fuse feature information at at least two scales to obtain the feature extraction result of the first image pair includes: The first feature map is processed by deformable convolution using a deep aggregation network to obtain the third feature map of the reference image; The second feature map is processed by deformable convolution using a deep aggregation network to obtain the fourth feature map of the search region image; The third feature map is convolved to obtain the fifth feature map, wherein the number of channels in the fifth feature map is a preset number; The fourth feature map is convolved to obtain the sixth feature map, wherein the number of channels in the sixth feature map is a preset number. The fifth and sixth feature maps are flattened to obtain the first feature vector of the reference image and the second feature vector of the search region image.

4. The method according to claim 1, characterized in that, The feature fusion of the first feature vector and the second feature vector based on the multi-head attention mechanism to obtain the fused feature vector of the first image pair includes: The first feature vector is encoded based on a multi-head self-attention mechanism to obtain the first encoded vector of the first feature vector. The second feature vector is encoded based on a multi-head self-attention mechanism to obtain the second encoded vector of the second feature vector. The first and second encoded vectors are fused and decoded based on a multi-head cross-attention mechanism to obtain a fused feature vector of the first and second feature vectors.

5. The method according to claim 4, characterized in that, The process of fusing and decoding the first and second encoded vectors based on a multi-head cross-attention mechanism to obtain a fused feature vector of the first and second feature vectors includes: Spatial position encoding is performed on the first encoding vector and the second encoding vector to obtain a third encoding vector and a fourth encoding vector. The third encoding vector includes the spatial position encoding of the first encoding vector and the first encoding vector, and the fourth encoding vector includes the spatial position encoding of the second encoding vector and the second encoding vector. Using the second encoding vector, the third encoding vector, and the fourth encoding vector as input, a multi-head cross-attention network is used to calculate the first fused feature vector of the first feature vector and the second feature vector; The first fused feature vector is processed using a feedforward neural network in residual form to obtain the second fused feature vector; The fusion feature vector of the first feature vector and the second feature vector is calculated based on the first fusion feature vector and the first fusion feature vector.

6. The method according to claim 1, characterized in that, The step of performing prediction processing on the fused feature vector to obtain the position prediction result of the tracked object includes: The fused feature vector is processed using a multilayer perceptron to obtain the position prediction result of the tracked object.

7. The method according to claim 1, characterized in that, The frame containing the predicted location of the tracked object is the next frame of the search area image, and the method further includes: Based on the location prediction results, the motion trajectory of the tracked object in the video stream is predicted.

8. A target tracking device, characterized in that, The device includes: The image pair acquisition module is used to acquire a first image pair of the tracked object, the first image pair including a reference image and a search area image, and the first image pair is used to predict the position of the tracked object in the next frame of the search area image; A multi-level feature extraction module is used to perform multi-level feature extraction on the first image pair to obtain feature information of the first image pair at at least two scales. The deep aggregation module is used to perform feature fusion on feature information at at least two scales using a deep aggregation network to obtain the feature extraction result of the first image pair. The feature extraction result includes a first feature vector of the reference image and a second feature vector of the search region image. The feature fusion module is used to fuse the first feature vector and the second feature vector based on a multi-head attention mechanism to obtain the fused feature vector of the first image pair. The prediction module is used to perform prediction processing on the fused feature vector to obtain the position prediction result of the tracked object; The multi-level feature extraction module includes: The downsampling submodule is used to downsample the reference image and the search region image at at least two scales to obtain a first sampled image of the reference image at each scale and a second sampled image of the search region image at each scale. The residual network submodule is used to extract features from the first sampled image and the second sampled image using a residual network to obtain a first feature map of the reference image at each scale and a second feature map of the search region image at each scale. The Hanning window module is used to calculate the edge penalty score for each predicted bounding box using the Hanning window function; The prediction score calculation module is used to perform a weighted calculation on the edge penalty score and foreground probability for each predicted bounding box to obtain the prediction score for each predicted bounding box. The final location prediction result determination module is used to determine the final location prediction result of the tracked object as the prediction bounding box with the highest prediction score based on the prediction score.

9. The apparatus according to claim 8, characterized in that, The deep aggregation module includes: The first upsampling fusion submodule is used to perform deformable convolution processing on the first feature map using a deep aggregation network to obtain the third feature map of the reference image; The second upsampling fusion submodule is used to perform deformable convolution processing on the second feature map using a deep aggregation network to obtain the fourth feature map of the search region image; The first channel reduction submodule is used to perform convolution processing on the third feature map to obtain the fifth feature map, wherein the number of channels in the fifth feature map is a preset number; The second channel reduction submodule is used to perform convolution processing on the fourth feature map to obtain the sixth feature map, wherein the number of channels in the sixth feature map is a preset number. The flattening submodule is used to flatten the fifth feature map and the sixth feature map respectively to obtain the first feature vector of the reference image and the second feature vector of the search region image.

10. The apparatus according to claim 8, characterized in that, The feature fusion module includes: The first encoding submodule is used to encode the first feature vector based on a multi-head self-attention mechanism to obtain the first encoded vector of the first feature vector; The first encoding submodule is used to encode the second feature vector based on a multi-head self-attention mechanism to obtain the second encoding vector of the second feature vector; The fusion decoding submodule is used to perform fusion decoding processing on the first encoding vector and the second encoding vector based on a multi-head cross-attention mechanism to obtain a fused feature vector of the first feature vector and the second feature vector.

11. An electronic device, characterized in that, include: A processor and a memory, wherein the processor executes a computer program stored in the memory to implement the target tracking method according to any one of claims 1 to 7.

12. A readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the device, the device is able to perform the target tracking method as described in any one of method claims 1 to 7.

Images