Target tracking method and device, mobile equipment, storage medium and program product

Abstract

The invention relates to the technical field of image processing, and discloses a target tracking method and device, mobile equipment, a storage medium and a program product, and the method comprises the steps: inputting a target template image and a search region image into a trained target tracking network, and obtaining a target template feature and a search region feature; filtering the target template features to obtain target features; performing fine-grained matching based on the target features and the search area features to obtain a response diagram; and performing feature classification and feature regression based on the response graph, and obtaining a target tracking result according to the obtained classification feature graph and regression feature graph. According to the method, the target template features are filtered to obtain the target features, interference of background clutters and similarity interferents on matching is filtered, the method can adapt to changes of different scenes, the detection accuracy is improved, an anchor frame does not need to be preset in the tracking process, the flexibility is higher, and the method is concise and effective.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a target tracking method, device, mobile device, storage medium and program product. Background Art

[0002] Computer vision is an important branch of artificial intelligence, which aims to study how to make computers perceive, analyze and process the real world intelligently like the human visual system. Object tracking is a basic and challenging task in computer vision. Object tracking means that given the position and state of the target in the initial frame of a video sequence, the tracker needs to predict the position and state of the target in each subsequent frame.

[0003] In recent years, target tracking has received extensive attention and has made great progress due to its wide applications in intelligent transportation, human-computer interaction, intelligent monitoring and other fields. Among different types of tracking algorithms, tracking algorithms based on deep convolutional neural networks have gradually replaced traditional manual feature tracking algorithms and become the mainstream in the tracking field due to their powerful feature extraction capabilities and efficient tracking methods.

[0004] However, in real-world tracking scenarios, it is difficult for the tracker to adapt to changes in the scene due to interference from adverse factors such as camera occlusion, background clutter, rapid motion, and lighting changes. Summary of the invention

[0005] In view of this, the present invention provides a target tracking method, apparatus, mobile device, storage medium and program product to solve the problem that traditional target tracking methods are easily interfered with, making it difficult for the tracker to adapt to different scenarios.

[0006] In a first aspect, the present invention provides a target tracking method, the method comprising:

[0007] Input the target template image and the search area image into the trained target tracking network to obtain the target template features and the search area features;

[0008] Perform feature filtering on the target template features to obtain the target features;

[0009] Perform fine-grained matching based on target features and search area features to obtain a response map;

[0010] Feature classification and feature regression are performed based on the response map, and the target tracking result is obtained according to the obtained classification feature map and regression feature map.

[0011] Beneficial effects: The features of the target template image and the search area image are extracted respectively, and then the target template features are filtered to obtain the target features, so as to filter the interference of background clutter and similar interference objects on the matching process, and can adapt to the changes of different scenes, thereby improving the detection accuracy. Then, fine-grained matching is performed based on the target features and the search area features, and classification regression is performed according to the obtained response map to achieve target tracking. There is no need to preset anchor frames during the tracking process, which is more flexible, concise and effective.

[0012] In an optional implementation, feature filtering is performed on the target template feature to obtain the target feature, including:

[0013] Map the preset target detection frame to the target template feature to obtain the feature of the region of interest;

[0014] Perform bilinear interpolation on the features of the region of interest to obtain the target features.

[0015] Beneficial effects: By extracting the features of the target template and performing bilinear interpolation on the region of interest, the matching area in the subsequent search area matching is reduced, and the influence of interference factors such as background clutter on the matching accuracy is reduced.

[0016] In an optional implementation, the target template image and the search area image are input into a trained target tracking network to obtain target template features and search area features, including:

[0017] Inputting the target template image into the template branch of the target tracking network, and inputting the search area image into the search branch of the target tracking network; wherein the template branch and the search branch are mobile networks with the same structure and shared parameters;

[0018] Feature extraction is performed respectively according to the template branch and the search branch to obtain target template features corresponding to the target template image and search area features corresponding to the search area image.

[0019] Beneficial effects: The template branch and the search branch extract the features of the target template image and the search area image by sharing parameters, effectively reducing the number of model parameters, improving the generalization ability and computational efficiency of the model, and thus achieving rapid positioning and accurate tracking of the target.

[0020] In an optional embodiment, the mobile network includes a depth-separable convolutional layer; before inputting the target template image into the template branch of the target tracking network and inputting the search area image into the search branch of the target tracking network, the method further includes:

[0021] Get the remaining resources of the current device;

[0022] According to the size of the remaining resources, the width multiplication factor and the resolution multiplication factor of the depthwise separable convolutional layer are adjusted; the width multiplication factor is used to adjust the number of channels when extracting features of the mobile network, and the resolution multiplication factor is used to adjust the resolution of the input image of the mobile network.

[0023] Beneficial effect: The resolution of the network input image and the number of channels of the network during feature extraction are adjusted according to the remaining resource size of the current device, which reduces the number of parameters and calculations while maintaining performance, facilitating lightweight deployment of the network.

[0024] In an optional implementation, feature classification and feature regression are performed based on the response graph, and a target tracking result is obtained according to the obtained classification feature graph and regression feature graph, including:

[0025] Based on the response map, feature classification is performed to obtain a two-channel classification feature map; the two channels of the classification feature map are used to represent the foreground classification score and the background classification score respectively;

[0026] The target bounding box is predicted based on the response map to obtain a four-channel regression feature map; the four channels of the regression feature map are used to represent the distance from the target center to the four borders of the target bounding box;

[0027] The classification feature map and the regression feature map are combined to obtain the target tracking result.

[0028] Beneficial effect: By performing feature classification and feature regression on the response map respectively, a two-channel classification feature map and a four-channel regression feature map are obtained, so that the classification feature map and the regression feature map can be combined to obtain the detection result and realize target tracking.

[0029] In an optional implementation, the classification feature map and the regression feature map are combined to obtain a target tracking result, including:

[0030] Based on the area with the highest foreground classification score in the classification feature map, the target position is obtained;

[0031] Determine the target bounding box based on the regression feature map and the target center corresponding to the target position in the regression feature map;

[0032] Based on the target position and target bounding box, the target tracking result is obtained.

[0033] Beneficial effect: By combining the classification feature map and the regression feature map, the area with the highest foreground score in the foreground classification feature is selected as the target position, and the distance from the target position to the predicted bounding box is obtained through the regression feature map to generate the final detection result. There is no need to preset anchor frames during the tracking process, which is more flexible and saves storage and computing resources.

[0034] In an optional embodiment, before performing feature classification and feature regression based on the response graph, the method further includes:

[0035] Based on the preset cosine window function, the weight of the response graph is adjusted.

[0036] Beneficial effects: The response graph is processed through a cosine window to improve the stability of the target tracking process and the accuracy of the results.

[0037] In an optional implementation, fine-grained matching is performed based on the target features and the search area features to obtain a response map, including:

[0038] Convolve the target features using the preset convolution layer;

[0039] The convolved target features are used as the convolution kernel, and the search area features are convolved channel by channel to obtain the response map.

[0040] Beneficial effect: The target features are convolved, and the convolved target features are used as convolution kernels to perform channel-by-channel convolution on the search area features, so that the generated response map contains rich target template feature information.

[0041] In an optional implementation, before inputting the target template image and the search area image into the trained target tracking network, the method further includes:

[0042] Construct the target loss function based on the cross entropy loss and intersection-over-union loss;

[0043] The initial target tracking network is trained based on the target loss function to obtain a trained target tracking network.

[0044] Beneficial effect: A target loss function is constructed based on the cross entropy loss and the intersection-over-union loss, so as to train the initial target tracking network so as to use the trained target tracking network for target tracking.

[0045] In a second aspect, the present invention provides a target tracking device, the device comprising:

[0046] The first processing module is used to input the target template image and the search area image into the trained target tracking network to obtain the target template features and the search area features;

[0047] The second processing module is used to perform feature filtering on the target template feature to obtain the target feature;

[0048] The third processing module is used to perform fine-grained matching based on the target features and the search area features to obtain a response map;

[0049] The fourth processing module is used to perform feature classification and feature regression based on the response map, and obtain the target tracking result according to the obtained classification feature map and regression feature map.

[0050] In a third aspect, the present invention provides a mobile device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the target tracking method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0051] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the target tracking method of the first aspect or any corresponding embodiment thereof.

[0052] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the target tracking method of the first aspect or any corresponding embodiment thereof.

[0053] The beneficial effects of the present invention are:

[0054] The features of the target template image and the search area image are extracted respectively, and then the target template features are filtered to obtain the target features, so as to filter the interference of background clutter and similar interference objects on the matching process, and can adapt to the changes of different scenes, thereby improving the detection accuracy. Then, fine-grained matching is performed based on the target features and the search area features, and classification regression is performed according to the obtained response map to achieve target tracking. There is no need to preset anchor frames during the tracking process, which is more flexible, concise and effective. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 is a flow chart of a target tracking method according to an embodiment of the present invention;

[0057] Figure 2 is a flow chart of another target tracking method according to an embodiment of the present invention;

[0058] Figure 3 is a schematic diagram of the structure of a target tracking network according to an embodiment of the present invention;

[0059] Figure 4A is a schematic diagram of the principle of a cross-correlation matching method according to an embodiment of the present invention;

[0060] Figure 4B is a schematic diagram of the principle of a fine-grained matching method according to an embodiment of the present invention;

[0061] Figure 5 is a schematic diagram of bilinear interpolation according to an embodiment of the present invention;

[0062] Figure 6 is a schematic diagram of a principle of target detection according to an embodiment of the present invention;

[0063] Figure 7 is a flow chart of another target tracking method according to an embodiment of the present invention;

[0064] Figure 8 is a flow chart of another target tracking method according to an embodiment of the present invention;

[0065] Fig. 9 is a structural block diagram of a target tracking device according to an embodiment of the present invention;

[0066] Fig.10 It is a schematic diagram of the hardware structure of a mobile device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0068] Among different types of tracking algorithms, tracking algorithms based on deep convolutional neural networks have gradually replaced traditional manual feature tracking algorithms due to their powerful feature extraction capabilities and efficient tracking methods, becoming the mainstream in the tracking field. However, although these tracking algorithms have higher accuracy, compared with traditional algorithms, they require more storage and computing resources to complete their work better, which makes these algorithms difficult to deploy on mobile terminals or edge devices. In addition, in real-world tracking scenarios, adaptive target tracking is still a big challenge due to adverse factors such as camera occlusion, background clutter, rapid motion, and lighting changes.

[0069] At present, the search area to be tracked and the template area image can be input into the network for cross-correlation matching, multiple predicted target candidate frames can be obtained, and the position and state of the target in the search area can be finally determined through screening. However, the traditional cross-correlation matching algorithm is easily affected by background clutter and similar interference objects.

[0070] In addition, the location and size of the target can be determined during the tracking process through the classification branch and regression branch of the anchor-based Region Proposal Network (RPN). However, the preset anchor points and anchor boxes not only limit the flexibility of the tracking process, but also the classification and regression of a large number of preset anchor boxes causes a waste of storage resources and computing resources.

[0071] Therefore, an embodiment of the present invention provides a target tracking method. After extracting features from a target template image and a search area image, the target template features are first filtered to obtain target features, so as to filter out interference from background clutter and similar interference objects on target matching, and can adapt to changes in different scenes, thereby improving detection accuracy. Then, fine-grained matching is performed based on the target features and the search area features, and classification regression is performed based on the obtained response map to achieve target tracking, and there is no need to preset anchor frames during the tracking process, which is more flexible and saves storage and computing resources.

[0072] According to an embodiment of the present invention, an embodiment of a target tracking method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0073] In this embodiment, a target tracking method is provided, which can be used for mobile devices or interrupts, such as mobile phones, tablet computers, etc. Figure 1 is a flow chart of a target tracking method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0074] Step S101, input the target template image and the search area image into the trained target tracking network to obtain the target template features and the search area features.

[0075] In this embodiment, the target tracking network can track the preset tracking target through an end-to-end tracking method. The algorithm of the target tracking network mainly includes three modules: a twin network backbone, a fine-grained matching module, and an adaptive classification regression module.

[0076] Specifically, the backbone of the twin network is composed of a convolutional neural network, which is mainly used to extract features from the target template image and the search area image to obtain the target template features and the search area features. The feature extraction process is one of the key steps in target tracking and greatly affects the performance and accuracy of the algorithm. Therefore, choosing a suitable backbone network is crucial to the performance of the twin network. When selecting a backbone network, it is necessary to consider the lightweight and adaptability of the network to ensure that the algorithm can be applied in real time on embedded devices and mobile devices.

[0077] Step S102, filtering the target template features to obtain target features.

[0078] It should be noted that, in general, after the target template image and the search area image are feature extracted, the target template image features need to be used to match the target in the search area features and generate a response map for detection. For example, after the target and template features are extracted, the traditional target tracking algorithm uses the target template features as the convolution kernel and performs a convolution operation in the search area to match the target. However, the target template features contain not only the target features, but also a large amount of background clutter and similar interferences. This information can easily have an adverse effect on the matching process, making it difficult for the algorithm to adapt to scene changes.

[0079] In an embodiment of the present invention, a target is matched in a search area based on a fine-grained matching method. Before the formal matching work is carried out, it is first necessary to perform feature filtering on the target template features to eliminate the influence of background clutter and similar interferences on the subsequent matching process, thereby improving the matching accuracy.

[0080] Step S103: Perform fine-grained matching based on the target features and the search area features to obtain a response map.

[0081] Specifically, after executing step S102, interference such as background clutter has been well filtered. At this time, the algorithm uses a fine-grained matching module and uses the target template features as the convolution kernel to match the target in the search area by channel-by-channel convolution and generate a response map containing rich target template features.

[0082] Step S104, performing feature classification and feature regression based on the response graph, and obtaining a target tracking result according to the obtained classification feature graph and regression feature graph.

[0083] Specifically, the algorithm's adaptive classification and regression module performs target detection based on a frameless approach. It does not need to preset anchor frames during tracking, but directly performs classification and regression on feature maps to achieve detection, which is more flexible and effective. The adaptive classification and regression module takes the response map generated by the fine-grained matching module as input to generate the final classification feature map and regression feature map.

[0084] Specifically, the adaptive classification regression module is divided into a classification branch and a regression branch. The classification branch is responsible for obtaining the foreground classification information map and the background classification information map of the target. The higher the score of the corresponding position on the foreground classification information map, the greater the probability that this position is the target. The above method can obtain the prediction information of the target position. The regression branch also takes the response map as input. According to the response map, it directly regresses the distances from different center points on the response map to the surroundings of the predicted bounding box, and indirectly obtains the predicted bounding box based on the predicted distance. Finally, the classification score map and the regression boundary map are combined to obtain the final tracking result.

[0085] The target tracking method provided in this embodiment extracts the features of the target template image and the search area image respectively, and then performs feature filtering on the target template features to obtain target features, so as to filter the interference of background clutter and similar interference objects on the matching process, and can adapt to the changes of different scenes, thereby improving the detection accuracy. Then, fine-grained matching is performed based on the target features and the search area features, and classification regression is performed based on the obtained response map to achieve target tracking, without the need to preset anchor frames during the tracking process, which is more flexible, concise and effective.

[0086] In this embodiment, a target tracking method is provided, which can be used for mobile devices or interrupts, such as mobile phones, tablet computers, etc. Figure 2 is a flow chart of a target tracking method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0087] Step S201, input the target template image and the search area image into the trained target tracking network to obtain the target template features and the search area features.

[0088] Specifically, the above step S201 includes:

[0089] Step S2011, input the target template image into the template branch of the target tracking network, and input the search area image into the search branch of the target tracking network.

[0090] Specifically, Figure 3 As shown in the figure, the template branch and the search branch are mobile networks (MobileNet) with the same structure and shared parameters. That is, the twin network backbone consists of a template branch and a search branch, and these two branches share the same network structure and parameters. The twin network backbone can be implemented by using MobileNet. MobileNet is a widely used lightweight convolutional neural network that uses a deep separable convolutional structure, which greatly reduces the number of parameters and computation while maintaining performance, which is conducive to lightweight deployment of the network.

[0091] In an embodiment of the present invention, MobileNet in the algorithm is only used for feature extraction, and does not provide detection work, that is, the twin network backbone does not include the pooling layer, fully connected layer and softmax classification layer of MobileNet. Only the feature map extracted by the network features is used for subsequent matching and classification regression, thereby further reducing the storage and computing resources occupied by the model.

[0092] Furthermore, the spatial information of the search area image is crucial for the matching and detection of the target position, see again Figure 3 ,This embodiment removes multiple downsampling operations of MobileNet, so that the feature map resolutions of the template branch and the search branch are maintained at 15×15 and 31×31 respectively, so as to improve the algorithm’s perception of the target spatial position in the subsequent matching and detection processes.

[0093] In some optional implementations, the mobile network includes a depthwise separable convolutional layer. Before executing step S2011, the remaining resources of the current device may be obtained, and the width multiplier and resolution multiplier of the depthwise separable convolutional layer may be adjusted according to the size of the remaining resources. The width multiplier is used to adjust the number of channels when extracting features from the mobile network, and the resolution multiplier is used to adjust the resolution of the input image of the mobile network.

[0094] This embodiment adjusts the resolution of the network input image and the number of channels of the network during feature extraction in combination with the remaining resource size of the current device, thereby reducing the number of parameters and calculations while maintaining performance, which is conducive to lightweight deployment of the network.

[0095] Step S2012, performing feature extraction according to the template branch and the search branch respectively, to obtain the target template features corresponding to the target template image and the search area features corresponding to the search area image.

[0096] Specifically, the template branch and the search branch can extract the features of the target template image and the search area image by sharing parameters, thereby realizing rapid positioning and tracking of the target. Among them, the template branch is responsible for processing the target template image, while the search branch processes the search area image. The feature maps output by the two branches will be used for subsequent similarity matching and target tracking tasks. The feature sharing mechanism in the feature extraction process can effectively reduce the number of parameters of the model and improve the generalization ability and computational efficiency of the model. By sharing the network structure and parameters, effective information transfer and learning can be carried out between different images and targets, thereby realizing accurate tracking of the target.

[0097] Step S202, filtering the target template features to obtain target features.

[0098] Specifically, the above step S202 includes:

[0099] Step S2021, mapping the preset target detection frame to the target template feature to obtain the region of interest feature.

[0100] Specifically, after the feature extraction of the twin network backbone, the target template features and search area features need to be sent to the fine-grained matching module to match the target. In traditional twin network-based target tracking algorithms, the matching algorithm is usually a cross-correlation matching algorithm, such as Figure 4A As shown in , the template feature is used as the convolution kernel to perform channel-by-channel convolution on the search area to generate the final response map. This embodiment uses a new fine-grained matching method, such as Figure 4B As shown, after obtaining the target template features, the target features are extracted through ROI_ALINE operation (ALINE uses bilinear interpolation) according to the preset target detection frame and target template features.

[0101] Specifically, the preset target detection box position is mapped to the target template feature map through the Region of Interest Align operation, and then the region of interest features are extracted from the template features of the entire input target image. The mapping method is shown in the following formula:

[0102]

[0103] Among them, w im Indicates the width of the original image, h im Represents the height of the original image, i, j are the horizontal and vertical coordinates of the current position on the feature map, w, h are the width and height of the feature map, and s is the total step size of the network convolution.

[0104] Step S2022, performing bilinear interpolation on the features of the region of interest to obtain target features.

[0105] Specifically, an Aline operation is performed on the features of the region of interest (Aline uses a bilinear interpolation method), that is, key features in the features of the region of interest are further extracted, such as Figure 5 As shown in the figure, the matching area is reduced by bilinear interpolation. In the two-dimensional grid, assuming that the coordinates of the known point are (x 1 ,y 1 ), (x 1 ,y 2 ), (x 2 ,y 1 ), (x 2 ,y 2 ), and the coordinates of the unknown point are (x, y), then the horizontal interpolation of bilinear interpolation can be expressed as:

[0106]

[0107]

[0108] Then, the value at the final position (x, y) can be expressed as:

[0109]

[0110] Among them, f(x 1 ,y 1 )、f(x 2 ,y 1 )、f(x 1 ,y 2 )、f(x 2 ,y 2 ) represent the values ​​of the corresponding coordinate positions.

[0111] See again Figure 5 ,Through bilinear interpolation, the value of the center of each position after the feature map is evenly divided into nine parts can be obtained, so as to narrow the features of the matching target template and reduce the impact of background clutter on the matching.

[0112] In the embodiment of the present application, by performing region of interest feature extraction and bilinear interpolation on the target template features, the matching area in the subsequent search area matching is reduced, and the influence of interference factors such as background clutter on the matching accuracy is reduced.

[0113] Step S203: Perform fine-grained matching based on the target features and the search area features to obtain a response map.

[0114] Specifically, the above step S203 includes:

[0115] Step S2031, convolve the target features using a preset convolutional layer.

[0116] Specifically, for the matching operation, the target features extracted by bilinear interpolation are first passed through a preset convolution layer with a convolution kernel of 3×3 to adapt to the subsequent matching operation. At this time, the resolution of the target features is refined to a size of 1×1.

[0117] Step S2032, using the convolved target features as the convolution kernel, performing channel-by-channel convolution on the search area features to obtain a response map.

[0118] Specifically, in the matching process, the convolved target features are used as the convolution kernel, and the search area features are convolved channel by channel to generate a response map. After the above operation, each channel and each point of the response map contains the template feature information of the target.

[0119] Step S204, performing feature classification and feature regression based on the response graph, and obtaining a target tracking result according to the obtained classification feature graph and regression feature graph.

[0120] Specifically, the above step S204 includes:

[0121] Step S2041, performing feature classification based on the response map to obtain a two-channel classification feature map; the two channels of the classification feature map are used to represent the foreground classification score and the background classification score respectively.

[0122] Specifically, before executing step S204, the weight of the response map can also be adjusted based on a preset cosine window function. In order to improve the stability of the target tracking algorithm and reduce some poor tracking results, the response map is processed by a cosine window. The cosine window penalty applies a cosine window function to each pixel value of the response map to adjust the weight of the response map. The shape of this window function is elliptical, the center position is aligned with the target position, and there is a higher weight at the center, and the weight is gradually reduced outward, thereby reducing the response value of the edge value and improving stability.

[0123] Specifically, the response map obtained after matching is sent to the adaptive classification and regression module to obtain the final tracking result. The adaptive classification and regression module is divided into two branches: classification and regression. It should be noted that after feature extraction, the target template image and the search area image are matched separately for the classification branch and the regression branch. This operation can adapt the matching process to different detection tasks.

[0124] like Figure 6 As shown in the figure, when the adaptive classification and regression module detects the target, it does not detect the target by presetting the anchor box, but directly detects the target position and edge in a pixel-by-pixel manner. For example, assuming that (x, y) is any point on the template feature of the search area, it can be based on Reverse mapping to the original image position.

[0125] Specifically, the classification branch predicts the probability that the current position is the foreground, see again Figure 3 The algorithm directly generates a two-channel classification feature map. The two channels of this feature map are the foreground classification score and background classification score of the target. In the foreground classification score map, the higher the score of the point, the greater the probability of being the target.

[0126] Step S2042, predicting the target bounding box based on the response map to obtain a four-channel regression feature map; the four channels of the regression feature map are used to represent the distance from the target center to the four borders of the target bounding box.

[0127] Specifically, for the regression feature map, the task of regression is to regress the distance from the current position to the predicted bounding box. If d = (d l , d t , d r , d b ) represents the distance of the predicted four borders, assuming that Represent the upper left corner and lower right corner of the true bounding box respectively, then this regression task can be expressed as:

[0128]

[0129]

[0130]

[0131]

[0132] Step S2043, combining the classification feature map and the regression feature map to obtain the target tracking result.

[0133] Specifically, the target position is obtained based on the area with the highest foreground classification score in the classification feature map. According to the regression feature map and the target center corresponding to the target position in the regression feature map, the target bounding box is determined based on the target position and the target bounding box to obtain the target tracking result. By combining the classification feature map and the regression feature map, the area with the highest foreground score in the foreground classification feature is selected as the target position, and the distance from the target position to the predicted bounding box is obtained through the regression feature map of the area to generate the final detection result. There is no need to preset the anchor frame during the tracking process, which is more flexible and saves storage and computing resources.

[0134] In an embodiment of the present application, by performing feature classification and feature regression on the response map respectively, a two-channel classification feature map and a four-channel regression feature map are obtained, so that the classification feature map and the regression feature map are combined to obtain the detection result and realize target tracking.

[0135] The target tracking method provided in this embodiment inputs the target template image to be tracked and the search area image to be detected into the tracking network. The target template image and the search area image are successively subjected to feature extraction of the twin network backbone, matching of the fine-grained matching module, and classification regression of the classification regression network, and then a two-channel foreground / background classification score map and a four-channel regression feature map are generated, and the final target tracking result is generated by combining the two. The tracking algorithm runs in an end-to-end manner, and the final tracking result can be generated without human intervention in the middle.

[0136] In this embodiment, a target tracking method is provided, which can be used for mobile devices or interrupts, such as mobile phones, tablet computers, etc. Figure 7is a flow chart of a target tracking method according to an embodiment of the present invention. Figure 7 As shown, the process includes the following steps:

[0137] Step S701, constructing a target loss function according to the cross entropy loss and the intersection-over-union loss.

[0138] Specifically, during the training process, the classification loss and regression loss are unified, and the cross entropy loss is used as the classification loss, which can be expressed as:

[0139]

[0140] Among them, p x，y Represents the classification score at position (x, y) on the classification feature map, N pos is the number of positive samples, L cls represents the classification loss, Represents the supervision value of the classification feature map position (x, y), that is, the true value.

[0141] Furthermore, the intersection over union (IOU) loss is used as the regression loss and is expressed as:

[0142]

[0143] Among them, t x，y represents the classification score at position (x, y) on the regression feature map, L reg represents the regression loss, Represents the supervision value of the position (x, y) on the regression feature map, that is, the true value.

[0144] This embodiment selects some positive samples for training, and the regression loss is only calculated for the positive sample positions to avoid the impact of negative values ​​on regression. The target loss function of the entire network can be set as:

[0145] L=λ 1 L cls +λ 2 L reg

[0146] Among them, λ 1 =λ 2 = 1. For the weight problem, the classification regression weight is not trained, but the parameter is simply set to λ 1 =λ 2 =1, and get the loss of the entire network.

[0147] Step S702: training the initial target tracking network based on the target loss function to obtain a trained target tracking network.

[0148] Specifically, multiple video sequences are used for training, and two frames are randomly selected from a single video sequence each time as the target template image and the search area image for training, until all frames are trained. The initial parameters of the twin network backbone are used as the MobileNet pre-trained on ImageNet, and the parameters of the fine-grained matching module and the parameters of the adaptive classification and regression module are randomly initialized during the first training.

[0149] Furthermore, we train for 20 epochs, and in the first 10 epochs, we lock the parameters of MobileNet and do not train them, but only train the parameters of the fine-grained matching module and the parameters of the adaptive classification and regression module. In the last 10 epochs, we unlock the parameters of MobileNet and fine-tune them to suit the tracking task.

[0150] This embodiment constructs a target loss function based on the cross entropy loss and the intersection-over-union loss, so as to train the initial target tracking network, so as to use the trained target tracking network to perform target tracking.

[0151] Step S703: Input the target template image and the search area image into the trained target tracking network to obtain the target template features and the search area features. Figure 2 Step S201 of the illustrated embodiment will not be described in detail here.

[0152] Step S704: filter the target template features to obtain target features. Figure 2 Step S202 of the illustrated embodiment will not be described in detail here.

[0153] Step S705: Perform fine-grained matching based on the target features and the search area features to obtain a response map. Figure 2 Step S203 of the illustrated embodiment will not be described in detail here.

[0154] Step S706: Perform feature classification and feature regression based on the response graph, and obtain the target tracking result according to the obtained classification feature graph and regression feature graph. Figure 2 Step S204 of the illustrated embodiment will not be described in detail here.

[0155] The target tracking method provided in this embodiment constructs a target loss function based on cross entropy loss and intersection-over-union loss, thereby training the initial target tracking network, so as to use the trained target tracking network to extract the features of the target template image and the search area image respectively, and then perform feature filtering on the target template features to obtain target features, so as to filter the interference of background clutter and similar interference objects on the matching process, and can adapt to the changes of different scenes, thereby improving the detection accuracy. Then, fine-grained matching is performed based on the target features and the search area features, and classification regression is performed based on the obtained response map to achieve target tracking, without the need to preset anchor frames during the tracking process, which is more flexible, concise and effective.

[0156] The target tracking scheme of the present invention is described in detail below with reference to a specific application example. Figure 8 As shown, the application example includes the following steps:

[0157] Step 1: Input the target template image and the search area image into the template branch and the search branch of the tracking algorithm respectively for feature extraction.

[0158] The backbone network of the tracking algorithm is composed of convolutional neural networks. This choice directly affects the tracking efficiency and tracking accuracy of the algorithm. Traditional high-precision target tracking algorithms are usually composed of extremely deep convolutional neural networks such as ResNet and RestNeXt, but this choice makes it difficult to apply the tracking algorithm to mobile terminals or edge devices. This application example uses MobileNet as the backbone of the twin network. The MobileNet architecture designs a depth-separable convolution layer and sets a width multiplier (WidthMultiplier) and a resolution multiplier (Resolution Multiplier). The width multiplier controls the number of channels in each layer of the network, and reduces the number of parameters and calculations of the model by reducing the number of channels. The resolution multiplier is used to reduce the resolution of the input image, thereby reducing the input size of the network and further reducing the computational complexity.

[0159] See again Figure 3 , the target template image and the search area image first need to be extracted through the feature extraction of the twin network backbone. Figure 3 It can be seen that the main body of the twin network is divided into a template branch and a search branch. These two branches are composed of two parallel deep convolutional neural networks with the same structure, and the two branches share the parameters of the convolutional neural network to ensure that the same steps of feature extraction are performed on the target template and the search area image. During the tracking process, the target template image can be temporarily stored after feature extraction by the target template branch. At this time, only the search branch is running. This method can save computational complexity and improve the efficiency of the algorithm.

[0160] Step 2: Send the target template image features and the search area image features into the matching algorithm for fine-grained matching to generate a response map.

[0161] After obtaining the target template features, the target features are extracted through the ROI_ALINE operation (ALINE uses bilinear interpolation) according to the real detection box on the target template to reduce the matching area when matching in the search area.

[0162] Step 3: Send the response map to the adaptive classification and regression module for classification and bounding box regression, and combine the results of the two to generate the tracking result.

[0163] In addition, when training the tracking algorithm, for the classification branch, the algorithm uses the cross entropy loss as the loss function, and for the regression branch, the algorithm uses the intersection-over-union loss as the loss function. cls represents the classification loss, L reg As a regression loss, the comprehensive loss of the entire network can be expressed as L = λ 1 L cls +λ 2 L reg .

[0164] The present invention inputs the target template to be tracked and the search area image to be detected into the tracking network. The target template image and the search area image are sequentially subjected to feature extraction of the twin network backbone, similarity matching function matching, and classification regression of the classification regression network to generate a two-channel foreground and background classification score map and a four-channel regression feature map, and the final tracking result is generated by combining the two. The tracking algorithm runs in an end-to-end manner, and the final tracking result can be generated without human intervention in the middle.

[0165] The present invention proposes a lightweight tracking algorithm with higher matching accuracy and a simpler and more effective target detection method, which can reduce the impact of background clutter and similar interferers on the tracking process while maintaining its lightweight, and can improve the efficiency, accuracy and flexibility of target tracking through a simpler target detection method.

[0166] In this embodiment, a target tracking device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0167] This embodiment provides a target tracking device, such as Fig. 9 As shown, including:

[0168] The first processing module 901 is used to input the target template image and the search area image into the trained target tracking network to obtain the target template features and the search area features;

[0169] The second processing module 902 is used to perform feature filtering on the target template feature to obtain the target feature;

[0170] The third processing module 903 is used to perform fine-grained matching based on the target features and the search area features to obtain a response map;

[0171] The fourth processing module 904 is used to perform feature classification and feature regression based on the response graph, and obtain the target tracking result according to the obtained classification feature graph and regression feature graph.

[0172] In some optional embodiments, the device is also used for:

[0173] Construct the target loss function based on the cross entropy loss and intersection-over-union loss;

[0174] The initial target tracking network is trained based on the target loss function to obtain a trained target tracking network.

[0175] In some optional implementations, the first processing module 901 is further configured to:

[0176] Inputting the target template image into the template branch of the target tracking network, and inputting the search area image into the search branch of the target tracking network; wherein the template branch and the search branch are mobile networks with the same structure and shared parameters;

[0177] Feature extraction is performed respectively according to the template branch and the search branch to obtain target template features corresponding to the target template image and search area features corresponding to the search area image.

[0178] In some optional embodiments, the mobile network includes a depth-separable convolutional layer; before inputting the target template image into the template branch of the target tracking network and inputting the search area image into the search branch of the target tracking network, the first processing module 901 is further used to:

[0179] Get the remaining resources of the current device;

[0180] According to the size of the remaining resources, the width multiplication factor and the resolution multiplication factor of the depthwise separable convolutional layer are adjusted; the width multiplication factor is used to adjust the number of channels when extracting features of the mobile network, and the resolution multiplication factor is used to adjust the resolution of the input image of the mobile network.

[0181] In some optional implementations, the second processing module 902 is further configured to:

[0182] Map the preset target detection frame to the target template feature to obtain the feature of the region of interest;

[0183] Perform bilinear interpolation on the features of the region of interest to obtain the target features.

[0184] In some optional implementations, the third processing module 903 is further configured to:

[0185] Convolve the target features using the preset convolution layer;

[0186] The convolved target features are used as the convolution kernel, and the search area features are convolved channel by channel to obtain the response map.

[0187] In some optional implementations, before performing feature classification and feature regression based on the response graph, the fourth processing module 904 is further used to:

[0188] Based on the preset cosine window function, the weight of the response graph is adjusted.

[0189] In some optional implementations, the fourth processing module 904 is further configured to:

[0190] Based on the response map, feature classification is performed to obtain a two-channel classification feature map; the two channels of the classification feature map are used to represent the foreground classification score and the background classification score respectively;

[0191] The target bounding box is predicted based on the response map to obtain a four-channel regression feature map; the four channels of the regression feature map are used to represent the distance from the target center to the four borders of the target bounding box;

[0192] The classification feature map and the regression feature map are combined to obtain the target tracking result.

[0193] In some optional implementations, the fourth processing module 904 is further configured to:

[0194] Based on the area with the highest foreground classification score in the classification feature map, the target position is obtained;

[0195] Determine the target bounding box based on the regression feature map and the target center corresponding to the target position in the regression feature map;

[0196] Based on the target position and target bounding box, the target tracking result is obtained.

[0197] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0198] The target tracking device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0199] The embodiment of the present invention also provides a mobile device having the above Fig. 9 The target tracking device shown.

[0200] See also Fig.10 , Fig.10 is a structural diagram of a mobile device provided by an optional embodiment of the present invention, such as Fig.10 As shown, the mobile device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the mobile device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple mobile devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.10 A processor 10 is taken as an example.

[0201] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0202] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0203] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the mobile device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the mobile device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0204] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0205] The mobile device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0206] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the mobile device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0207] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0208] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0209] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A target tracking method, It is characterized in that The method comprises: Input the target template image and the search area image into the trained target tracking network to obtain the target template features and the search area features; Performing feature filtering on the target template feature to obtain the target feature; Performing fine-grained matching based on the target features and the search area features to obtain a response map; Feature classification and feature regression are performed based on the response graph, and the target tracking result is obtained according to the obtained classification feature graph and regression feature graph.

2. The method according to claim 1, It is characterized in that The step of filtering the target template features to obtain target features includes: Mapping a preset target detection frame to the target template feature to obtain a region of interest feature; Bilinear interpolation is performed on the features of the region of interest to obtain target features.

3. The method according to claim 1, It is characterized in that The target template image and the search area image are input into the trained target tracking network to obtain the target template features and the search area features, including: Inputting a target template image into a template branch of the target tracking network, and inputting a search area image into a search branch of the target tracking network; wherein the template branch and the search branch are mobile networks with the same structure and shared parameters; Feature extraction is performed respectively according to the template branch and the search branch to obtain target template features corresponding to the target template image and search area features corresponding to the search area image.

4. The method according to claim 3, It is characterized in that The mobile network includes a depth-separable convolutional layer; before inputting the target template image into the template branch of the target tracking network and inputting the search area image into the search branch of the target tracking network, the method further includes: Get the remaining resources of the current device; According to the size of the remaining resources, adjust the width multiplication factor and the resolution multiplication factor of the depthwise separable convolutional layer; wherein the width multiplication factor is used to adjust the number of channels during feature extraction of the mobile network, and the resolution multiplication factor is used to adjust the resolution of the input image of the mobile network.

5. The method according to claim 1, It is characterized in that The performing of feature classification and feature regression based on the response graph, and obtaining a target tracking result according to the obtained classification feature graph and regression feature graph, comprises: Performing feature classification based on the response map to obtain a two-channel classification feature map; the two channels of the classification feature map are used to represent a foreground classification score and a background classification score, respectively; The target bounding box is predicted based on the response map to obtain a four-channel regression feature map; the four channels of the regression feature map are respectively used to characterize the distance from the target center to the four borders of the target bounding box; The classification feature map and the regression feature map are combined to obtain a target tracking result.

6. The method according to claim 5, It is characterized in that The step of combining the classification feature map and the regression feature map to obtain a target tracking result includes: Obtaining a target position based on the area with the highest foreground classification score in the classification feature map; Determine a target bounding box according to the regression feature map and a target center corresponding to the target position in the regression feature map; Based on the target position and the target bounding box, a target tracking result is obtained.

7. The method according to claim 1, It is characterized in that Before performing feature classification and feature regression based on the response graph, the method further includes: Based on a preset cosine window function, the weight of the response graph is adjusted.

8. The method according to claim 1, It is characterized in that The step of performing fine-grained matching based on the target features and the search area features to obtain a response graph includes: Convolving the target features using a preset convolutional layer; The target features after convolution are used as convolution kernels, and the search area features are convolved channel by channel to obtain a response map.

9. The method according to claim 1, It is characterized in that Before inputting the target template image and the search area image into the trained target tracking network, the method further includes: Construct the target loss function based on the cross entropy loss and intersection-over-union loss; The initial target tracking network is trained based on the target loss function to obtain a trained target tracking network.

10. A target tracking device, It is characterized in that The device comprises: The first processing module is used to input the target template image and the search area image into the trained target tracking network to obtain the target template features and the search area features; A second processing module is used to perform feature filtering on the target template feature to obtain the target feature; A third processing module is used to perform fine-grained matching based on the target features and the search area features to obtain a response map; The fourth processing module is used to perform feature classification and feature regression based on the response map, and obtain the target tracking result according to the obtained classification feature map and regression feature map.

11. A mobile device, It is characterized in that include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the target tracking method according to any one of claims 1 to 9 by executing the computer instructions.

12. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the target tracking method according to any one of claims 1 to 9.

13. A computer program product, It is characterized in that The method comprises computer instructions, wherein the computer instructions are used to cause a computer to execute the target tracking method according to any one of claims 1 to 9.

Images