Target tracking method, device, computer equipment and storage medium

Abstract

The present application relates to a target tracking method, device, computer equipment and storage medium. The method includes: obtaining a video frame sequence, determining a channel pruning parameter and a target depth feature map corresponding to a target area according to the convolution result of the corresponding feature of the first frame image in the video frame sequence, determining a target search area and a second depth feature map corresponding to the target search area in the target frame image, wherein the second depth feature map corresponds to n feature channels, performing channel pruning on the second depth feature map according to the channel pruning parameter to obtain a second retained feature map, wherein the second retained feature map corresponds to m feature channels, where m is less than n, reducing the amount of data involved in subsequent target positioning coordinate calculations through channel pruning, determining the positioning coordinates of the target area in the target frame image according to the depth features after channel pruning, accelerating the data processing speed of target object tracking and positioning, and improving the real-time tracking of the target object.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a target tracking method, device, computer equipment and storage medium. Background Art

[0002] Visual tracking technology is one of the hottest research directions in computer vision. Given a bounding box of a specified target object in an initial frame, visual tracking aims to continuously locate the target object in subsequent frames. Thanks to the rapid development of deep learning, deep feature maps extracted by deep networks are used to represent the appearance of targets in visual tracking instead of traditional manual features, which greatly improves the accuracy of tracking algorithms. However, due to the high-dimensionality and huge data of deep learning features, the computational complexity of the tracking algorithm is greatly increased, resulting in a slower running speed of the visual tracking algorithm and poor real-time tracking of the target object. Summary of the invention

[0003] In order to solve the technical problem that the tracking algorithm has a large amount of calculation, resulting in poor tracking real-time performance, the present application provides a target tracking method, apparatus, computer equipment and storage medium.

[0004] In a first aspect, the present application provides a target tracking method, comprising:

[0005] Acquire a video frame sequence, wherein the video frame sequence includes a plurality of continuous frame images, and the frame images include an object to be tracked;

[0006] Based on the convolution result of the corresponding features of the target area in the first frame picture, determine the channel pruning parameters and the target depth feature map corresponding to the target area, wherein the first frame picture is the frame picture located at the first position in the video frame sequence, and the target area is the area in the frame picture containing the object to be tracked;

[0007] Determine a target search area in a target frame picture and a second depth feature map corresponding to the target search area, wherein the target frame picture is any one of the frame pictures except the first frame picture, the target search area includes the target area, and the second depth feature map includes second sub-feature maps corresponding to n feature channels;

[0008] Performing channel pruning on the second depth feature map according to the channel pruning parameter to obtain a corresponding second retained feature map, wherein the second retained feature map includes second sub-feature maps corresponding to m feature channels, m is less than n, and n and m are different positive integers;

[0009] Based on the convolution result of the second retained feature map and the target depth feature map, the target positioning coordinates of the target area in the target frame image are determined.

[0010] In a second aspect, the present application provides a target tracking device, comprising:

[0011] An acquisition module, used for acquiring a video frame sequence, wherein the video frame sequence includes a plurality of continuous frame images, and the frame images include an object to be tracked;

[0012] A first determination module is used to determine the channel pruning parameters and the target depth feature map corresponding to the target area based on the convolution result of the corresponding features of the target area in the first frame picture, wherein the first frame picture is the frame picture located at the first position in the video frame sequence, and the target area is the area in the frame picture containing the object to be tracked;

[0013] A second determination module is used to determine a target search area in a target frame picture and a second depth feature map corresponding to the target search area, wherein the target frame picture is any one of the frame pictures except the first frame picture, the target search area includes the target area, and the second depth feature map includes second sub-feature maps corresponding to n feature channels;

[0014] A channel pruning module is used to perform channel pruning on the second depth feature map according to the channel pruning parameter to obtain a corresponding second retained feature map, wherein the second retained feature map includes second sub-feature maps corresponding to m feature channels, m is less than n, and n and m are different positive integers;

[0015] A positioning module is used to determine the target positioning coordinates of the target area in the target frame image based on the convolution result of the second retained feature map and the target depth feature map.

[0016] In a third aspect, the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0017] Acquire a video frame sequence, wherein the video frame sequence includes a plurality of continuous frame images, and the frame images include an object to be tracked;

[0018] Based on the convolution result of the corresponding features of the target area in the first frame picture, determine the channel pruning parameters and the target depth feature map corresponding to the target area, wherein the first frame picture is the frame picture located at the first position in the video frame sequence, and the target area is the area in the frame picture containing the object to be tracked;

[0019] Determine a target search area in a target frame picture and a second depth feature map corresponding to the target search area, wherein the target frame picture is any one of the frame pictures except the first frame picture, the target search area includes the target area, and the second depth feature map includes second sub-feature maps corresponding to n feature channels;

[0020] Performing channel pruning on the second depth feature map according to the channel pruning parameter to obtain a corresponding second retained feature map, wherein the second retained feature map includes second sub-feature maps corresponding to m feature channels, m is less than n, and n and m are different positive integers;

[0021] Based on the convolution result of the second retained feature map and the target depth feature map, the target positioning coordinates of the target area in the target frame image are determined.

[0022] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0023] Acquire a video frame sequence, wherein the video frame sequence includes a plurality of continuous frame images, and the frame images include an object to be tracked;

[0024] Based on the convolution result of the corresponding features of the target area in the first frame picture, determine the channel pruning parameters and the target depth feature map corresponding to the target area, wherein the first frame picture is the frame picture located at the first position in the video frame sequence, and the target area is the area in the frame picture containing the object to be tracked;

[0025] Determine a target search area in a target frame picture and a second depth feature map corresponding to the target search area, wherein the target frame picture is any one of the frame pictures except the first frame picture, the target search area includes the target area, and the second depth feature map includes second sub-feature maps corresponding to n feature channels;

[0026] Performing channel pruning on the second depth feature map according to the channel pruning parameter to obtain a corresponding second retained feature map, wherein the second retained feature map includes second sub-feature maps corresponding to m feature channels, m is less than n, and n and m are different positive integers;

[0027] Based on the convolution result of the second retained feature map and the target depth feature map, the target positioning coordinates of the target area in the target frame image are determined.

[0028] Based on the target tracking method, a video frame sequence for target object tracking and positioning is obtained, the video frame sequence includes a plurality of continuous frame images, the frame images include the object to be tracked, a channel pruning parameter and a target depth feature map corresponding to the target area are determined according to the convolution result of the corresponding features of the first frame image in the video frame sequence, a target search area and a second depth feature map corresponding to the target search area are determined in the target frame image, the target frame image is any frame image except the first frame image, the target search area includes the target area, the second depth feature map includes a second sub-feature map corresponding to n feature channels, the second depth feature map is channel pruned according to the channel pruning parameter to obtain a second retained feature map, the second retained feature map includes a second sub-feature map corresponding to m feature channels, m is less than n, the amount of data involved in subsequent target positioning coordinate calculation is reduced by channel pruning, and the target positioning coordinates of the target area in each frame image are determined according to the convolution result of the second retained feature map after channel pruning and the target depth feature map, thereby accelerating the data processing speed of target object tracking and positioning and improving the real-time tracking of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0031] Figure 1 A schematic diagram of a target tracking method according to an embodiment of the present invention;

[0032] Figure 2 A schematic diagram of a target tracking method according to an embodiment of the present invention;

[0033] Figure 3 is a schematic diagram showing a target frame image in one embodiment;

[0034] Figure 4 A schematic diagram of a target tracking method according to an embodiment of the present invention;

[0035] Figure 5 is a structural block diagram of a target tracking device in one embodiment;

[0036] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0038] In one embodiment, Figure 1 A flow chart of a target tracking method in an embodiment is shown in FIG. Figure 1 , a target tracking method is provided. This embodiment mainly uses the method applied to a target tracking device as an example, and the target tracking method specifically includes the following steps:

[0039] Step S110: acquiring a video frame sequence, wherein the video frame sequence includes a plurality of continuous frame images, and the frame images include an object to be tracked.

[0040] Specifically, the video frame sequence corresponds to a time sequence, and each frame image corresponds to a timestamp. The position of the object to be tracked in different frames may be the same or different. If the position of the object to be tracked in different frames is the same, it means that the object to be tracked has not moved in the time period corresponding to the different frames; if the position of the object to be tracked in different frames is different, it means that the object to be tracked has moved in the time period corresponding to the different frames. Therefore, in order to realize the tracking and positioning of the object to be tracked, it is necessary to determine the position of the object to be tracked in each frame image.

[0041] Each frame image in the video frame sequence is input into the baseline tracker to achieve tracking and positioning of the target object.

[0042] Step S120, based on the convolution result of the corresponding features of the target area in the first frame image, determine the channel pruning parameters and the target depth feature map corresponding to the target area.

[0043] The first frame image is the frame image that is located at the first position in the video frame sequence, and the target area is the area in the frame image that contains the object to be tracked.

[0044] Specifically, the target area is the minimum area containing the object to be tracked, and the size of the target area is recorded as the second preset size. The corresponding target depth feature map can be obtained by convolution processing of the corresponding features of the target area. The size of the target depth feature map is the same as the size of the target area. The target depth feature map includes a first sub-feature map corresponding to m feature channels, and the size of the first sub-feature map is also the same as the size of the target area. The number of feature channels is used to indicate the depth of the convolution layer. The deep high-dimensional features of 512 layers are usually used in target tracking. Different feature channels are used to indicate different convolution layers. Different numbers of feature channels also determine the amount of feature data contained in the corresponding depth feature map. The channel pruning parameter is used to prune the depth features corresponding to some channels in the multiple feature channels corresponding to the depth feature map, so as to reduce the amount of data involved in the tracking calculation.

[0045] Step S130, determining a target search area in a target frame image and a second depth feature map corresponding to the target search area.

[0046] Among them, the target frame image is any one of the frame images except the first frame image, the target search area includes the target area, and the second depth feature map includes a second sub-feature map corresponding to n feature channels.

[0047] Specifically, the amount of depth feature data corresponding to the frame image is large. In order to reduce the amount of data for tracking calculation, the tracking calculation range is narrowed based on the frame image, that is, the target search area is determined in each frame image, and the size of the target search area is smaller than the size of the frame image, but larger than the size of the target area. The size of the target search area is recorded as the first preset size, and features are extracted from the target search area to obtain a corresponding second depth feature map. The second depth feature map includes a second sub-feature map corresponding to n feature channels, and the second depth feature map and the second sub-feature map are both the same size as the target search area.

[0048] Step S140, performing channel pruning on the second depth feature map according to the channel pruning parameters to obtain a corresponding second retained feature map.

[0049] The second retained feature map includes second sub-feature maps corresponding to m feature channels, m is less than n, and n and m are different positive integers.

[0050] Specifically, the amount of data involved in the tracking calculation is further reduced, and the second depth feature map is channel pruned according to the channel pruning parameters, that is, some channels of the n feature channels corresponding to the second depth feature map are removed to obtain the second retained feature map after channel pruning. The second retained feature map corresponds to m feature channels, m is less than n, that is, the difference between m and n is the number of channels removed by pruning, so as to reduce the amount of data of the depth feature, which is used to optimize the tracking calculation speed.

[0051] Step S150, determining the target positioning coordinates of the target area in the target frame image based on the convolution result of the second retained feature map and the target depth feature map.

[0052] Specifically, channel pruning is used to reduce the amount of data involved in the subsequent target positioning coordinate calculation. The target positioning coordinates of the target area in each frame image are determined based on the convolution results of the second retained feature map after channel pruning and the target depth feature map, thereby accelerating the data processing speed of target object tracking and positioning and improving the real-time tracking of the target object.

[0053] In one embodiment, the channel pruning parameters are determined based on the convolution result of the corresponding features of the target area in the first frame image, that is, step S120, including:

[0054] Step S1201, selecting a first frame search area in the first frame image with the target area as the center according to a first preset size, wherein the first preset size is larger than a second preset size of the target area;

[0055] Step S1202, determining the first depth feature map corresponding to the target area and the third depth feature map corresponding to the first frame search area respectively through feature extraction, wherein the first depth feature map includes first sub-feature maps corresponding to n feature channels, and the third depth feature map includes third sub-feature maps corresponding to n feature channels;

[0056] Step S1203, performing convolution processing on the first sub-feature map and the third sub-feature map corresponding to each of the feature channels to obtain a response map matrix corresponding to each of the feature channels, wherein the response map matrix includes a plurality of first response elements, and each of the first response elements corresponds to a different position in the first frame search area;

[0057] Step S1204, determining the background variance corresponding to each of the feature channels according to the difference between the corresponding value of the first response element belonging to the background area in each of the response map matrices and the element mean, wherein the channel pruning parameter includes the background variance corresponding to each of the feature channels, the background area is the area other than the target area in the first frame search area, and the element mean is the average value of the corresponding values ​​of all the first response elements belonging to the background area in the response map matrix.

[0058] Specifically, the scope of the search area is determined by the same size in each frame image, that is, the size of the search area in different frame images is the first preset size, and the first preset size is usually 2 to 3 times the second preset size. In the first frame image, the first frame search area is selected according to the first preset size with the target area as the center, and the first frame search area and the target area are respectively subjected to feature extraction processing using a feature extraction network, so as to obtain a first depth feature map corresponding to the target area and a third depth feature map corresponding to the first frame search area. The depth feature map includes depth features corresponding to multiple feature channels, and the depth features are represented by a graph, namely a sub-feature map. At this time, the number of feature channels of the first depth feature map is the same as the number of feature channels of the third depth feature map, both of which are n. The size of the first depth feature map is the same as the size of the target area, and the size of the third depth feature map is the same as the size of the first frame search area.

[0059] The first sub-feature map and the third sub-feature map corresponding to each feature channel are convolved, and the convolution formula is:

[0060] f c =X c *Z c

[0061] Among them, * is used to indicate the convolution operation, c is the channel index of the feature channel, X c It is used to indicate the third sub-feature map corresponding to the c-th feature channel, Z c Used to indicate the first sub-feature map corresponding to the c-th feature channel, f c Indicates the response map matrix corresponding to the c-th feature channel, where the size of the response map matrix is ​​the same as the size of the first frame search area, which is a first preset size.

[0062] According to the difference between the corresponding value of the first response element belonging to the background area in the corresponding response map matrix of each feature channel and the element mean, the background variance corresponding to each feature channel is determined, that is, the calculation formula of the background variance is as follows:

[0063]

[0064] Among them, Figure 2 As shown in FIG. 1 , SR indicates the search area, OR indicates the target area, and BR is used to indicate the area other than the target area in the search area. BR is recorded as the background area. It is used to indicate the i-th first response element in the corresponding response map matrix of the c-th feature channel belonging to the background area. It is used to indicate all the first response elements belonging to the background area in the corresponding response map matrix of the c-th feature channel. It is used to indicate the average value of the corresponding values ​​of each first response element belonging to the background area in the corresponding response map matrix of the cth feature channel, NBR Used to indicate the number of all first response elements belonging to the background area in the corresponding response map matrix of the c-th feature channel.

[0065] After calculating the background variance of the corresponding response map matrix of all feature channels, the background variance vector V = [V1, V2, V3, ..., V n ] T , represents the background variance corresponding to a total of n feature channels, V1 represents the background variance corresponding to the first feature channel, V n Represents the background variance corresponding to the nth feature channel.

[0066] In one embodiment, based on the convolution result of the corresponding features of the target area in the first frame image, determining the target depth feature map corresponding to the target area, that is, step S120, further includes:

[0067] Step S1205, performing channel pruning on the third depth feature map according to the background variance corresponding to each of the feature channels to obtain a third retained feature map after channel pruning, wherein the third retained feature map includes a third sub-feature map corresponding to m retained channels, the retained channel is any one of the feature channels, and the size of the third sub-feature map is the first preset size;

[0068] Step S1206, determining a plurality of sample regions and regression kernel parameters corresponding to the sample regions in the third sub-feature map corresponding to each of the retained channels, wherein the size of the sample region is the second preset size;

[0069] Step S1207, determining a channel weight corresponding to each of the reserved channels according to the regression kernel parameters corresponding to the plurality of sample regions corresponding to each of the reserved channels;

[0070] Step S1208: Determine a target depth feature map corresponding to the target area according to the first sub-feature maps corresponding to each of the reserved channels and the channel weights corresponding to each of the reserved channels.

[0071] Specifically, according to the background variance corresponding to each of the feature channels, the third depth feature map is channel pruned using a channel pruning function to obtain a third retained feature map after channel pruning. The third retained feature map is recorded as is a channel pruning function, X is used to indicate the third depth feature map, V is used to indicate the background variance vector, the third retained feature map includes third sub-feature maps corresponding to m retained channels, the retained channels are feature channels retained after channel pruning, and the sizes of the third retained feature map after channel pruning and each third sub-feature map are the first preset size, that is, the same as the size of the search area of ​​the first frame.

[0072] In the third sub-feature map corresponding to each of the retained channels, a plurality of sample areas are determined, the size of the sample area is a second preset size, that is, the size of the sample area is the same as the size of the target area, the sample center coordinates of each sample area are marked as (a, b), and a sliding window is formed in each third sub-feature map with the second preset size. The sliding window moves in the third sub-feature map according to a preset step size each time, and the area within the sliding window is taken as a sample area, and the sample center coordinates of the sample area in the third sub-feature map are determined, so that a plurality of sample areas can be determined in the third sub-feature map by moving the sliding window, and a corresponding Gaussian label is determined according to each sample area. The Gaussian label is shown in the following formula:

[0073]

[0074] Among them, σ is the kernel width, and then the ridge regression loss function is calculated according to the gradient descent method based on the Gaussian label. The ridge regression loss function is:

[0075] L reg =||Y(a, b)-W*X′ a，b || 2 +λ‖W‖ 2

[0076] Among them, W is the regression kernel parameter, X′ a，b The third reserved feature map is used to indicate the sample area, λ is the regression coefficient, and L is calculated according to the gradient descent method. reg The regression kernel parameters corresponding to the minimum value are taken as the regression kernel parameters corresponding to the sample area. In this way, the regression kernel parameters corresponding to each sample area in the third sub-feature map can be determined, that is, each third sub-feature map corresponds to multiple regression kernel parameters. Since each third sub-feature map corresponds to a reserved channel, each reserved channel corresponds to multiple regression kernel parameters.

[0077] According to the regression kernel parameters corresponding to the plurality of sample regions corresponding to the respective reserved channels, the channel weights corresponding to the respective reserved channels are determined, that is, the channel weights corresponding to the respective reserved channels are calculated according to the regression kernel parameters corresponding to the respective sample regions in the third sub-feature map. The channel weight calculation formula is as follows:

[0078]

[0079] in, It is used to indicate the regression kernel parameters corresponding to the ith sample area of ​​the cth retained channel, W c Used to indicate the regression kernel parameter set corresponding to the cth retained channel, θ cIt is used to indicate the channel weight corresponding to the cth reserved channel, Tanh is used to indicate the hyperbolic tangent function, and a channel weight vector is formed according to the channel weights corresponding to all reserved channels, which is recorded as θ = [θ1, θ2, θ3, …θ m ] T , indicating that there are a total of m channel weights corresponding to the reserved channels.

[0080] According to the learned background variance vector and channel weight vector, the first sub-feature map corresponding to the target area is subjected to feature pruning based on background discreteness and adaptive weight enhancement based on target perception, and the target depth feature map corresponding to the improved target area is obtained, which is recorded as:

[0081]

[0082] Wherein, Z is used to indicate the first depth feature map before channel pruning, that is, it includes multiple first sub-feature maps, V is used to indicate the background variance vector, is the channel pruning function, Z′ is the target deep feature map after channel pruning and weight enhancement of each retained channel. By weight enhancement of each retained channel, it is used to learn deep semantic perception features and improve the representation ability of deep features for target objects.

[0083] When determining the location coordinates of the target area in each subsequent frame, the target depth feature map is used as a template.

[0084] In one embodiment, the determining of the target search area in the target frame picture and the second depth feature map corresponding to the target search area, that is, step S130 includes:

[0085] Step S1301, obtaining the coordinates of the center point of the target area in the previous frame picture adjacent to the target frame picture, and obtaining the first positioning coordinates;

[0086] Step S1302, selecting a target search area in the target frame image according to the first preset size and taking the first positioning coordinate as the center;

[0087] Step S1303: extract features from the target search area to obtain a second depth feature map corresponding to the target search area.

[0088] Specifically, the target frame picture is the positioning coordinates of the target area determined in the frame pictures other than the first frame picture in the video frame sequence. It is necessary to first obtain the coordinates of the center point of the target area in the previous frame picture adjacent to the current target frame picture, that is, to obtain the first positioning coordinates, and use the first positioning coordinates as the center to select the target search area in the target frame picture according to the first preset size, that is, the target search area and the first frame search area are search areas of the same size in different frame pictures. The feature extraction network is used to extract features of the target search area in the target frame picture to obtain a second depth feature map corresponding to the target search area.

[0089] Similarly, when determining the search area for each frame image, the positioning coordinates of the target area in the adjacent previous frame image are used as the center, and then the search area is selected according to the size of the search area.

[0090] In one embodiment, the second depth feature map is channel pruned according to the channel pruning parameter to obtain a corresponding second retained feature map, that is, step S140 includes:

[0091] Step S1401, taking a feature channel whose background variance is less than or equal to a preset variance among the multiple feature channels as a reserved channel;

[0092] Step S1402: Generate the second retained feature map according to the second sub-feature maps corresponding to each of the retained channels.

[0093] Specifically, the sensitivity of the corresponding feature channel to the change of the target object can be judged according to the numerical value of the background variance corresponding to each feature channel. That is, the larger the background variance value, the more sensitive it is to the change of the target object. The preset variance can be customized according to the actual application scenario, and the feature channel with a background variance less than or equal to the preset variance is retained for the appearance representation of the target object, that is, the feature channel is used as a reserved channel, and the feature channel with a background variance greater than the preset variance is removed from the original pre-trained depth feature. Here, the second sub-feature map corresponding to the feature channel with a background variance greater than the preset variance is removed from the second depth feature map, that is, the depth features in the second sub-feature map corresponding to the feature channel with a background variance greater than the preset variance are removed from the pre-trained depth features corresponding to the second depth feature map. A second retained feature map is formed according to the second sub-feature maps corresponding to each retained channel, and the second retained feature map is recorded as Where X t is the second depth feature map.

[0094] Other scenes that require channel pruning also follow the same principle. Feature channels with background variance greater than the preset variance are removed, and only the depth features corresponding to each retained channel are retained.

[0095] In one embodiment, the target positioning coordinates of the target area in the target frame picture are determined based on the convolution result of the second retained feature map and the target depth feature map, that is, step S150 includes:

[0096] Step S1501, performing convolution processing on the second retained feature map and the target feature map to obtain a target response map, wherein the target response map includes a plurality of second response elements, and an element value of each second response element indicates a confidence that the second response element belongs to the target area;

[0097] Step S1502, taking the second response element corresponding to the element value with the largest numerical value as the target element, and determining the second positioning coordinates of the target element in the target search area;

[0098] Step S1503: determining the target positioning coordinates of the target area in the target frame image according to the first positioning coordinates, the second positioning coordinates and the first preset size.

[0099] Specifically, the second retained feature map and the target feature map are convolved to obtain a target response map. The calculation formula of the target response map is as follows:

[0100]

[0101] Among them, F t It is used to indicate the target response map with a feature channel number of 1, Z′ is used to indicate the corresponding depth feature of the target depth feature map, and X t′ Used to indicate the depth features corresponding to the second retained feature map. Since the target depth feature map includes the first sub-feature maps corresponding to the m retained channels, and the second retained depth feature map includes the second sub-feature maps corresponding to the m retained channels, the target depth feature map is convolved with the second retained depth feature map, that is, the first sub-feature map corresponding to each retained channel is convolved with the second sub-feature map of the corresponding retained channel to obtain the response map matrix corresponding to each retained channel, and then the response map matrices corresponding to each retained channel are added and integrated into a target response map F corresponding to a feature channel. t , f i t Used to indicate the response map matrix after the convolution of the first sub-feature map and the second sub-feature map corresponding to the i-th retained channel.

[0102] The size of the target response map is the same as the size of the search area. The target response map includes multiple second response elements. The element value of each second response element indicates the confidence that the second response element belongs to the target area. The larger the element value, the greater the probability that the corresponding second response element belongs to the target area. The second response element corresponding to the element value with the largest numerical value is taken as the target element, and the second positioning coordinates of the target element in the target search area are determined. However, the second positioning coordinates are the coordinates of the target area in the target search area, not the coordinates of the target area in the target frame image. Therefore, it is necessary to combine the first positioning coordinates, the second positioning coordinates and the first preset size of the search area to calculate the target positioning coordinates of the target area in the target frame image. The calculation formula is as follows:

[0103]

[0104] in, and Used to indicate the target positioning coordinates, x and y are used to indicate the second positioning coordinates, x center and center is used to indicate the first positioning coordinate, d is used to indicate the size coefficient, in this embodiment, d=0.5, w is used to indicate the width of the search area, and h is used to indicate the height of the search area, that is, the first preset size includes the width and the height, such as Figure 3 As shown, Figure 3 The current picture area in is used to indicate the picture area of ​​the target frame picture, the current search area indicates the target search area in the target frame picture, point A indicates the first positioning coordinate, and point B indicates the second positioning coordinate.

[0105] like Figure 4 As shown, each frame image in the video frame sequence is input into the baseline tracker, and the deep features in the search area and the target area are first extracted through the feature extraction network, and then channel pruning based on background discreteness and semantically aware channel weighting are performed, so as to achieve the positioning coordinates of the target area tracked in each frame image, thereby reducing the amount of data of the deep features involved in the tracking calculation and improving the representation ability of the features, thereby achieving the effect of optimizing the tracking speed and tracking accuracy of the baseline tracker.

[0106] In one embodiment, after determining the target positioning coordinates of the target area in the target frame picture according to the first positioning coordinates, the second positioning coordinates and the first preset size, that is, after step S150, the method further includes:

[0107] Step S160, determining the ranking of the target frame image in the video frame sequence;

[0108] Step S170, when the ranking rank fails to match the last rank in the video frame sequence, the next frame picture adjacent to the target frame picture is used as the target frame picture, and the step of determining the target search area in the target frame picture and the second depth feature map corresponding to the target search area is executed, until the step of determining the target positioning coordinates of the target area in the target frame picture is determined.

[0109] Specifically, since each frame image is arranged consecutively in chronological order in the video frame sequence, the ranking of the target frame image in the video frame sequence is determined, so as to judge whether the target frame image is the last frame image, that is, the last frame image is the frame image at the end of the video frame sequence. When the ranking rank of the target frame image fails to match the last rank, it indicates that the target frame image is not the last frame image, then the next frame image adjacent to the target frame image is used as the target frame image to continue to execute the above-mentioned steps S130 to S160, until the ranking rank of the target frame image successfully matches the last rank, indicating that the target frame image is the last frame image, then the tracking and positioning of the target object is terminated.

[0110] Figure 1 FIG. 1 is a flow chart of a target tracking method in one embodiment. It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0111] In one embodiment, Figure 5 As shown, a target tracking device is provided, comprising:

[0112] An acquisition module 210, configured to acquire a video frame sequence, wherein the video frame sequence includes a plurality of continuous frame images, and the frame images include an object to be tracked;

[0113] A first determination module 220 is used to determine a channel pruning parameter and a target depth feature map corresponding to the target area based on a convolution result of a corresponding feature of the target area in a first frame image, wherein the first frame image is the frame image located at the first position in the video frame sequence, and the target area is an area in the frame image containing the object to be tracked;

[0114] A second determination module 230 is used to determine a target search area in a target frame picture and a second depth feature map corresponding to the target search area, wherein the target frame picture is any one of the frame pictures except the first frame picture, the target search area includes the target area, and the second depth feature map includes second sub-feature maps corresponding to n feature channels;

[0115] A channel pruning module 240 is used to perform channel pruning on the second depth feature map according to the channel pruning parameter to obtain a corresponding second retained feature map, wherein the second retained feature map includes second sub-feature maps corresponding to m feature channels, m is less than n, and n and m are different positive integers;

[0116] The positioning module 250 is used to determine the target positioning coordinates of the target area in the target frame image based on the convolution result of the second retained feature map and the target depth feature map.

[0117] In one embodiment, the first determining module 220 is specifically configured to:

[0118] A first frame search area is framed in the first frame image with the target area as the center according to a first preset size, wherein the first preset size is larger than a second preset size of the target area;

[0119] Determine the first depth feature map corresponding to the target area and the third depth feature map corresponding to the first frame search area respectively through feature extraction, wherein the first depth feature map includes a first sub-feature map corresponding to n feature channels, and the third depth feature map includes a third sub-feature map corresponding to n feature channels;

[0120] Performing convolution processing on the first sub-feature map and the third sub-feature map corresponding to each of the feature channels to obtain a response map matrix corresponding to each of the feature channels, wherein the response map matrix includes a plurality of first response elements, and each of the first response elements corresponds to a different position in the first frame search area;

[0121] The background variance corresponding to each of the feature channels is determined according to the difference between the corresponding value of the first response element belonging to the background area in each of the response map matrices and the element mean, wherein the channel pruning parameter includes the background variance corresponding to each of the feature channels, the background area is the area other than the target area in the first frame search area, and the element mean is the average value of the corresponding values ​​of all the first response elements belonging to the background area in the response map matrix.

[0122] In one embodiment, the first determining module 220 is specifically configured to:

[0123] Performing channel pruning on the third depth feature map according to the background variance corresponding to each of the feature channels to obtain a third retained feature map after channel pruning, wherein the third retained feature map includes a third sub-feature map corresponding to m retained channels, the retained channel is any one of the feature channels, and the size of the third sub-feature map is the first preset size;

[0124] Determining a plurality of sample regions and regression kernel parameters corresponding to the sample regions in the third sub-feature map corresponding to each of the retained channels, wherein the size of the sample region is the second preset size;

[0125] Determine the channel weight corresponding to each of the reserved channels according to the regression kernel parameters corresponding to the plurality of sample regions corresponding to each of the reserved channels;

[0126] A target depth feature map corresponding to the target area is determined according to the first sub-feature maps corresponding to each of the retained channels and the channel weights corresponding to each of the retained channels.

[0127] In one embodiment, the second determining module 230 is specifically configured to:

[0128] Obtaining the coordinates of the center point of the target area in the previous frame image adjacent to the target frame image to obtain the first positioning coordinates;

[0129] A target search area is framed in the target frame image with the first positioning coordinate as the center according to the first preset size;

[0130] Perform feature extraction on the target search area to obtain a second depth feature map corresponding to the target search area.

[0131] In one embodiment, the channel pruning module 240 is specifically used to:

[0132] The feature channels whose background variance is less than or equal to the preset variance among the multiple feature channels are taken as reserved channels;

[0133] The second retained feature map is generated according to the second sub-feature maps corresponding to each of the retained channels.

[0134] In one embodiment, the positioning module 250 is specifically used for:

[0135] Performing convolution processing on the second retained feature map and the target feature map to obtain a target response map, wherein the target response map includes a plurality of second response elements, and an element value of each second response element indicates a confidence that the second response element belongs to the target area;

[0136] Taking the second response element corresponding to the element value with the largest numerical value as the target element, and determining the second positioning coordinates of the target element in the target search area;

[0137] The target positioning coordinates of the target area in the target frame picture are determined according to the first positioning coordinates, the second positioning coordinates and the first preset size.

[0138] In one embodiment, the second determining module 230 is further configured to:

[0139] Determine the ranking of the target frame image in the video frame sequence;

[0140] When the ranking rank fails to match the last rank in the video frame sequence, the next frame picture adjacent to the target frame picture is used as the target frame picture, and the steps of determining the target search area in the target frame picture and the second depth feature map corresponding to the target search area are executed, until the step of determining the target positioning coordinates of the target area in the target frame picture is determined.

[0141] Figure 6 FIG. 1 shows an internal structure diagram of a computer device in an embodiment. The computer device may specifically be a target tracking device. Figure 6 As shown, the computer device includes a processor, a memory, a network interface, an input device and a display screen connected through a system bus. Among them, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program, which, when executed by the processor, enables the processor to implement a target tracking method. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to execute a target tracking method. The display screen of the computer device may be a liquid crystal display or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0142] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] In one embodiment, the target tracking device provided by the present application can be implemented in the form of a computer program. The computer program can be Figure 6The computer device shown in the figure is run. The memory of the computer device can store various program modules constituting the target tracking device, such as: Figure 5 The acquisition module, the first determination module, the second determination module, the channel pruning module and the positioning module shown in the figure. The computer program composed of each program module enables the processor to execute the steps of the target tracking method in each embodiment of the present application described in this specification.

[0144] Figure 6 The computer device shown can be Figure 5 The acquisition module in the target tracking device shown acquires a video frame sequence, wherein the video frame sequence includes a plurality of continuous frame images, and the frame images include the object to be tracked. The computer device can determine the channel pruning parameters and the target depth feature map corresponding to the target area through the first determination module based on the convolution result of the corresponding features of the target area in the first frame image, wherein the first frame image is the frame image located at the first position in the video frame sequence, and the target area is the area in the frame image containing the object to be tracked. The computer device can determine the target search area in the target frame image and the second depth feature map corresponding to the target search area through the second determination module, wherein the target frame image is any one of the frame images other than the first frame image, the target search area contains the target area, and the second depth feature map includes the second sub-feature map corresponding to n feature channels. The computer device can perform channel pruning on the second depth feature map according to the channel pruning parameters through the channel pruning module to obtain the corresponding second retained feature map, wherein the second retained feature map includes the second sub-feature map corresponding to m feature channels, m is less than n, and n and m are different positive integers. The computer device can determine the target positioning coordinates of the target area in the target frame image by executing a convolution result based on the second retained feature map and the target depth feature map through a positioning module.

[0145] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the method described in any one of the above embodiments is implemented when the processor executes the computer program.

[0146] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in any one of the above embodiments is implemented.

[0147] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0148] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements limited by the characterizing sentence "includes one...", and does not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0149] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A target tracking method, characterized in that: The method comprises: Acquire a video frame sequence, wherein the video frame sequence includes a plurality of continuous frame images, and the frame images include an object to be tracked; Based on the convolution result of the corresponding features of the target area in the first frame picture, determine the channel pruning parameters and the target depth feature map corresponding to the target area, wherein the first frame picture is the frame picture located at the first position in the video frame sequence, and the target area is the area in the frame picture containing the object to be tracked; Determine a target search area in a target frame picture and a second depth feature map corresponding to the target search area, wherein the target frame picture is any frame picture in the video frame sequence except the first frame picture, the target search area includes the target area, and the second depth feature map includes second sub-feature maps corresponding to n feature channels; Performing channel pruning on the second depth feature map according to the channel pruning parameter to obtain a corresponding second retained feature map, wherein the second retained feature map includes second sub-feature maps corresponding to m feature channels, m is less than n, and n and m are different positive integers; Determining the target positioning coordinates of the target area in the target frame image based on the convolution result of the second retained feature map and the target depth feature map; The determining of the channel pruning parameters based on the convolution result of the corresponding features of the target area in the first frame image includes: A first frame search area is framed in the first frame image with the target area as the center according to a first preset size, wherein the first preset size is larger than a second preset size of the target area; Determine, by feature extraction, a first depth feature map corresponding to the target area and a third depth feature map corresponding to the first frame search area, respectively, wherein the first depth feature map includes a first sub-feature map corresponding to n feature channels, and the third depth feature map includes a third sub-feature map corresponding to n feature channels; Performing convolution processing on the first sub-feature map and the third sub-feature map corresponding to each of the feature channels to obtain a response map matrix corresponding to each of the feature channels, wherein the response map matrix includes a plurality of first response elements, and each of the first response elements corresponds to a different position in the first frame search area; The background variance corresponding to each of the feature channels is determined according to the difference between the corresponding value of the first response element belonging to the background area in each of the response map matrices and the element mean, wherein the channel pruning parameter includes the background variance corresponding to each of the feature channels, the background area is the area other than the target area in the first frame search area, and the element mean is the average value of the corresponding values ​​of all the first response elements belonging to the background area in the response map matrix.

2. The method according to claim 1, characterized in that Based on the convolution result of the corresponding features of the target area in the first frame image, the target depth feature map corresponding to the target area is determined, including: Performing channel pruning on the third depth feature map according to the background variance corresponding to each of the feature channels to obtain a third retained feature map after channel pruning, wherein the third retained feature map includes a third sub-feature map corresponding to m retained channels, the retained channel is any one of the feature channels, and the size of the third sub-feature map is the first preset size; Determining a plurality of sample regions and regression kernel parameters corresponding to the sample regions in the third sub-feature map corresponding to each of the retained channels, wherein the size of the sample region is the second preset size; Determine the channel weight corresponding to each of the reserved channels according to the regression kernel parameters corresponding to the plurality of sample regions corresponding to each of the reserved channels; A target depth feature map corresponding to the target area is determined according to the first sub-feature maps corresponding to each of the retained channels and the channel weights corresponding to each of the retained channels.

3. The method according to claim 2, characterized in that The determining of a target search area in a target frame picture and a second depth feature map corresponding to the target search area includes: Obtaining the coordinates of the center point of the target area in the previous frame image adjacent to the target frame image to obtain the first positioning coordinates; A target search area is framed in the target frame image with the first positioning coordinate as the center according to the first preset size; Perform feature extraction on the target search area to obtain a second depth feature map corresponding to the target search area.

4. The method according to claim 3, characterized in that The performing channel pruning on the second depth feature map according to the channel pruning parameter to obtain a corresponding second retained feature map includes: The feature channels whose background variance is less than or equal to the preset variance among the multiple feature channels are taken as reserved channels; The second retained feature map is generated according to the second sub-feature maps corresponding to each of the retained channels.

5. The method according to claim 4, characterized in that The determining the target positioning coordinates of the target area in the target frame picture based on the convolution result of the second retained feature map and the target depth feature map includes: Performing convolution processing on the second retained feature map and the target depth feature map to obtain a target response map, wherein the target response map includes a plurality of second response elements, and an element value of each second response element indicates a confidence that the second response element belongs to the target area; Taking the second response element corresponding to the element value with the largest numerical value as the target element, and determining the second positioning coordinates of the target element in the target search area; The target positioning coordinates of the target area in the target frame picture are determined according to the first positioning coordinates, the second positioning coordinates and the first preset size.

6. The method according to claim 5, characterized in that After determining the target positioning coordinates of the target area in the target frame picture according to the first positioning coordinates, the second positioning coordinates and the first preset size, the method further includes: Determine the ranking of the target frame image in the video frame sequence; When the ranking rank fails to match the last rank in the video frame sequence, the next frame picture adjacent to the target frame picture is used as the target frame picture, and the steps of determining the target search area in the target frame picture and the second depth feature map corresponding to the target search area are executed, until the step of determining the target positioning coordinates of the target area in the target frame picture is determined.

7. A target tracking device, characterized in that: The device comprises: An acquisition module, used for acquiring a video frame sequence, wherein the video frame sequence includes a plurality of continuous frame images, and the frame images include an object to be tracked; A first determination module is used to determine the channel pruning parameters and the target depth feature map corresponding to the target area based on the convolution result of the corresponding features of the target area in the first frame picture, wherein the first frame picture is the frame picture located at the first position in the video frame sequence, and the target area is the area in the frame picture containing the object to be tracked; A second determination module is used to determine a target search area in a target frame picture and a second depth feature map corresponding to the target search area, wherein the target frame picture is any one of the frame pictures except the first frame picture, the target search area includes the target area, and the second depth feature map includes second sub-feature maps corresponding to n feature channels; A channel pruning module is used to perform channel pruning on the second depth feature map according to the channel pruning parameter to obtain a corresponding second retained feature map, wherein the second retained feature map includes second sub-feature maps corresponding to m feature channels, m is less than n, and n and m are different positive integers; A positioning module, used to determine the target positioning coordinates of the target area in the target frame image based on the convolution result of the second retained feature map and the target depth feature map; The first determining module is further used for: A first frame search area is framed in the first frame image with the target area as the center according to a first preset size, wherein the first preset size is larger than a second preset size of the target area; Determine, by feature extraction, a first depth feature map corresponding to the target area and a third depth feature map corresponding to the first frame search area, respectively, wherein the first depth feature map includes a first sub-feature map corresponding to n feature channels, and the third depth feature map includes a third sub-feature map corresponding to n feature channels; Performing convolution processing on the first sub-feature map and the third sub-feature map corresponding to each of the feature channels to obtain a response map matrix corresponding to each of the feature channels, wherein the response map matrix includes a plurality of first response elements, and each of the first response elements corresponds to a different position in the first frame search area; The background variance corresponding to each of the feature channels is determined according to the difference between the corresponding value of the first response element belonging to the background area in each of the response map matrices and the element mean, wherein the channel pruning parameter includes the background variance corresponding to each of the feature channels, the background area is the area other than the target area in the first frame search area, and the element mean is the average value of the corresponding values ​​of all the first response elements belonging to the background area in the response map matrix.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images