A visual tracking model construction method and device

Abstract

The application discloses a visual tracking model construction method and device. When pruning a network, the method calculates L2 norms of each convolution channel of each selected convolution layer in a selected convolution layer set in a visual tracking network, and takes a predetermined proportion of convolution channels with the lowest L2 norms as a first pruning channel set. The method calculates L2 norms of each convolution channel of each unselected convolution layer in a residual module with the same down-sampling in the visual tracking network, and takes a predetermined proportion of convolution channels with the lowest L2 norms as a second pruning channel set of each residual module with the same down-sampling. The method calculates the intersection of the second pruning channel sets between the residual modules to obtain a third pruning channel set. The visual tracking network is pruned according to the first pruning channel set and the third pruning channel set. The application can reduce the performance decline of the visual tracking network caused by excessive sparseness, and balance the performance and efficiency of the visual tracking network.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision target tracking and artificial intelligence, and particularly relates to a visual tracking model construction method and device. BACKGROUND

[0002] Convolutional Neural Networks (CNNs), especially Deep Neural Networks (DNNs), play an increasingly important role in computer vision tasks. With the gradual advancement of deep learning technology, neural networks have become wider, deeper and more complex, which, while bringing better representation ability and performance, has also continuously reduced the execution efficiency of visual models, especially in low-power application scenarios such as mobile terminals and edge computing terminals. A large amount of image data makes it difficult for existing visual models to be applied to low-power platforms (such as small unmanned aerial vehicles) with high real-time requirements.

[0003] Pruning the visual model is an important means to balance the performance and efficiency of the model. However, existing visual model pruning schemes often act on all convolution kernels during sparse training, which causes convolution kernels not in the pruning target range to participate in sparse training, thereby affecting the performance of the visual network. In addition, since many visual models often use network structures similar to resnet, which use short_cut (direct connection) for connection, when pruning, a linkage effect occurs, that is, when pruning the network layers of the current layer, because of the existence of short_cut, the layers that need to be added also need to subtract the corresponding layers to be added, thereby causing some layers that do not need to be pruned to be subtracted, which also affects the overall performance of the visual network. SUMMARY

[0004] The application discloses a visual tracking model construction method and device to alleviate the problem of performance degradation of the visual tracking network caused by excessive sparseness and achieve a balance between the performance and efficiency of the visual tracking network.

[0005] According to a first aspect of the application, a visual tracking model construction method is disclosed, comprising:

[0006] constructing a visual tracking network and training the visual tracking network using an image sequence dataset; wherein the visual tracking network is composed of a plurality of residual modules;

[0007] performing sparse training on the trained visual tracking network;

[0008] pruning the visual tracking network after sparse training to obtain a visual tracking model; comprising:

[0009] compute L2 norm of each convolution channel of each selected convolution layer in the selected convolution layer set in the visual tracking network, and take a predetermined proportion of convolution channels with the lowest L2 norm as a first pruning channel set;

[0010] compute L2 norm of each convolution channel of each unselected convolution layer in the residual module with the same down-sampling in the visual tracking network, which does not belong to the selected convolution layer set, and take a predetermined proportion of convolution channels with the lowest L2 norm as a second pruning channel set of each residual module with the same down-sampling;

[0011] compute the intersection of the second pruning channel sets between all residual modules with the same down-sampling to obtain a third pruning channel set;

[0012] prune the visual tracking network according to the first pruning channel set and the third pruning channel set to obtain a visual tracking model.

[0013] In other examples, the sparse training of the trained visual tracking network includes: when each training round ends, computing L2 norm of each convolution channel of each selected convolution layer in the selected convolution layer set in the visual tracking network, and setting a predetermined proportion of convolution channels with the lowest L2 norm to 0 to complete the sparse training of the current round.

[0014] In other examples, at the beginning of the next training round, the convolution channels set to 0 are allowed to recover to a non-0 state.

[0015] According to the second aspect of the present application, a visual tracking model construction device is disclosed, comprising:

[0016] a network training module configured to train a visual tracking network using an image sequence data set; wherein the visual tracking network is composed of a plurality of residual modules;

[0017] a sparse training module configured to perform sparse training on the trained visual tracking network;

[0018] a model generation module configured to prune the visual tracking network after sparse training to obtain a visual tracking model, comprising:

[0019] a first calculation module for computing L2 norm of each convolution channel of each selected convolution layer in the selected convolution layer set in the visual tracking network, and taking a predetermined proportion of convolution channels with the lowest L2 norm as a first pruning channel set;

[0020] a second calculation module configured to calculate an L2 norm of each convolution channel of each unselected convolution layer in the residual modules with the same down-sampling and not belonging to the selected convolution layer set, and take a predetermined proportion of convolution channels with the lowest L2 norm as a second pruning channel set of each residual module with the same down-sampling;

[0021] an intersection calculation module configured to calculate an intersection of the second pruning channel sets among all the residual modules with the same down-sampling to obtain a third pruning channel set;

[0022] a pruning module configured to prune the visual tracking network according to the first pruning channel set and the third pruning channel set to obtain a visual tracking model.

[0023] According to a third aspect of the present disclosure, another method for constructing a visual tracking model is disclosed, comprising:

[0024] constructing a visual tracking network and training the visual tracking network using an image sequence dataset;

[0025] performing sparse training on the trained visual tracking network;

[0026] pruning the sparse trained visual tracking network, calculating an L2 norm of each convolution channel of each selected convolution layer in the selected convolution layer set in the visual tracking network, and pruning a predetermined proportion of convolution channels with the lowest L2 norm to obtain a visual tracking model.

[0027] In some other examples, the sparse training on the trained visual tracking network comprises: at the end of each training round, calculating an L2 norm of each convolution channel of each selected convolution layer in the selected convolution layer set in the visual tracking network, and setting a predetermined proportion of convolution channels with the lowest L2 norm to zero to complete the sparse training of the current round.

[0028] In some other examples, at the beginning of the next training round, the set-to-zero convolution channels are allowed to recover to a non-zero state.

[0029] According to a fourth aspect of the present disclosure, another apparatus for constructing a visual tracking model is disclosed, comprising:

[0030] a network training module configured to train a visual tracking network using an image sequence dataset;

[0031] a sparse training module configured to perform sparse training on the trained visual tracking network; wherein when each training round ends, the sparse training module calculates the L2 norm of each convolutional channel of each selected convolutional layer in the selected convolutional layer set in the visual tracking network, and sets to 0 a predetermined proportion of convolutional channels with the lowest L2 norm, to complete the sparse training of the current round;

[0032] a model generation module configured to perform pruning on the visual tracking network after sparse training, calculate the L2 norm of each convolutional channel of each selected convolutional layer in the selected convolutional layer set in the visual tracking network, and prune a predetermined proportion of convolutional channels with the lowest L2 norm, to obtain a visual tracking model.

[0033] The visual tracking network is a resnet network.

[0034] Compared with the prior art, the application only performs sparse training on selected convolutional kernels, which can reduce the performance decline of the visual tracking network caused by excessive sparseness. In addition, by performing intersection pruning, the unsparsed convolutional kernels can be pruned while ensuring a sufficient number of pruned kernels, so as to balance the performance and efficiency of the visual tracking network. BRIEF DESCRIPTION OF DRAWINGS

[0035] Other features, objects and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments, made with reference to the accompanying drawings:

[0036] Figure 1 A visual tracking model construction method flowchart according to embodiment 1 of the application;

[0037] Figure 2 A residual module structure example;

[0038] Figure 3 A visual tracking model construction device composition schematic diagram for implementing the method of embodiment 1;

[0039] Figure 4 A visual tracking model construction method flowchart according to embodiment 2 of the application;

[0040] Figure 5 A visual tracking model construction device composition schematic diagram for implementing the method of embodiment 2. DETAILED DESCRIPTION

[0041] The application will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that, for the convenience of description, only the parts related to the application are shown in the drawings.

[0042] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0043] Figure 1 A flowchart of a visual tracking model construction method according to Embodiment 1 of the present application is shown in the figure. As shown in the figure, the method comprises the following steps:

[0044] Step 101, constructing a visual tracking network and training the visual tracking network using an image sequence dataset;

[0045] Exemplarily, the visual tracking network is a resnet network, which is composed of multiple residual modules.

[0046] Training samples are constructed using an image sequence, and the samples in the image sequence should contain multiple object categories, scales, poses, illuminations and occlusion conditions. The samples are labeled to indicate the position and scale of the target object in the samples, and the resnet network is trained using the training samples.

[0047] Step 102, performing sparse training on the trained resnet network;

[0048] When performing sparse training on the trained resnet network, at the end of each training epoch, the L2 norm of each convolutional channel of each selected convolutional layer in the selected convolutional layer set in the resnet network is calculated, and the predetermined proportion of convolutional channels with the lowest L2 norm are set to 0 to complete the sparse training of the current epoch. At the same time, at the beginning of the next training epoch, the convolutional channels that have been set to 0 are allowed to recover to a non-0 state.

[0049] Repeat the training for N epochs until the loss function converges, and complete the sparse training of the resnet network.

[0050] In the present application, the selected convolutional layer set includes convolutional layers in the resnet network that are located outside the residual module (res_block) and whose outputs are not connected to the inputs of the residual module, and convolutional layers located inside the residual module except the last layer. For example, Figure 2 As shown in one residual module, the residual module includes a 1*1 layer on the input side, a 3*3 layer and a 1*1 layer on the output side, and a short_cut (direct connection) structure. The 1*1 layer on the input side and the 3*3 layer are selected convolutional layers, and the 1*1 layer on the output side does not belong to the selected convolutional layers.

[0051] Step 103, pruning the resnet network after sparse training to obtain a visual tracking model;

[0052] When pruning the resnet network after sparse training, the L2 norm of each convolutional channel of each selected convolutional layer in the selected convolutional layer set in the resnet network is calculated, and a predetermined proportion of convolutional channels with the lowest L2 norm are pruned to obtain a visual tracking model.

[0053] This embodiment can alleviate the problem of network performance decline caused by excessive sparsity by only sparsely training selected convolutional kernels.

[0054] It can be understood that the visual tracking model can also be retrained to restore the sparse network parameters to a normal distribution state to obtain a final visual tracking model.

[0055] Figure 3 A schematic diagram of a visual tracking model construction device for implementing the method of embodiment 1 is shown. As shown in the figure, the device includes:

[0056] The network training module 301 is configured to train the visual tracking network using an image sequence dataset.

[0057] For example, the visual tracking network is a resnet network composed of multiple residual modules.

[0058] The module 301 uses an image sequence to construct training samples, and the samples in the image sequence should contain multiple object categories, scales, poses, illuminations and occlusion conditions. The samples are labeled to indicate the position and scale of the target object in the sample, and the resnet network is trained using the training samples.

[0059] The sparse training module 302 is configured to sparsely train the trained visual tracking network.

[0060] When the resnet network is sparsely trained, the L2 norm of each convolutional channel of each selected convolutional layer in the selected convolutional layer set in the resnet network is calculated at the end of each training epoch, and a predetermined proportion of convolutional channels with the lowest L2 norm are set to 0 to complete the sparse training of the current epoch. At the same time, when the next training epoch starts, the convolutional channels that are set to 0 are allowed to recover to a non-zero state.

[0061] Repeat N training epochs until the loss function converges to complete the sparse training of the resnet network.

[0062] The model generation module 303 is configured to prune the visual tracking network subjected to sparse training to obtain a visual tracking model.

[0063] When pruning the resnet network subjected to sparse training, the module 303 calculates the L2 norm of each convolution channel of each selected convolution layer in the selected convolution layer set in the resnet network, and prunes a predetermined proportion of convolution channels with the lowest L2 norm to obtain a visual tracking model.

[0064] Figure 4 A flowchart of a visual tracking model construction method according to Embodiment 2 of the present application is shown in the figure. As shown in the figure, the method comprises the following steps:

[0065] Step 401: Construct a visual tracking network and train the visual tracking network using an image sequence dataset.

[0066] Exemplarily, the visual tracking network is a resnet network composed of multiple residual modules.

[0067] Training samples are constructed using an image sequence. The samples in the image sequence should contain multiple object categories, scales, poses, illuminations and occlusion conditions. The samples are labeled to indicate the location and scale of the target object in the samples, and the resnet network is trained using the training samples.

[0068] Step 402: Sparse train the trained resnet network.

[0069] When sparse training the trained resnet network, at the end of each training epoch, the L2 norm of each convolution channel of each selected convolution layer in the selected convolution layer set in the resnet network is calculated, and a predetermined proportion of convolution channels with the lowest L2 norm are set to 0 to complete the sparse training of the current epoch. At the same time, at the beginning of the next training epoch, the convolution channels set to 0 are allowed to recover to a non-0 state.

[0070] Repeat the training for N epochs until the loss function converges, and complete the sparse training of the resnet network.

[0071] In the present application, the selected convolution layer set includes convolution layers in the resnet network located outside the residual module (res_block) and having an output not connected to the input of the residual module, and convolution layers located inside the residual module except the last layer.

[0072] Step 403: Prune the resnet network subjected to sparse training to obtain a visual tracking model.

[0073] In the embodiment, when pruning the resnet network that has undergone sparse training, the following steps are included:

[0074] In step 4031, the L2 norm of each convolution channel of each selected convolution layer in the selected convolution layer set in the resnet network is calculated, and a predetermined proportion of convolution channels with the lowest L2 norm are taken as a first pruning channel set.

[0075] In step 4032, the L2 norm of each convolution channel of each unselected convolution layer in the residual module with the same down-sampling that does not belong to the selected convolution layer set is calculated, and a predetermined proportion of convolution channels with the lowest L2 norm are taken as a second pruning channel set of each residual module with the same down-sampling.

[0076] For example, the network includes three residual modules with the same down-sampling, and according to the L2 norm calculation result, the residual module 1 has a second pruning channel set composed of channels 1, 2, 5, and 6, the residual module 2 has a second pruning channel set composed of channels 1, 2, 3, and 6, and the residual module 3 has a second pruning channel set composed of channels 1, 2, 5, and 6.

[0077] In step 4033, the intersection of the second pruning channel sets of all residual modules with the same down-sampling is calculated to obtain a third pruning channel set.

[0078] For example, the intersection of the second pruning channel sets of the three residual modules is calculated to obtain a third pruning channel set composed of channels 1 and 2 of each residual module.

[0079] In step 4034, the resnet network is pruned according to the first pruning channel set and the third pruning channel set to obtain a visual tracking model.

[0080] Since in the resnet network, the output result of the last layer (1*1, 256) in each residual module needs to be added to the input of the residual module, pruning this layer will cause misalignment with the input of the residual module, and then the input also needs to be pruned, which leads to the need for pruning the front residual module, and this linkage causes excessive pruning that affects the performance of the neural network. The embodiment can avoid the problem of network performance degradation caused by excessive pruning by merging the channels to be pruned through the intersection method in the presence of short_cut. Compared with embodiment 1, more convolution channels can be removed, the model is more lightweight, the running speed is improved, and the consumption of computing resources is reduced.

[0081] Similarly, the visual tracking model can also be retrained to restore the sparse network parameters to a normal distribution state to obtain a final visual tracking model.

[0082] Figure 5 An exemplary schematic diagram of a device for constructing a visual tracking model for implementing the method of embodiment 2 is shown in FIG. 2. As shown, the device includes:

[0083] A network training module 501 configured to train a visual tracking network using a sequence of images dataset;

[0084] Exemplarily, the visual tracking network is a resnet network. The module 501 constructs training samples using a sequence of images, which should contain multiple object classes, scales, poses, illuminations and occlusion conditions. The samples are labeled to indicate the location and scale of the target object, and the resnet network is trained using the training samples.

[0085] A sparse training module 502 configured to perform sparse training on the trained visual tracking network;

[0086] When performing sparse training on the trained resnet network, the module 502 calculates the L2 norm of each convolutional channel of each selected convolutional layer in the selected convolutional layer set at the end of each training epoch, and sets the predetermined proportion of convolutional channels with the lowest L2 norm to zero to complete the sparse training of the current epoch. At the same time, when the next training epoch starts, the convolutional channels set to zero are allowed to recover to a non-zero state.

[0087] The sparse training of the resnet network is completed by repeating N training epochs until the loss function converges.

[0088] A model generation module 503 configured to prune the visual tracking network after sparse training to obtain a visual tracking model;

[0089] The model generation module 503 includes:

[0090] A first calculation module 5031 for calculating the L2 norm of each convolutional channel of each selected convolutional layer in the selected convolutional layer set in the resnet network, and setting the predetermined proportion of convolutional channels with the lowest L2 norm as a first set of pruning channels;

[0091] A second calculation module 5032 for calculating the L2 norm of each convolutional channel of each unselected convolutional layer in the residual module with the same down-sampling in the resnet network, which does not belong to the selected convolutional layer set, and setting the predetermined proportion of convolutional channels with the lowest L2 norm as a second set of pruning channels for each residual module with the same down-sampling;

[0092] The intersection calculation module 5033 is configured to calculate the intersection of the second pruning channel set between all residual modules with the same down-sampling, to obtain a third pruning channel set;

[0093] The pruning module 5034 is configured to prune the resnet network according to the first pruning channel set and the third pruning channel set, to obtain a visual tracking model.

[0094] Although the present application is described in detail through the above embodiments, the present application is not limited to the above embodiments only, and the technical solutions of the embodiments of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for constructing a visual tracking model, characterized in that, The method comprises the following steps: constructing a visual tracking network and training the visual tracking network by using an image sequence dataset; wherein the visual tracking network is composed of multiple residual modules; performing sparse training on the trained visual tracking network, including: when each training round ends, calculating the L2 norm of each convolution channel of each selected convolution layer in a selected convolution layer set in the visual tracking network, and setting the convolution channels with the lowest L2 norm to 0 in a predetermined proportion to complete the sparse training of the current round; pruning the visual tracking network to obtain a visual tracking model, including: calculating the L2 norm of each convolution channel of each selected convolution layer in the selected convolution layer set in the visual tracking network, and setting the convolution channels with the lowest L2 norm in a predetermined proportion as a first pruning channel set; calculating the L2 norm of each convolution channel of each unselected convolution layer in the residual modules with the same down-sampling in the visual tracking network, which does not belong to the selected convolution layer set, and setting the convolution channels with the lowest L2 norm in a predetermined proportion as a second pruning channel set of each residual module with the same down-sampling; calculating the intersection of the second pruning channel sets between all residual modules with the same down-sampling to obtain a third pruning channel set; pruning the visual tracking network according to the first pruning channel set and the third pruning channel set to obtain a visual tracking model. 2.The visual tracking model construction method of claim 1, wherein, When the next training round starts, the convolution channels set to 0 are allowed to recover to a non-0 state. 3.The visual tracking model construction method of claim 1 or 2, characterized in that, The visual tracking network is a resnet network.

4. A visual tracking model construction apparatus characterized by comprising: The method comprises the following steps: a network training module configured to train a visual tracking network by using an image sequence dataset; wherein the visual tracking network is composed of multiple residual modules; a sparse training module configured to perform sparse training on the trained visual tracking network, including: when each training round ends, calculating the L2 norm of each convolution channel of each selected convolution layer in a selected convolution layer set in the visual tracking network, and setting the convolution channels with the lowest L2 norm to 0 in a predetermined proportion to complete the sparse training of the current round; a model generation module configured to prune the visual tracking network to obtain a visual tracking model, including: a first calculation module for calculating the L2 norm of each convolution channel of each selected convolution layer in the selected convolution layer set in the visual tracking network, and setting the convolution channels with the lowest L2 norm in a predetermined proportion as a first pruning channel set; a second calculation module for calculating the L2 norm of each convolution channel of each unselected convolution layer in the residual modules with the same down-sampling in the visual tracking network, which does not belong to the selected convolution layer set, and setting the convolution channels with the lowest L2 norm in a predetermined proportion as a second pruning channel set of each residual module with the same down-sampling; an intersection calculation module for calculating the intersection of the second pruning channel sets between all residual modules with the same down-sampling to obtain a third pruning channel set; A pruning module is configured to prune the visual tracking network according to the first pruning channel set and the third pruning channel set to obtain a visual tracking model.

5. The visual tracking model building apparatus according to claim 4, wherein, At the beginning of the next training round, the convolutional channels that are set to 0 are allowed to recover to a non-0 state.

6. The visual tracking model construction apparatus according to claim 4 or 5, characterized by, The visual tracking network is a resnet network.

Images