Lightweight target tracking method for mobile terminal devices

Abstract

The present invention belongs to the field of computer vision technology, and specifically is a lightweight tracking method for mobile terminal devices. The present invention uses a pre-trained detection algorithm whose complexity can adapt to the mobile terminal device to detect the target to be tracked in the video to be processed, and provides an initialized framed target in real time and with high accuracy; then, in the tracking network, a lightweight twin network is used to extract and match features between the initial object and the image of the subsequent frame of the video, and the position with the highest matching degree is found as the position of the predicted target. The method of the present invention can provide high tracking accuracy and run on mobile devices in real time and efficiently.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a lightweight tracking method for mobile terminal equipment. Background Art

[0002] Object tracking, a popular research topic in computer vision and image processing, has broad applications in transportation, surveillance, security, robotics, animation, and other fields. Traditional tracking algorithms offer advantages in mature, simple models, and ease of deployment. However, they generally suffer from poor tracking accuracy and real-time performance, and are unable to address common tracking issues and challenges. With the advancement of chip technology and the increased computing power of computing devices, deep learning neural networks have emerged. Deep learning networks offer superior tracking accuracy for object tracking compared to traditional algorithms and correlation filters. Research on object tracking holds great practical value and promise, prompting numerous researchers to optimize and research deep learning tracking algorithms.

[0003] A common architecture used in deep learning networks for tracking algorithms is the Siamese network. Its implementation involves simultaneously feeding the network the first frame and subsequent frames, with the first frame serving as a continuous input template. By comparing the two, the region in the subsequent frames that has the highest similarity to the target selected in the template is identified as the bounding box of the tracked target. The more accurate the Siamese network's matching, the higher the accuracy of the tracking network. In recent years, tracking algorithms have been constantly updated. The goal of these iterations is to improve the precision and accuracy of target tracking and better address issues such as occlusion, illumination fluctuations, deformation, motion blur, and out-of-field viewing. However, improved accuracy often requires a more complex network structure, making it difficult to deploy tracking algorithms on mobile edge devices with limited computing power. Therefore, designing a tracking network with low computational complexity, parameter count, and computing power requirements is crucial to meet these requirements. Furthermore, to address the pain point of manually selecting a bounding box to initialize the tracked target, designing an object tracking system that automatically initializes the target can greatly improve the practicality of the tracking network. Summary of the Invention

[0004] The purpose of the present invention is to provide a lightweight tracking method for mobile terminal devices with low algorithm complexity and small computing power requirements, so as to solve the difficulty of deploying single target tracking algorithms on mobile devices.

[0005] The lightweight tracking method for mobile terminal devices provided by the present invention uses a pre-trained detection model with a complexity that can adapt to the mobile terminal device to detect the target to be tracked in the video to be processed, and provides an initialized frame target in real time and with high precision. Then, in the tracking network, a lightweight twin network is used to extract and match features between the initial object and the images of subsequent frames of the video, and the position with the highest matching degree is found as the predicted target position. The specific steps are as follows:

[0006] Step 1: Pre-train a model for detecting targets of a specified category and deploy the detection model on a mobile device.

[0007] Step 2: Initialize the detection of the tracking target; use the model trained in step 1 to detect the target to be tracked in the video stream, and select the target with the highest confidence among all possible detected targets as the initial tracking target to complete the task of framing the target in the initial frame;

[0008] Step 3: Target tracking in the video stream; place the target framed in the initial frame in step 2 and the image of the next frame into the target tracking twin network (also known as the tracking network) for feature extraction and matching, and select the position with the highest matching degree as the predicted target position; repeat step 3 to complete target tracking in subsequent video streams.

[0009] Furthermore, in step 1, the pre-training model for detecting the specified category of targets is specifically performed as follows:

[0010] Step 1.1: Prepare the image data of the target object category to be tracked, organize it into a dataset, annotate the specific location and area of ​​the target of interest in each image data, and generate a label text file;

[0011] Step 1.2: Use the labeled dataset to train the detection network. The detection network is based on YOLO and the backbone network is replaced by a lightweight Shuffle Net network. After the image is input, a four-layer convolutional neural network Shuffle Net is used in the backbone network. The number of input channels of the four-layer convolutional network is 24, 116, 232 and 464 respectively. A feature pyramid network combining FPN and PAN is added to the last three layers of the convolutional network. In the sampling process, bilinear interpolation is used instead of pooling to effectively reduce the computational complexity of the feature extraction network. The FPN network continuously samples the semantic information of the high-level network and transmits it to the low-level network. The PAN network downsamples the semantic information of the low-level network and transmits it to the high-level network. The two complement each other, fuse the information between different layers of the network, and enhance the target positioning information of the network. Because in the single-stage detection network, most of the predicted result boxes are negative samples, and only a small part are positive samples, and the positioning quality and the specific classification score will affect the training of the model, in order to avoid the low accuracy of the training model caused by the above problems, the loss function uses Generalized Focal Loss (GFL), the formula is given in (1); the output of the detection network includes: the positioning box, the background information before regression, and the category class information;

[0012] (1)

[0013] in, y is the actual label, y l and y r Representatives and y The nearest predicted left and right position of the label, and yes y l and y r The probability of β It is the absolute distance between the prediction result and the label. The loss function combines the positioning quality and classification loss to form the Generalized Focal loss.

[0014] Step 1.3: Evaluate the performance of the detection network and select the network with the best detection accuracy as the detection algorithm for subsequent deployment; the trained network can better achieve the detection function of the relevant targets.

[0015] Furthermore, in step 2, the detection of the initial tracking target is specifically performed as follows:

[0016] Step 2.1: Pass the first frame of the video stream into the detection network;

[0017] Step 2.2: The detection network extracts and matches features from the input image to detect the target.

[0018] Step 2.3: Select the target to be tracked in the first frame image, return the coordinate information of the upper left corner of the target object (x, y) and the width and height (h, w) of the target object selection, and pass them to the tracking network in step 3;

[0019] Step 2.4: If the target detection prediction fails, continue to repeat the above step 2.2 until the target appears.

[0020] Furthermore, in step 3, the target tracking of the video stream is specifically performed as follows:

[0021] Step 3.1: Improve the backbone network of the LightTrack target tracking network based on the twin network to achieve lightweight target tracking network (i.e. as a tracking network); specifically, input a picture of size 256*256 and number of channels 3, use a six-layer convolutional network in the backbone network, use the Inverted Residual Blocks structure, depth-wise separable convolutions and Mobile net V2 (MBConv) convolution modules as core modules, use 3*3 convolution in the first layer, depth-wise separable convolution in the second layer, and MBConv convolution modules in the next four layers; introduce the attention mechanism module after the backbone network, where the attention mechanism module includes channel attention mechanism and spatial attention mechanism; among them:

[0022] The implementation process of the channel attention mechanism is as follows: the features of each layer are globally average pooled and globally max pooled respectively, the two pooling results are processed and added using a shared fully connected layer, and then a sigmoid is applied to obtain the weight of each channel of the input feature layer (between 0 and 1), and finally this weight is multiplied by the original input feature layer;

[0023] The spatial attention mechanism is implemented by taking the maximum and average values ​​across all feature points in each input feature layer, stacking the two results, and rescaling the number of channels through a 1x1 convolutional layer. A sigmoid function is then applied to obtain the weights of the feature points (between 0 and 1), which are then multiplied by the original input feature layer. The introduction of the attention mechanism can highlight important features.

[0024] The prediction head network uses a three-layer convolutional network. The first and second layers use depthwise separable convolutional modules, and the third layer uses a 3x3 convolutional module. The classification head network has a similar network structure to the prediction head. The overall prediction head network predicts and outputs the foreground and background, while the classification head network predicts and outputs the target position information.

[0025] Step 3.2: Based on the above, in the training phase, first use ImageNET to pre-train a backbone network;

[0026] Step 3.3: Use common tracking datasets, including COCO, GOT-10K, Image NET DET, etc., to train the head network and backbone network, and use the neural network architecture search method to find the best tracking network structure;

[0027] Step 3.4: The target to be tracked detected in step 2 is used as the initial tracking object of the tracking network. It and the image of the subsequent frame of the video stream are used as the input of the tracking network for feature extraction;

[0028] Step 3.5: Complete the feature matching of the two input parts and obtain the prediction result with the highest score as the position prediction of the tracked target in the subsequent frame image.

[0029] This paper utilizes an optimized lightweight object detection network to detect and identify objects in the initial frame image, automating the initial frame functionality provided by the target tracking network. Using an optimized twin tracking network, it provides a lightweight target tracking algorithm suitable for mobile devices. Compared to other algorithms, this lightweight tracking algorithm demonstrates superior accuracy and can address common target tracking challenges, such as deformation, inversion, angle changes, and occlusion. It also performs well in long-term tracking. Most importantly, it reduces the number of network model parameters and computational power, thereby achieving a lightweight tracking model and making it more suitable for deployment on mobile edge devices with limited computing power. In comparisons of this method with other high-performing tracking algorithms on the Got10k dataset, the method achieves similar or even higher tracking accuracy (approximately 0.613) despite having approximately 2M parameters and 530M FLOPs of computation, far less than other tracking algorithms. Therefore, this method is easier to deploy on mobile edge devices with limited computing power and ensures guaranteed tracking accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a schematic diagram of the overall flow of the target tracking algorithm provided by the present invention.

[0031] Figure 2 This is a flow chart of the detection algorithm provided by the present invention.

[0032] Figure 3 This is a flow chart of the tracking algorithm provided by the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention will be further described below in conjunction with embodiments and drawings.

[0034] Figure 1 This is a schematic diagram of the overall flow of the target tracking algorithm provided by the present invention. Figure 1 As shown in the figure, this paper provides a lightweight target tracking algorithm applied on mobile terminal devices. The specific steps are as follows:

[0035] Step 1: Deploy the pre-trained model for classifying and detecting objects of a specified category to the mobile device.

[0036] Specific steps: Prepare image data of the target object category to be tracked and organize it into a dataset. Use the image annotation tool Labelme to annotate the specific location and area of ​​the target of interest in each image and generate a label text file. This file contains the category, upper-left corner coordinates (x, y), and length and width (w, h) of each image. This labeled dataset is then used to train an object detection network. The object detection network used is based on YOLO, but with a lightweight Shuffle Net as the backbone. This Shuffle Net reduces the complexity of the detection network model and makes it easier to deploy on mobile devices. The detection network is then evaluated for performance, and the one with the highest detection accuracy is selected as the detection algorithm for subsequent deployment. The trained network should be able to effectively detect the relevant objects.

[0037] Step 2: Use the detection model trained in step 1 to detect the target to be tracked in the video stream. Select the target with the highest confidence among all possible detected targets as the target for initial tracking. The higher the confidence, the more accurate the target detected by the network model is. The task of initial frame target identification is completed by screening the confidence. Specific steps:

[0038] The first frame of the video stream is passed into the detection network to detect whether there is an object of interest in the image; the pre-trained detection model learns the features of the target to be tracked, so it can extract features from the input image, match features, and thus detect the target; the target to be tracked predicted by the detection model in the first frame is selected, and according to the target with the highest confidence ranking, the upper left corner coordinate information (x, y) of the target object and the width and height (h, w) of the selected target object are returned and passed to the tracking network in step 3; if the target detection fails in the first frame, that is, the target to be tracked is not predicted or the confidence of the detected target is too low, then the above detection part is repeated until a target that meets the requirements is detected.

[0039] Step 3: Place the target framed in the initial frame in step 2 and the image of the next frame into the target tracking twin network for feature extraction and matching. Select the position with the highest matching degree as the predicted target position, and repeat step 3 to complete the tracking of subsequent video streams. Specific steps:

[0040] First, the LightTrack target tracking network based on the twin network is improved to achieve lightweight target tracking network. Specifically, the backbone network structure consists of six convolutional layers, using the Inverted Residual Blocks (IRBs) structure, Depth-wise Separable Convolutions (DSCs), and Mobile Net V2 (MBConv) convolution modules as core modules. The first layer uses 3*3 convolutions, the second layer uses Depth-wise Separable Convolutions, and the next four layers all use MBConv convolution modules.

[0041] The attention mechanism module is introduced after the backbone network. The attention mechanism module here includes channel attention mechanism and spatial attention mechanism; among them:

[0042] The implementation process of the channel attention mechanism is as follows: the features of each layer are globally average pooled and globally max pooled respectively, the two pooling results are processed and added using a shared fully connected layer, and then a sigmoid is applied to obtain the weight of each channel of the input feature layer (between 0 and 1), and finally this weight is multiplied by the original input feature layer;

[0043] The spatial attention mechanism is implemented by taking the maximum and average values ​​across all feature points in each input feature layer, stacking the two results, and rescaling the number of channels through a 1x1 convolutional layer. A sigmoid function is then applied to obtain the weights of the feature points (between 0 and 1), which are then multiplied by the original input feature layer. The introduction of the attention mechanism can highlight important features.

[0044] In the prediction head network, a three-layer convolutional network is used. The first and second layers use depthwise separable convolution modules, and the third layer uses a 3*3 convolution module. In the classification head network, similar to the network structure of the prediction head, the overall prediction head network predicts and outputs the foreground and background, and the classification head network predicts and outputs the target position information, further compressing the above-mentioned tracking model as a whole to reduce its parameter and computational complexity.

[0045] Based on the aforementioned network structure, the training phase first pre-trains a backbone network using ImageNET. Specific parameters include an image size of 3*256*256, 1000 image categories, a crop ratio of 0.875, and 1280 features. Common tracking datasets, including COCO, GOT-10K, and ImageNET DET, are then used to train the head network and backbone network. A neural network architecture search is used to find the optimal tracking network architecture. Specific parameters for the tracking network include a search region size of 256, a template size of 128, a stride of 16, and a maximum search network model computational overhead of 470M FLOPs, with a parameter count of less than 2M. The target to be tracked detected in step 2 is used as the initial tracking object for the tracking network. This object, along with images from subsequent frames of the video stream, serves as the tracking network input for feature extraction. The tracking model of the twin network performs feature matching on the two inputs, and the prediction with the highest score is used as the position prediction for the target in subsequent frames. The network prediction is repeated with the template in subsequent frames for subsequent tracking predictions.

[0046] When deploying on mobile devices, we used Android Studio as the mobile app development tool and NCNN as the mobile algorithm deployment tool to convert the network model into a mobile model. The reason for using this tool is that the NCNN framework is not only a high-performance neural network forward computing framework optimized for mobile devices, but also has no third-party dependencies, is cross-platform, and its mobile CPU processing speed is faster than all currently known open source frameworks. The deployed mobile app achieved excellent tracking accuracy and stability.

Claims

1. A lightweight tracking method for a mobile terminal device, characterized in that: Use a pre-trained detection model with a complexity that can adapt to mobile terminal devices to detect the target to be tracked in the video to be processed, and provide an initial frame target in real time and with high precision. Then, in the tracking network, use a lightweight twin network to extract and match features between the initial object and the images of subsequent frames of the video, and find the position with the highest matching degree as the predicted target location. The specific steps are as follows: Step 1: Pre-train a detection model for detecting targets of a specified category and deploy the detection model on a mobile device. The detection model is based on YOLO, but with a lightweight Shuffle Net as the backbone network. After inputting an image, the Shuffle Net uses a four-layer convolutional neural network as the backbone network. The input channels of the four layers are 24, 116, 232, and 464, respectively. A feature pyramid network combining FPN and PAN is added to the last three layers of the convolutional network. In the sampling process, bilinear interpolation is used. The FPN network upsamples the semantic information of the high-level network and continuously transmits it to the low-level network. The PAN network downsamples the semantic information of the low-level network and transmits it to the high-level network. The two complement each other, integrating the information between different layers of the network and enhancing the target positioning information of the network. Step 2: Initialize the detection of the tracking target; use the model trained in step 1 to detect the target to be tracked in the video stream, and select the target with the highest confidence among all possible detected targets as the initial tracking target to complete the task of framing the target in the initial frame; Step 3: Target tracking in the video stream. The target identified in the initial frame in step 2 and the image in the next frame are placed in the target tracking twin network, i.e., the tracking network, for feature extraction and matching. The position with the highest matching degree is selected as the predicted target position. Repeat step 3 to complete target tracking in subsequent video streams. The LightTrack target tracking network based on the twin network is improved in the backbone network to achieve lightweight target tracking network, namely as the tracking network; specifically, the input size is 256*256, the number of channels is 3, a six-layer convolutional network is used in the backbone network, and the convolution module MBConv composed of the inverse residual network structure, depthwise separable convolution and Mobile net V2 is used as the core module; the first layer uses 3*3 convolution, the second layer uses depthwise separable convolution, and the following four layers all use MBConv convolution modules; the attention mechanism module is introduced after the backbone network, and the attention mechanism module here includes channel attention mechanism and spatial attention mechanism; In the prediction head network, a three-layer convolutional network is used. The first and second layers use depth-wise separable convolution modules, and the third layer uses a 3*3 convolution module. In the classification head network, the overall prediction head network predicts and outputs the foreground and background, and the classification head network predicts and outputs the target position information.

2. The lightweight tracking method for a mobile terminal device according to claim 1, characterized in that: The pre-training model described in step 1 to detect the specified category of targets is as follows: Step 1.1: Prepare the image data of the target object category to be tracked, organize it into a dataset, annotate the specific location and area of ​​the target of interest in each image data, and generate a label text file; Step 1.2: Use the labeled dataset to train the detection model; The loss function uses GFL, and the formula is given in (1); The output of the detection model includes: positioning box, regression background information, and category information; （1） in, y is the actual label, y l and y r Representatives and y The nearest predicted left and right position of the label, and yes y l and y r The probability of β It is the absolute distance between the prediction result and the label. The loss function combines the positioning quality and classification loss to form the Generalized Focal loss. Step 1.3: Evaluate the performance of the detection model and select the model with the best detection accuracy as the detection method for subsequent deployment; the trained model can achieve the detection function of the relevant target.

3. The lightweight tracking method for a mobile terminal device according to claim 2, characterized in that: Initialize the detection of the tracking target described in step 2. The specific process is as follows: Step 2.1: Pass the first frame of the video stream into the detection model; Step 2.2: The detection model extracts and matches features from the input image to detect the target. Step 2.3: Select the target to be tracked in the first frame image, return the coordinate information of the upper left corner of the target object (x, y) and the width and height (h, w) of the target object selection, and pass them to the tracking network in step 3; Step 2.4: If the target detection prediction fails, continue to repeat the above step 2.2 until the target appears.

4. The lightweight tracking method for a mobile terminal device according to claim 3, characterized in that: The specific process of target tracking in the video stream described in step 3 is as follows: Step 3.1: The implementation process of the channel attention mechanism is as follows: perform global average pooling and global maximum pooling on the features of each layer respectively, use a shared fully connected layer to process the two pooling results and add them together, then take sigmoid to obtain the weight of each channel of the input feature layer, and finally multiply this weight with the original input feature layer; The spatial attention mechanism is implemented by taking the maximum and average values ​​of all feature points in each input feature layer, stacking the two results together, and rescaling the number of channels through a 1x1 convolutional layer. A sigmoid function is then applied to obtain the weights of the feature points, which are then multiplied by the original input feature layer. The introduction of the attention mechanism can highlight important features. Step 3.2: Based on the above, in the training phase, first use ImageNET to pre-train the backbone network; Step 3.3: Use tracking datasets selected from COCO, GOT-10K, and Image NET DET to train the head network and backbone network, and use neural network architecture search to find the best tracking network architecture; Step 3.4: The target to be tracked detected in step 2 is used as the initial tracking object of the tracking network. It and the image of the subsequent frame of the video stream are used as the input of the tracking network for feature extraction; Step 3.5: Complete the feature matching of the two input parts and obtain the prediction result with the highest score as the position prediction of the tracked target in the subsequent frame image.

Images