A real-time target tracking method and embedded device based on twin network
By combining depth-separable correlation filtering and point-by-point correlation filtering in target tracking, the backbone network is extracted using MobileNet V3-Small feature and lightweight embedded deployment, the problem of insufficient accuracy and real-time in complex scenarios is solved. , realizes efficient and accurate target tracking, suitable for embedded devices.
Patent Information
- Application Number
- CN202411176569.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2044-08-26
AI Technical Summary
Traditional target tracking methods reduce accuracy and lack real-time performance in complex scenarios. Traditional twin networks have large parameters and high computational complexity when dealing with complex scenarios, which is difficult to meet real-time requirements. In addition, deep neural networks have high requirements for computing performance. Traditional CPUs , ARM and DSP are difficult to meet the real-time processing needs of the industry.
Using a real-time target tracking method based on twin networks, combining deep separation correlation filtering and point-by-point correlation filtering, MobileNet V3-Small is used as feature extraction backbone network, enhances robustness through multi-scale feature aggregation, and lightweight embedded deployment The method converts the model to an RKNN model and assigns processing tasks to the CPU, GPU, and NPU for efficient operation.
It realizes efficient and accurate target tracking in complex scenarios, reduces the calculation amount, improves the target positioning accuracy, is suitable for embedded devices, meets the requirements of lightweight and efficient algorithms, and improves the operation of the model on embedded devices. Efficiency and stability.
Smart Images
Figure CN118674752B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of target tracking, and in particular to a real-time target tracking method and an embedded device based on a twin network. Background Art
[0002] Visual target tracking is an important research direction in computer vision, and has extremely wide applications in many fields such as military unmanned aerial vehicles, precision guidance, air warning, civilian video surveillance, human-computer interaction, and unmanned driving. However, traditional target tracking methods often face the problems of reduced accuracy and insufficient real-time performance when dealing with complex scenes (such as target occlusion, scale changes, and rapid movement). Therefore, it is particularly important to develop an efficient, accurate, and robust real-time target tracking technology.
[0003] Traditional target tracking algorithms are usually based on manual features, including edges, textures, color histograms, etc. These algorithms use hand-designed models and motion estimation strategies such as mean-shift and Kalman filtering to predict the position of the target in the video frame. KCF algorithm, STC algorithm and DSST are based on correlation filtering methods to implement target trackers in industrial applications. However, the performance of manual features is limited when dealing with complex scenes, lighting changes and other problems, and the prediction of target edges is poor. It cannot cope with complex scale changes, and the target will gradually be lost due to the long-term error accumulation of video target tracking.
[0004] In recent years, with the rapid development of deep learning technology, target tracking methods based on deep neural networks have become a research hotspot in the field of computer vision. This type of method can automatically learn and extract rich feature representations from a large amount of training data, significantly improving the accuracy and robustness of target tracking. Among many deep learning architectures, the Siamese Network stands out in the target tracking task due to its unique structural design and efficient performance. The Siamese Network constructs two branches with shared parameters, the template branch and the search branch, to process the target template image and the search area image respectively. After the two branches extract their respective features, they determine whether there is a target in the search area by calculating the similarity between the features and predict the specific location of the target. This structure not only simplifies the tracking problem and converts it into a feature similarity matching problem, but also greatly improves the speed and accuracy of target tracking.
[0005] However, although the twin network has made significant progress in the field of target tracking, the traditional twin network method still has certain limitations when dealing with complex scenes. On the one hand, in order to improve the accuracy and robustness of tracking, the twin network often needs to build a more complex network structure, which leads to a large number of algorithm parameters and high computational complexity, making it difficult to meet real-time requirements. On the other hand, as the target's appearance, scale, and motion pattern change, the traditional twin network may not be able to effectively adapt in terms of feature extraction and similarity calculation, resulting in a decrease in tracking performance.
[0006] In addition, in the industry, the deep neural network (DNN) target tracking algorithm has extremely high computing performance requirements, and traditional CPU, ARM, DSP and other platforms are difficult to meet real-time processing needs. To this end, intelligent processing chips such as GPU, FPGA and NPU have become key. However, the high-performance GPU market is monopolized by foreign countries. Although FPGA is powerful, it is costly, complex in design and limited in flexibility. In contrast, my country's independently developed NPU has achieved a good balance between processing performance and power consumption, is suitable for embedded applications, and can efficiently deploy deep neural networks to meet the industry's needs for real-time target tracking. Therefore, the market needs an embedded deployment solution for real-time target tracking methods based on NPU. Summary of the invention
[0007] The purpose of the present invention is to address the problems existing in the prior art and provide a real-time target tracking method and embedded device based on a twin network. By combining improved feature extraction, similarity calculation and multi-tasking processing strategies, efficient and accurate target tracking in complex scenarios is achieved, while meeting the requirements of embedded devices for lightweight and efficient algorithms.
[0008] To achieve the above object, the technical solution adopted by the present invention is:
[0009] A real-time target tracking method based on a twin network includes: reading the first frame of a video or image sequence and determining a tracking target frame; inputting the target frame of the first frame as a template into a template feature backbone network to extract features and obtain a target feature map; reading the next frame, constructing a search area in proportion according to the target frame in the previous frame, and extracting features from the search area using a search area feature backbone network to obtain a search area feature map; using deep separable correlation filtering and point-by-point correlation filtering to calculate the similarity between the target feature map and the search area feature map; based on the similarity calculation result, performing information calculation of a target center position classification branch and a target coordinate frame regression branch to determine the position of the target in the current frame;
[0010] MobileNet V3-Small is used as the template feature backbone network and the search area feature backbone network. In the feature extraction process, three feature extraction layers are used to extract features respectively, and the output of each feature extraction layer is saved to obtain the first layer feature map, the second layer feature map and the third layer feature map. The multi-scale feature aggregation method is used to splice the three layers of feature maps to enhance the robustness of feature extraction.
[0011] The multi-scale feature aggregation method is used to splice the first layer feature map, the second layer feature map, and the third layer feature map, including: reading the first layer feature map, performing two convolution operations to obtain preliminary depth features; reorganizing the information of the preliminary depth features, dividing the feature space and splicing along the channel dimension to obtain the first output feature; reading the second and third layer feature maps, performing convolution operations to smooth the features and respectively obtain the second output feature and the third output feature; splicing the three layers of output features according to the channel dimension to obtain a fused feature; connecting a convolution layer after the fused feature, reducing the number of channels through the convolution operation, extracting and integrating features, and obtaining a final feature map for subsequent target tracking tasks;
[0012] The information of the preliminary depth features is reorganized, the feature space is divided and spliced along the channel dimension to obtain the first output feature, including: a single feature space in the depth feature map is evenly divided into n smaller feature blocks; these smaller feature blocks are spliced on the channel dimension according to the division order to form a first output feature with a channel number n times the number of original feature channels.
[0013] Use visualization tools to visualize the three feature maps. The first layer of feature maps is a shallow feature map, which mainly focuses on the relationship between the target and the background of the whole image. The second layer of feature maps is a mid-level feature map, which gradually focuses on the target itself, focusing on the outline and key shape information of the target. The third layer of feature maps is more focused on the modeling of the target itself.
[0014] The similarity calculation using deep separable correlation filtering and point-by-point correlation filtering includes: the deep separable correlation filtering calculates the similarity between the template and the search area within the spatial context range to obtain a classification similarity score; the point-by-point correlation filtering calculates the spatial similarity between the template and the search area in the channel dimension to obtain a regression similarity score;
[0015] The information calculation of the target center position classification branch and the target coordinate frame regression branch is performed based on the similarity calculation result, including: splicing the classification similarity score and the regression similarity score in a channel splicing manner; mapping the spliced features to the target center position classification branch and the target coordinate frame regression branch; finding the highest point coordinates on the confidence map of the target center position classification branch, and then finding and determining the precise coordinate frame of the target according to the coordinate information in the target coordinate frame regression branch;
[0016] After the information calculation of the target center position classification branch and the target coordinate frame regression branch is completed, it includes: re-constructing the search area in proportion according to the target coordinate frame calculated in the current frame, completing the update of the search area, and repeating the next frame image target positioning until the video or image sequence tracking is completed, thereby realizing continuous tracking of the target.
[0017] A lightweight embedded deployment method, comprising: converting a trained twin network model into an .onnx model in ONNX format; then, using the RKNN Toolkit tool provided by the Rockchip platform to further convert the .onnx model into an RKNN model; transferring the converted RKNN model to an RK3588M device; calling an NPU interface to load the twin network model; loading the logic processing steps onto the CPU for execution; and loading the image processing steps onto the GPU for execution.
[0018] An embedded device includes: a first NPU module, used to run the template feature backbone network; a second NPU module, used to run the search area feature backbone network; a third NPU module, used to calculate the similarity between the target feature map and the search area feature map by using deep separable correlation filtering and point-by-point correlation filtering; a CPU module, used to read the image and transmit it to the corresponding NPU module; and also used to perform information calculation of the target center position classification branch and the target coordinate frame regression branch based on the similarity calculation result to accurately determine the position of the target in the current frame; a GPU module, used for image preprocessing and post-processing.
[0019] Compared with the prior art, the present invention has the following beneficial effects:
[0020] 1. By adopting deep separable correlation filtering and point-by-point correlation filtering technology, the amount of calculation is effectively reduced while the accuracy of target positioning is improved, so that the tracking algorithm can achieve fast and accurate target tracking in complex scenarios;
[0021] 2. The combined use of deep separable correlation filtering and point-by-point correlation filtering not only improves the ability to capture spatial position information, but also enhances the coupling of local features and improves the model's ability to represent small targets;
[0022] 3. MobileNet V3-Small is used as the feature extraction backbone network. This is a lightweight convolutional neural network that can significantly reduce computational complexity and model size while maintaining high performance, making it very suitable for mobile and embedded vision applications;
[0023] 4. The multi-scale feature aggregation method enhances the robustness of feature extraction. By splicing feature maps at different levels, it integrates rich multi-scale information, enabling the model to capture information at different abstract levels from low to high levels, and enhancing the model's ability to cope with complex situations such as target scale changes and occlusions;
[0024] 5. Provides a lightweight embedded deployment method, including model format conversion, tool usage, model loading and task allocation, so that the trained model can be efficiently integrated into embedded devices such as RK3588M;
[0025] 6. By allocating different processing tasks to different processors such as CPU, GPU and NPU, the maximum utilization of resources is achieved, and the running efficiency and stability of the model on embedded devices are improved;
[0026] 7. It provides visualization operations on feature maps to help understand and analyze the information levels of feature maps at different levels, and verifies the effectiveness of the multi-scale hierarchical feature aggregation module. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0028] Figure 1 This is a flow chart of a target tracking method in one embodiment of the present application;
[0029] Figure 2 This is a schematic diagram of the structure of MobileNet V3-Small in one embodiment of the present application;
[0030] Figure 3 This is a schematic diagram of the process of a multi-scale feature aggregation method in one embodiment of the present application;
[0031] Figure 4 This is a flow chart of an information reorganization method in one embodiment of the present application;
[0032] Figure 5This is a schematic diagram of the calculation process of deep separable correlation filtering and point-by-point correlation filtering in one embodiment of the present application;
[0033] Figure 6 This is a schematic diagram of an embedded device in an embodiment of the present application. DETAILED DESCRIPTION
[0034] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0035] like Figure 1 As shown, in the first aspect of the present application, a real-time target tracking method based on a twin network is provided, comprising the following steps:
[0036] S1. Read the first frame of a video or image sequence and determine the tracking target frame.
[0037] Specifically, if it is a video file, you can use the video processing library to read it frame by frame; if it is an image sequence, you can load the first image in sequence.
[0038] In the first frame, the target is detected manually or automatically by an algorithm and an initial target frame is generated, which is input into the subsequent tracking algorithm as a template to search and locate the same target in subsequent frames;
[0039] S2. Input the target frame of the first frame image as a template into the template feature backbone network to extract features and obtain a target feature map.
[0040] Specifically, the twin network of this embodiment is composed of two backbone networks (such as MobileNet V3-Small) with shared parameters, and each backbone network processes the template image and the search area image respectively; wherein the template image is the image within the tracking target box determined in the initial frame, and the search area image is the image area that may contain the target in the subsequent frames.
[0041] The feature backbone network outputs one or more feature maps to extract high-dimensional features of the template target, including key information of the target, such as edges, textures, and shapes, so as to facilitate identification and tracking of the target in subsequent frames;
[0042] S3, read the next frame image, construct the search area in proportion according to the target frame in the previous frame, use the search area feature backbone network to extract features of the search area, and obtain the search area feature map.
[0043] Specifically, the search area is constructed in proportion to the target frame determined in the previous frame. The size of the search area should be set to be slightly larger than the target frame so as to include all possible positions where the target may appear in the next frame. The proportion is set to 2, for example.
[0044] Similar to the template feature backbone network, the search area feature backbone network also has the same network structure, extracting high-dimensional features of the search area, which also contains information such as edges, textures, shapes, etc., about all possible target locations in the search area.
[0045] After feature extraction, the search area feature backbone network outputs one or more feature maps, which encode the key information of all possible target locations in the search area and will be used for subsequent similarity comparison with the template feature map to determine the specific location of the target in the next frame;
[0046] S4. Calculate the similarity between the target feature map and the search area feature map by using depth separable correlation filtering and point-by-point correlation filtering.
[0047] Specifically, the deep separable correlation filter performs deep filtering calculations on the spatial similarity of template features and search area features within the target context space; while reducing the amount of calculation, it effectively captures the spatial context information between the template and the search area.
[0048] Point-by-point correlation filtering is a 1×1 similarity calculation operation. Following the depth filter, it performs point-by-point similarity calculations on the template features and the search area features in the channel dimension, which helps to further explore the coupling of local features and improve the network model’s ability to represent small targets.
[0049] By combining the calculation results of the depth-separable correlation filter and the point-by-point correlation filter, a similarity score map between the template feature map and the search area feature map is obtained, and each value on this score map is used to represent the similarity between the template image and the search area image at the corresponding position;
[0050] S5. Based on the similarity calculation result, information calculation of the target center position classification branch and the target coordinate frame regression branch is performed to determine the position of the target in the current frame.
[0051] Specifically, the classification branch searches for the point with the highest confidence in the similarity score map, which represents the most likely center position of the target. The regression branch predicts the exact position of the target in the current frame based on the similarity score map using the parameters learned through training.
[0052] In some embodiments, MobileNet V3-Small is used as the template feature backbone network and the search area feature backbone network; in the feature extraction process, three feature extraction layers are used to extract features respectively, and the output of each feature extraction layer is saved to obtain the first layer feature map, the second layer feature map and the third layer feature map; a multi-scale feature aggregation method is used to splice the three layers of feature maps to enhance the robustness of feature extraction.
[0053] Specifically, MobileNet V3-Small is a lightweight convolutional neural network, which is particularly suitable for mobile and embedded vision applications. It can significantly reduce computational complexity and model size while maintaining high performance. The three feature extraction layers are used to extract features of different scales respectively, capturing information at different abstraction levels from low to high levels. The multi-scale feature aggregation method is used to splice the feature maps output by the three feature extraction layers. This step integrates feature information of different scales and different abstraction levels to enhance the robustness of feature extraction. Through the splicing operation, a feature map containing more levels and richer information can be obtained;
[0054] like Figure 2 As shown in the figure, compared with the MobileNet V1 structure, the network structure of MobileNet V3 adds the SE (Squeeze-and-Excitation Enhanced) module and changes the activation function. This improves the network's ability to express target features, effectively improves the tracking performance of the convolutional neural network in target tracking tasks, and brings practical and effective gains to the method of this embodiment;
[0055] like Figure 3 As shown, in some embodiments, the multi-scale feature aggregation method is used to splice the first layer feature map, the second layer feature map, and the third layer feature map, which specifically includes the following five steps:
[0056] (1) First layer feature map processing:
[0057] Read the first layer feature map, and set its channel number to C1;
[0058] Perform two convolution operations on the first layer feature map;
[0059] The number of input and output channels of the first convolution operation is C1, keeping the original channel information unchanged;
[0060] The number of input channels of the second convolution operation is C1, and the number of output channels is set to 64;
[0061] After the above two convolution operations, a preliminary depth feature with a shape of [64,6,6] is obtained;
[0062] (2) Reorganize the information of the preliminary deep features of shape [64,6,6]. The specific operation is:
[0063] Divide the 6x6 feature space into 4 groups of 3x3 features;
[0064] In the order of grouping, concatenate these four groups of 3x3 features along the channel dimension;
[0065] After information reorganization, the first layer feature is transformed from a single 6x6 feature to four 3x3 features, and the number of channels is expanded from 64 to 256 (i.e. 64*4), resulting in a first output feature with a shape of [256,3,3].
[0066] (3) Processing the second and third layer feature maps: Read the second layer feature map and the third layer feature map, and set their channel numbers to C2 and C3 respectively;
[0067] Convolution operations are constructed for the second-layer feature maps and the third-layer feature maps respectively, and their input channels and output channels are the same as their own feature channels, that is, the input channels and output channels are C2 and C3 respectively;
[0068] After performing the convolution operation, a smoothed second-layer feature map, a second output feature with a shape of [256, 3, 3], and a smoothed third-layer feature map, a third output feature with a shape of [512, 3, 3] are obtained;
[0069] (4) Feature stitching:
[0070] Concatenate the first output feature with a shape of [256, 3, 3], the second output feature with a shape of [256, 3, 3], and the third output feature with a shape of [512, 3, 3] according to the channel dimension;
[0071] After concatenation, the final fusion feature is obtained, and its shape is [1024,3,3], where 1024 is the number of channels and [3,3] represents a feature block with a height and width of 3;
[0072] (5) Convolutional layer conversion:
[0073] A convolution layer is added after the concatenated feature map. The number of input channels of the convolution layer is 1024 and the number of output channels is set to 96. This is used to reduce the number of channels from 1024 to 96, further extract and integrate features, and reduce the amount of computation and memory consumption.
[0074] Through the convolution operation, a new feature map is obtained, whose shape is [96,3,3]. This feature map contains rich multi-scale and multi-channel feature information and has high computational efficiency, which can be used for subsequent target tracking tasks.
[0075] In this embodiment, through the above five steps, multi-scale feature aggregation of the first layer, the second layer and the third layer feature maps is achieved, and fusion features containing rich information are obtained.
[0076] like Figure 4 As shown, in some embodiments, the information of the depth feature with the shape of [64,6,6] is reorganized, the feature space is divided and spliced along the channel dimension to obtain the first output feature, which specifically includes the following three steps:
[0077] (1) Feature space division:
[0078] The single [6,6] feature space in the depth feature map is evenly divided into 4 smaller feature blocks, each of which is [3,3]. In this way, the original [6,6] feature space is divided into four [3,3] feature blocks: upper left, upper right, lower left, and lower right.
[0079] (2) Feature block splicing:
[0080] According to the grouping order during division (such as upper left, upper right, lower left, and lower right), these four [3,3] feature blocks are regarded as extensions in the channel dimension and are spliced along the channel dimension.
[0081] During the splicing process, the number of channels of each feature block remains unchanged (i.e. 64), but through the splicing operation, the total number of channels increases to four times the original number, that is, the original deep feature has 64 channels, and the number of channels of the spliced feature map will increase to 64 * 4 = 256;
[0082] (3) Forming the first output feature:
[0083] After the above splicing operation, the original single [6,6] feature space is converted into a new feature map containing four [3,3] feature blocks, and the number of channels of the new feature map is increased to 256, realizing the transformation from a single [6,6] feature to four [3,3] features while maintaining the richness and depth of the features.
[0084] In this embodiment, through the above three information reorganization steps, the spatial information of the original feature map can be effectively utilized and converted into channel information, thereby reducing the spatial resolution of the feature map while retaining more original detail information, which has significant advantages when processing small targets and complex scenes.
[0085] In some embodiments, a visualization tool is used to perform a visualization operation on the three feature maps;
[0086] Among them, the first layer feature map is a shallow feature map, which mainly focuses on the relationship between the target and the whole image background; the first layer feature map receives richer original image information, so it contains more global information and background information, which helps the algorithm distinguish the difference between the target and the surrounding environment during the tracking process;
[0087] The second layer of feature maps is the middle layer of feature maps, which gradually focuses on the target itself, focusing on the target's contour and key shape information; compared with the first layer of feature maps, it pays more attention to the local details and key shape information of the target, such as the edge and corner points of the target;
[0088] The third-layer feature map is more focused on modeling the target itself. As the deepest feature map, the third-layer feature map has highly abstracted the feature representation of the target, almost completely focusing on the target itself, and providing rich internal and semantic information of the target. High-level semantic features are critical for achieving robust target tracking, especially in complex situations such as scale changes and occlusions.
[0089] In this embodiment, through the visual analysis of the three-layer feature map, it can be clearly seen that each layer of the feature map focuses on different information levels, which demonstrates the effectiveness of the multi-scale hierarchical feature aggregation module.
[0090] like Figure 5 As shown, in some embodiments, the similarity calculation using deep separable correlation filtering and point-by-point correlation filtering includes:
[0091] The deep separable correlation filter calculates the similarity between the template and the search area within the spatial context, effectively capturing the spatial location information of the target and generating a classification similarity score. This process not only reduces the amount of calculation, but also improves the model's classification accuracy for the target location, helping to accurately lock the target in complex scenes.
[0092] Point-by-point correlation filtering explores the spatial similarity between the template and the search area in the channel dimension and generates a regression similarity score. The point-by-point calculation method enhances the model's coupling ability to local features, especially in small target tracking tasks, which can significantly improve the feature discrimination and ensure accurate regression of the target coordinate frame.
[0093] In this embodiment, by integrating these two filtering technologies, the classification similarity score provides a reliable basis for target position classification, while the regression similarity score ensures the accuracy of the target coordinate frame, making the target tracking algorithm more robust when dealing with fast movement, scale changes, and occlusion.
[0094] In addition, the application of deep separable correlation filtering and point-by-point correlation filtering further reduces the number of model parameters, reduces computational complexity, and meets the resource-constrained requirements of embedded devices.
[0095] In some embodiments, the information calculation of the target center position classification branch and the target coordinate frame regression branch based on the similarity calculation result includes:
[0096] The classification similarity score and the regression similarity score are spliced in a channel splicing manner; by integrating similarity information of two different dimensions, a richer and more comprehensive feature representation is formed.
[0097] The concatenated features are mapped to the target center position classification branch and the target coordinate frame regression branch.
[0098] In the target center position classification branch, the features are processed to generate a confidence map. Each point on the map represents the similarity between the template and the search area at that position. By finding the coordinates of the highest point on the confidence map, the center position of the target in the current frame can be accurately determined.
[0099] In the target coordinate frame regression branch, the target's precise coordinate frame is further regressed based on the center position coordinate information provided by the classification branch. This process uses the rich feature information obtained by similarity calculation to achieve accurate estimation of the target size and position, thereby ensuring the accuracy of the tracking result.
[0100] In this embodiment, the similarity calculation result is converted into classification information of the target center position and accurate coordinate frame information, which improves the real-time performance and accuracy of target tracking.
[0101] In some embodiments, after the information calculation of the target center position classification branch and the target coordinate frame regression branch is completed, the following steps are included:
[0102] According to the target coordinate frame calculated in the current frame, the search area is reconstructed proportionally to complete the update of the search area; this ensures that the search area can closely follow the target movement and continue to capture the target's effective features in subsequent frames.
[0103] After completing the update of the search area, the algorithm will automatically enter the target localization process for the next frame of the image; this process will repeat the aforementioned steps of deep feature extraction, similarity calculation, target center position classification and target coordinate frame regression to ensure that the target position can be accurately located in each frame of the image.
[0104] The entire continuous tracking process continues until the video or image sequence is completely processed, achieving continuous tracking of the target.
[0105] This embodiment achieves stable and accurate tracking of targets in video or image sequences by dynamically adjusting the search area and continuously executing the target positioning process, thereby improving the real-time and continuity of target tracking.
[0106] A second aspect of the present application provides a lightweight embedded deployment method, comprising:
[0107] Convert the trained twin network model to an .onnx model in ONNX format;
[0108] Then, use the RKNN Toolkit provided by the Rockchip platform to further convert the .onnx model into an RKNN model;
[0109] Transfer the converted RKNN model to the RK3588M device;
[0110] Call the NPU interface to load the twin network model;
[0111] Load the logical processing steps onto the CPU for execution;
[0112] Load the image processing steps to run on the GPU.
[0113] Through the above steps, the trained target tracking model is efficiently integrated into embedded devices such as RK3588M, realizing real-time target tracking in resource-constrained environments. This not only reduces the computational complexity and resource consumption of the model, but also improves the operating efficiency and stability of the model on embedded devices.
[0114] like Figure 6 As shown, the third aspect of the present application provides an embedded device, including:
[0115] A first NPU module, used for running the template feature backbone network;
[0116] A second NPU module is used to run the search area feature backbone network;
[0117] A third NPU module is used to calculate the similarity between the target feature map and the search area feature map by using a deep separable correlation filter and a point-by-point correlation filter;
[0118] The CPU module is used to read the image and transmit it to the corresponding NPU module; it is also used to perform information calculation of the target center position classification branch and the target coordinate frame regression branch based on the similarity calculation result, and accurately determine the position of the target in the current frame;
[0119] GPU module, used for image pre-processing and post-processing.
[0120] By designing a multi-processor architecture and an efficient task allocation strategy, real-time target tracking in a resource-constrained environment is achieved. The platform not only improves the accuracy and robustness of target tracking, but also meets the comprehensive requirements for real-time, lightweight and high performance.
[0121] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0122] In the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish descriptions and cannot be understood as indicating or suggesting relative importance. It should also be understood that although the terms "first", "second", etc. are used to describe various elements in some embodiments of the present application in the text, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first table can be named as the second table, and similarly, the second table can be named as the first table without departing from the scope of the various described embodiments. The first table and the second table are both tables, but they are not the same table.
[0123] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0124] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A real-time target tracking method based on Siamese network, It is characterized in that include: Read the first frame of the video or image sequence and determine the tracking target frame; The target frame of the first frame image is input as a template into the template feature backbone network to extract features and obtain a target feature map; Read the next frame of image, build the search area in proportion according to the target frame in the previous frame, extract features of the search area using the search area feature backbone network, and obtain the search area feature map; Using deep separable correlation filtering and point-by-point correlation filtering to calculate the similarity between the target feature map and the search area feature map; Based on the similarity calculation results, the information calculation of the target center position classification branch and the target coordinate frame regression branch is performed to determine the position of the target in the current frame; The similarity calculation using deep separable correlation filtering and point-by-point correlation filtering includes: The deep separable correlation filter calculates the similarity between the template and the search area within the spatial context to obtain a classification similarity score; The point-by-point correlation filtering calculation template and the search area perform spatial similarity calculation in the channel dimension to obtain a regression similarity score; The information calculation of the target center position classification branch and the target coordinate frame regression branch based on the similarity calculation result includes: The classification similarity score and the regression similarity score are spliced by channel splicing; Map the concatenated features to the target center position classification branch and the target coordinate frame regression branch; The highest point coordinates are found on the confidence map of the target center position classification branch, and then the precise coordinate frame of the target is found and determined based on the coordinate information in the target coordinate frame regression branch.
2. A real-time target tracking method based on a twin network according to claim 1, It is characterized in that MobileNet V3-Small is used as the template feature backbone network and the search area feature backbone network; In the feature extraction process, three feature extraction layers are used to extract features respectively, and the output of each feature extraction layer is saved to obtain the first layer feature map, the second layer feature map and the third layer feature map; A multi-scale feature aggregation method is used to concatenate the three layers of feature maps to enhance the robustness of feature extraction.
3. A real-time target tracking method based on a twin network according to claim 2, It is characterized in that The multi-scale feature aggregation method is used to splice the three layers of feature maps, including: Read the first layer feature map and perform two convolution operations to obtain preliminary deep features; Reorganize the information of the preliminary deep features, divide the feature space and splice it along the channel dimension to obtain the first output feature; Read the second and third layer feature maps, perform convolution operations to smooth the features and obtain the second output features and the third output features respectively; The output features of the three layers are concatenated according to the channel dimension to obtain the fusion features; After the fusion features, a convolution layer is added to reduce the number of channels through the convolution operation, extract and integrate the features, and obtain the final feature map for subsequent target tracking tasks.
4. A real-time target tracking method based on a twin network according to claim 3, It is characterized in that The information of the preliminary depth features is reorganized, the feature space is divided and spliced along the channel dimension to obtain the first output feature, including: The single feature space in the deep feature map is evenly divided into n smaller feature blocks; according to the division order, these smaller feature blocks are spliced in the channel dimension to form the first output feature with a channel number that is n times the number of original feature channels.
5. A real-time target tracking method based on a twin network according to claim 2, It is characterized in that Use visualization tools to visualize the three feature maps; Among them, the first layer of feature map is a shallow feature map, which mainly focuses on the relationship between the target and the background of the whole image; The second layer of feature maps is the middle layer feature map, which gradually focuses on the target itself, paying attention to the target's contour and key shape information; The third layer of feature maps focuses more on modeling the target itself.
6. A real-time target tracking method based on a twin network according to claim 1, It is characterized in that After completing the information calculation of the target center position classification branch and the target coordinate frame regression branch, it includes: According to the target coordinate frame calculated in the current frame, the search area is reconstructed in proportion, the search area is updated, and the target positioning of the next frame image is repeated until the video or image sequence tracking is completed, thereby achieving continuous tracking of the target.
7. A lightweight embedded deployment method for the target tracking method according to any one of claims 1 to 6, It is characterized in that include: Convert the twin network described in any one of claims 1 to 6 into an .onnx model in ONNX format; Then, use the RKNN Toolkit provided by the Rockchip platform to further convert the .onnx model into an RKNN model; Transfer the converted RKNN model to the RK3588M device; Call the NPU interface to load the twin network; The logic processing step, which is based on the similarity calculation result, performs information calculation of the target center position classification branch and the target coordinate frame regression branch to determine the position of the target in the current frame, is loaded into the CPU for execution; Load the image processing steps to run on the GPU.
8. An embedded device using the target tracking method according to any one of claims 1 to 6, It is characterized in that include: A first NPU module, used for running the template feature backbone network; A second NPU module is used to run the search area feature backbone network; A third NPU module is used to calculate the similarity between the target feature map and the search area feature map by using a deep separable correlation filter and a point-by-point correlation filter; The deep separable correlation filter calculates the similarity between the template and the search area within the spatial context to obtain a classification similarity score; the point-by-point correlation filter calculates the spatial similarity between the template and the search area in the channel dimension to obtain a regression similarity score; the point-by-point correlation filter calculates the spatial similarity between the template and the search area in the channel dimension to obtain a regression similarity score; CPU module, used to read the image and transfer it to the corresponding NPU module; It is also used to calculate the information of the target center position classification branch and the target coordinate frame regression branch based on the similarity calculation results, and accurately determine the position of the target in the current frame; The classification similarity score and the regression similarity score are spliced by channel splicing; Mapping the spliced features to the target center position classification branch and the target coordinate frame regression branch; finding the highest point coordinates on the confidence map of the target center position classification branch, and then finding and determining the precise coordinate frame of the target according to the coordinate information in the target coordinate frame regression branch; GPU module, used for image pre-processing and post-processing.