A twin network target tracking system and method

Abstract

The present invention discloses a twin network target tracking system and method in the field of computer vision target tracking technology, including: obtaining an input feature map; performing convolution splitting based on the input feature map to obtain multiple feature sub-maps; performing convolution fusion based on multiple feature sub-maps to obtain a branch feature map; performing feature fusion based on the branch feature map to obtain an output vector; performing dimensionality upscaling processing based on the output vector to obtain an up-dimensional feature map; performing classification branching based on the up-dimensional feature map to obtain a classification map, a regression map, and a centrality map. The present invention can effectively solve the problems of weak feature extraction capability of the backbone network, shallow deep semantic information, and complex classification and regression methods, and proposes a new end-to-end deep model, which effectively improves the tracking performance of the solution.

Description

Technical Field

[0001] The present invention relates to a twin network target tracking system and method, and belongs to the technical field of computer vision target tracking. Background Art

[0002] Object tracking is an important branch of computer vision and has broad application prospects in video surveillance, autonomous driving, human-computer interaction and other fields. Object tracking algorithms automatically estimate the position of the object in each subsequent frame by giving the object position information in the first frame of a given video sequence. However, in complex real-world scenarios, the tracking process is affected by factors such as similar objects, illumination changes, scale changes, occlusion deformation, and low resolution. Therefore, designing a tracking algorithm that runs efficiently in real-world scenarios is a difficult task.

[0003] In recent years, as deep learning technology has become increasingly mature, it has been widely used in many fields. In the field of tracking, the twin network tracking algorithm based on deep learning has gradually become popular. It uses the end-to-end twin network structure to transform the tracking problem into a similarity learning problem. Based on the above idea, Bertinetto et al. proposed the fully convolutional twin network tracking algorithm (SiamFC), which has achieved a better balance between speed and accuracy compared with other types of algorithms. Since then, a series of excellent twin network tracking algorithms have been derived based on SiamFC. SiamRPN regards the tracking problem as a two-stage problem, and uses bounding box regression instead of multi-scale search to improve accuracy and efficiency. DaSiamRPN introduces different semantic distractors based on SiamRPN, enhances the discrimination ability of the classifier, and thus improves the tracking performance. SiamRPN++ uses ResNet instead of AlexNet to deepen the network, obtain richer feature semantic information, and the algorithm can also maintain good robustness in complex real-world scenarios. SiamMask adds a mask branch based on SiamFC to obtain a more accurate bounding box, further improving the tracking accuracy. DCSiam uses deformable convolution to enhance the feature extraction capability of the backbone network, and adopts a high-confidence update strategy to effectively curb the degradation of template information. Although the above SiamFC-based twin network tracking algorithms have achieved good results, they still have the following problems to be solved: First, the backbone network feature extraction capabilities of these tracking algorithms are weak. Second, they only focus on mining local semantic information and do not pay attention to global semantic information. Tracking drift is prone to occur when the target disappears and reappears. Finally, most algorithms use predefined anchor boxes for classification and regression, which has high algorithm complexity and low running speed. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a twin network target tracking system and method to effectively improve the tracking performance of the scheme.

[0005] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0006] In a first aspect, the present invention provides a twin network target tracking method, comprising:

[0007] Get the input feature map;

[0008] Perform convolution splitting based on the input feature map to obtain multiple feature sub-maps;

[0009] Perform convolution fusion based on multiple feature sub-graphs to obtain a branch feature graph;

[0010] Perform feature fusion based on the branch feature graph to obtain the output vector;

[0011] Perform dimensionality increase processing based on the output vector to obtain an increased dimensional feature map;

[0012] Based on the dimension-increased feature map, classification branches are performed to obtain classification maps, regression maps, and centrality maps.

[0013] Furthermore, convolution splitting is performed based on the input feature map to obtain multiple feature sub-maps, including:

[0014] A 1×1 convolution is used to split the input feature map into multiple feature sub-maps.

[0015] Furthermore, convolution fusion is performed based on multiple feature sub-graphs to obtain branch feature graphs, including:

[0016] The sum of the convolution outputs of each feature subgraph and the previous feature subgraph is used as the input of the subgraph for 3×3 convolution;

[0017] Concatenate multiple feature sub-graphs after 3×3 convolution;

[0018] Based on the concatenated feature sub-graphs, 1×1 convolution fusion is performed to obtain the feature graphs of the template branch and the search branch.

[0019] Furthermore, the template branch uses the first frame of the video sequence as the template region The search branch uses each subsequent frame as the search area The template area and the search area are subjected to the feature extraction module to obtain feature maps of sizes: and Where C is the number of channels, Among them, H x0 and W x0 is the length and width of the search area, Hz0 and W z0 are the length and width of the template area.

[0020] Furthermore, feature fusion is performed based on the branch feature graph to obtain an output vector, including:

[0021] Perform 1×1 convolution on the feature maps of the template branch and the search branch respectively;

[0022] Flatten the feature map after 1×1 convolution into a feature vector to obtain the template vector and search vector;

[0023] Feature fusion is performed based on the template vector and the search vector to obtain the output vector.

[0024] Furthermore, classification branches are performed based on the dimension-upgraded feature map, including:

[0025] Predict the category of each pixel on the dimensional feature map to obtain a classification map;

[0026] The regression branch calculates the distance from the pixel to the bounding box to obtain the regression map;

[0027] The distance from the pixel to the target center is calculated through the center branch to obtain the center map.

[0028] Furthermore, the number of the characteristic subgraphs is four.

[0029] In a second aspect, the present invention provides a twin network target tracking system, comprising:

[0030] Acquisition module: used to obtain input feature maps;

[0031] Convolution splitting module: used to perform convolution splitting based on the input feature map to obtain multiple feature sub-maps;

[0032] Convolution fusion module: used to perform convolution fusion based on multiple feature sub-graphs to obtain branch feature graphs;

[0033] Feature fusion module: used to perform feature fusion based on branch feature graphs to obtain output vectors;

[0034] Feature dimension-upgrading module: used to perform dimension-upgrading processing based on the output vector to obtain a dimension-upgraded feature map;

[0035] Classification branch module: used to perform classification branching based on the dimensionality-increased feature map to obtain classification maps, regression maps and centrality maps.

[0036] In a third aspect, a twin network target tracking device includes a processor and a storage medium;

[0037] The storage medium is used to store instructions;

[0038] The processor is used to operate according to the instructions to execute the steps of any of the above methods.

[0039] In a fourth aspect, a computer-readable storage medium stores a computer program, which implements the steps of any of the above methods when executed by a processor.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. The present invention discloses a twin network target tracking system and method for tracking targets, which can effectively solve the problems of weak feature extraction capability of the backbone network, shallow deep semantic information, and complex classification and regression methods. It proposes a new end-to-end deep model, which effectively improves the tracking performance of the solution.

[0042] Second, the present invention uses the improved Res2Net network as the feature extraction network, and enhances the feature extraction capability of the backbone network by expanding the receptive field of each network layer. Secondly, the EDE structure is proposed as a new Transformer main framework, that is, adding codecs from left to right in the loop layer, so that the algorithm can adaptively establish feature associations to obtain global semantic information and deeply mine target semantic information. Finally, pixel points are used as training samples, and the centrality branch is introduced to suppress low-quality bounding boxes, reducing the high complexity caused by too many operation parameters, thereby simplifying the complexity of classification and regression tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is an algorithm framework diagram provided by Embodiment 1 of the present invention;

[0044] Figure 2 This is the improved Res2Net network diagram provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0046] Embodiment 1:

[0047] A twin network target tracking method includes the following steps:

[0048] Step S1, the present invention improves Res2Net and uses it as a feature extraction network. It uses a new convolution method to replace the convolution method in ResNet. The feature map obtained by ResNet after 1×1 convolution is directly subjected to 3×3 convolution, while Res2Net first uses 1×1 convolution to split the input feature map into 4 feature sub-maps, and then performs 3×3 convolution on each feature sub-map except the first feature sub-map. The improved Res2Net of this method performs a 3×3 convolution on each feature sub-map, and uses the sum of the convolution output of each feature sub-map with the previous feature sub-map as the input of the 3×3 convolution corresponding to the sub-map. Finally, the four channel feature sub-maps are spliced ​​and then subjected to 1×1 convolution to fuse the information in different feature sub-maps. Among them, the feature extraction template includes a template branch and a search branch: the template branch uses the first frame of the video sequence as the template area The video is composed of a frame-by-frame video sequence, i.e., pictures. Feature extraction is performed on the video sequence to obtain a feature map. The search branch uses each subsequent frame as the search area. The template area and the search area are subjected to the feature extraction module to obtain feature maps of sizes: and Where C is the number of channels, Among them, H x0 and W x0 is the length and width of the search area, H z0 and W z0 are the length and width of the template area.

[0049] Step S2, the Transformer feature fusion module consists of two parts: a feature fusion loop layer and a decoder. The feature fusion loop layer structure is an EDE structure from left to right. First, the encoder in the feature fusion loop layer first uses a multi-head self-attention module to focus on the semantic information of the target. Then the decoder simultaneously receives the feature maps of this branch and another branch, and then uses a multi-head mutual attention module to fuse the semantic information from the two branches, fully capturing the global semantic information and focusing on the key semantic information of the object. Finally, the encoder is added on the basis of ED to mine the deep semantic information of the target. The feature fusion loop layer loops N times, and the output after N loops is used as the input of the decoder to fuse the information of the template branch and the search branch. The Transformer feature fusion module processes the feature maps f of the template branch and the search branch. z , f x Perform feature fusion. First, f z and f x Perform 1×1 convolution to reduce the channel dimension and the number of parameters. The dimension of the feature map after 1×1 convolution is d is the dimension of the feature map. Then the feature map is flattened into a feature vector to obtain the template vector and search vector Finally, fz and f x Perform feature fusion to obtain the output vector

[0050] Step S3, the target estimation module first transforms the feature vector Upgrading to feature maps The classification branch directly predicts the category of each pixel on T to obtain a classification map The regression branch calculates the distance from the pixel to the bounding box to obtain the regression map The center branch calculates the distance from the pixel to the target center to obtain the center map Each point (i, j) on the feature map can be mapped to the input search area point (x, y), (x0, y0) and (x1, y1) are the upper left corner and lower right corner of the real bounding box. For the classification map A cls Each point (i, j, :) contains a 2D vector representing the confidence of the foreground and background of the corresponding search area. reg Each point (i,j,:) contains a 4D vector m (i,j) =(l * ,t * ,r * ,b * ), where l*, t*, r*, b* represent the distances from the search area point to the left, top, right, and bottom boundaries of the bounding box, i.e., left, top, right, and bottom, representing the distances from the corresponding points in the input search area to the four edges of the predicted bounding box, and are defined as follows:

[0051]

[0052]

[0053] Among them, x, y are the coordinates of the pixel points in the search area, (x0, y0) are the coordinates of the upper left corner, (x1, y1) are the coordinates of the lower right corner, and the indicator function is defined as follows:

[0054]

[0055] Among them, m k (i, j are 4-dimensional vectors, superscript k = 0 is l*, k = 1 is t*, k = 2 is r*, k = 3 is b*. Since pixels far from the center of the target position tend to produce low-quality predicted bounding boxes, which will affect the tracking performance of the algorithm, a centrality branch is added in parallel to the classification branch to remove outliers. The centrality branch finally generates a centrality feature map A cen , A cen Each point (i, j, :) contains a 1D vector C(i, j), which represents the distance between the corresponding point in the search area and the target center:

[0056]

[0057] If the point (x, y) falls in the background area, the value of C(i, j) is 0.

[0058] The present invention discloses a twin network target tracking system method to track a target. The invention uses an improved Res2Net network as a feature extraction network, and enhances the feature extraction capability of the backbone network by expanding the receptive field of each network layer. Secondly, it is proposed to use the EDE structure as a new Transformer main framework, that is, to add codecs from left to right in the loop layer, so that the algorithm can adaptively establish feature associations to obtain global semantic information and deeply mine target semantic information. Finally, pixel points are used as training samples, and the centrality branch is introduced to suppress low-quality bounding boxes, which reduces the high complexity caused by too many operation parameters, thereby simplifying the complexity of classification and regression tasks.

[0059] Embodiment 2:

[0060] A twin network target tracking system, comprising:

[0061] Acquisition module: used to obtain input feature maps;

[0062] Convolution splitting module: used to perform convolution splitting based on the input feature map to obtain multiple feature sub-maps;

[0063] Convolution fusion module: used to perform convolution fusion based on multiple feature sub-graphs to obtain branch feature graphs;

[0064] Feature fusion module: used to perform feature fusion based on branch feature graphs to obtain output vectors;

[0065] Feature dimension-upgrading module: used to perform dimension-upgrading processing based on the output vector to obtain a dimension-upgraded feature map;

[0066] Classification branch module: used to perform classification branching based on the dimensionality-increased feature map to obtain classification maps, regression maps and centrality maps.

[0067] Embodiment three:

[0068] The embodiment of the present invention also provides a twin network target tracking device, including a processor and a storage medium;

[0069] The storage medium is used to store instructions;

[0070] The processor is used to operate according to the instructions to perform the steps of the following method:

[0071] Get the input feature map;

[0072] Perform convolution splitting based on the input feature map to obtain multiple feature sub-maps;

[0073] Perform convolution fusion based on multiple feature sub-graphs to obtain a branch feature graph;

[0074] Perform feature fusion based on the branch feature graph to obtain the output vector;

[0075] Perform dimensionality increase processing based on the output vector to obtain an increased dimensional feature map;

[0076] Based on the dimension-increased feature map, classification branches are performed to obtain classification maps, regression maps, and centrality maps.

[0077] Embodiment 4:

[0078] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the following method are implemented:

[0079] Get the input feature map;

[0080] Perform convolution splitting based on the input feature map to obtain multiple feature sub-maps;

[0081] Perform convolution fusion based on multiple feature sub-graphs to obtain a branch feature graph;

[0082] Perform feature fusion based on the branch feature graph to obtain the output vector;

[0083] Perform dimensionality increase processing based on the output vector to obtain an increased dimensional feature map;

[0084] Based on the dimension-increased feature map, classification branches are performed to obtain classification maps, regression maps, and centrality maps.

[0085] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0086] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0087] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0089] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A twin network target tracking method, characterized in that: include: Get the input feature map; Use 1×1 convolution to split the input feature map into multiple feature sub-maps; The sum of the convolution outputs of each feature subgraph and the previous feature subgraph is used as the input of the subgraph for 3×3 convolution; Concatenate multiple feature sub-graphs after 3×3 convolution; Based on the concatenated feature sub-graphs, 1×1 convolution is used to fuse the feature graphs of the template branch and the search branch. Perform 1×1 convolution on the feature maps of the template branch and the search branch respectively; Flatten the feature map after 1×1 convolution into a feature vector to obtain the template vector and search vector; Perform feature fusion based on the template vector and the search vector to obtain an output vector; Perform dimensionality increase processing based on the output vector to obtain an increased dimensional feature map; Based on the dimension-increased feature graph, classification branches are performed to obtain classification graphs, regression graphs, and centrality graphs; The template branch uses the first frame of the video sequence as the template area , the search branch uses each subsequent frame as the search area After the template area and the search area pass through the feature extraction module, the feature map sizes are and ,in, is the number of channels, , , , ,in, and are the length and width of the search area, and are the length and width of the template area; The feature fusion based on the template vector and the search vector to obtain the output vector includes: A Transformer feature fusion module is set, and the Transformer feature fusion module consists of a feature fusion loop layer and a decoder; the feature fusion loop layer structure is an EDE structure from left to right, the first encoder in the feature fusion loop layer focuses on the semantic information of the template vector and the search vector through a multi-head self-attention module, the decoder in the feature fusion loop layer fuses the semantic information from the two branches through a multi-head mutual attention module, fully captures the global semantic information and focuses on the key semantic information, and the second encoder in the feature fusion loop layer mines the deep semantic information of the target; the feature fusion loop layer loops N times, and the output after N loops is used as the input of the decoder, and fused to obtain the output vector.

2. The twin network target tracking method according to claim 1 is characterized in that: Classification branches based on the dimension-upgraded feature map include: Predict the category of each pixel on the dimensional feature map to obtain a classification map; The regression branch calculates the distance from the pixel to the bounding box to obtain the regression map; The distance from the pixel to the target center is calculated through the center branch to obtain the center map.

3. The twin network target tracking method according to claim 1 is characterized in that: The number of the characteristic subgraphs is four.

4. A twin network target tracking system, characterized in that: The system is configured to perform the steps of the method according to any one of claims 1 to 3, and the system comprises: Acquisition module: used to obtain input feature maps; Convolution splitting module: used to perform convolution splitting based on the input feature map to obtain multiple feature sub-maps; Convolution fusion module: used to perform convolution fusion based on multiple feature sub-graphs to obtain branch feature graphs; Feature fusion module: used to perform feature fusion based on branch feature graphs to obtain output vectors; Feature dimension-upgrading module: used to perform dimension-upgrading processing based on the output vector to obtain a dimension-upgraded feature map; Classification branch module: used to perform classification branching based on the dimensionality-increased feature map to obtain classification maps, regression maps and centrality maps.

5. A twin network target tracking device, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 3 are implemented.

Images