A twin vision tracking method based on multi-parallel interactive transformation network
Through multiple parallel interactive transformation network models and interactive attention modules, the deep information interaction between the template and the search area is enhanced, which solves the performance degradation problem of existing twin visual trackers in complex environments and achieves higher tracking accuracy and robustness.
Patent Information
- Application Number
- CN202510118386.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-24
- Publication Date
- 2025-09-23
- Estimated Expiration
- 2045-01-24
AI Technical Summary
The performance of existing twin visual trackers degrades under factors such as target appearance changes, occlusion, and background interference, affecting their robustness and reliability in practical applications. The lack of deep interaction between the template and the search area reduces the ability to distinguish.
A multi-parallel interactive transformation network model is adopted, and the deep information interaction between the template and the search area is enhanced through multi-parallel interactive attention modules. The multi-head self-attention and layer normalization network are used to improve the feature representation. The L1 loss and GIoU loss are combined for training and verification to enhance the distinguishing ability of the tracker.
It significantly improves the accuracy and robustness of visual object tracking, enhances the deep information interaction between the template and the search area, improves the tracker's discriminative ability, and maintains efficient computation.
Smart Images
Figure CN120088325B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a twin vision tracking method based on a multi-parallel interactive transformation network. Background Art
[0002] Visual object tracking is one of the most critical components in computer vision. With the continuous advancement of artificial intelligence and electronic devices, visual tracking plays a vital role in real-world scenarios such as precision guidance, search and rescue missions, and intelligent traffic monitoring. However, factors such as target appearance changes, occlusions, background interference, and dramatic target motion can degrade tracker performance, affecting its robustness and reliability in practical applications.
[0003] To address these issues, researchers have developed twin trackers, which effectively address these shortcomings by demonstrating outstanding performance through powerful convolutional networks. However, existing twin trackers use two independent, weight-sharing streams to describe the feature representations of the template and search area, without any interaction between the two streams. Consequently, these trackers can only achieve final information association through shallow cross-correlation or correlation filters, ignoring the deep interaction between the template and search area, potentially reducing the tracker's discriminative capabilities.
[0004] Currently, there is a lack of a good visual tracking method that can effectively improve the distinguishing ability of the tracker.
[0005] The information disclosed in this background technology section is only intended to enhance understanding of the overall background of the invention and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the Invention
[0006] The technical problem to be solved by the present invention is to overcome the above-mentioned defects and provide a twin visual tracking method based on a multi-parallel interactive transformation network.
[0007] In order to solve the above technical problems, the technical solution provided by the present invention is:
[0008] A twin visual tracking method based on a multi-parallel interactive transformation network, comprising: building a multi-parallel interactive transformation network model, which can track visual targets; training the multi-parallel interactive transformation network model based on a preconfigured data training set; verifying the trained multi-parallel interactive transformation network model based on a preset verification method; deploying the verified multi-parallel interactive transformation network model in a pre-set multi-parallel interactive attention module to enhance the deep information interaction between the template and the search area, thereby effectively responding to changes in the target appearance.
[0009] Optionally, the multi-parallel interactive transformation network model includes at least a framework part and a core part, the framework part includes at least a template area, a transmission branch and a search area, the core part includes at least several multi-parallel interactive attention modules connected in series, and each of the multi-parallel interactive attention modules includes at least an intra-region self-attention module, a template interaction block, a search interaction block and a transmission block.
[0010] Optionally, the template region and the search region share the same network weight.
[0011] Optionally, the calculation process of the self-attention module in the region is expressed by the following formula:
[0012]
[0013] in, and Respectively represent the template area and search area features after multi-head self-attention, and Represent the features of the template area and search area after layer normalization and feedforward network, is a template token, is the search token, f intra is the intra-region self-attention block, LN and FFN represent layer normalization and feed-forward neural network transfer respectively, and MHSA is the standard multi-head self-attention function.
[0014] Optionally, the calculation process of MHSA is expressed by the following formula:
[0015]
[0016] Among them, [q intra , k intra , v intra ] represent the query vector, key vector and value vector respectively, dk represents the dimension of the key vector, [q intra , k intra , v intra ]=xU qkv and is the corresponding linear transformation network, U qkv represents the weight matrix, Denotes that the matrix belongs to the real field and has dimension C×3C.
[0017] Optionally, the multi-parallel interactive transformation network model is trained based on a preconfigured data training set, including: initializing all filters, parameters and weights with random numbers; randomly selecting a number of template samples and test samples from the configured training data set as current input; using the template samples as input to the multi-parallel interactive transformation network, and calculating the loss of the network output; updating the network parameters according to the calculated loss of the network output; and repeating the above steps within a preset period.
[0018] Optionally, the loss of computing the network output is expressed as follows:
[0019]
[0020] in, and λ iou are the weight coefficients of L1 loss and GIoU loss, B i and are the true bounding box and the predicted bounding box respectively.
[0021] Optionally, based on a preset verification method, the trained multi-parallel interactive transformation network model is verified, including: based on a pre-configured template area, using the trained multi-parallel interactive transformation network model, extracting the template area feature code; based on a pre-selected search area, using the trained multi-parallel interactive transformation network model, extracting the search area feature code to obtain a response map; obtaining the unknown corresponding to the maximum value in the corresponding map to obtain the target position estimated in the current frame, thereby using the trained multi-parallel interactive transformation network model to perform online target tracking.
[0022] Optionally, the preconfigured template area includes: the first frame in a given sequence of images, and an initial rectangular frame of the target, with the target as the center, cutting out an image block, and adjusting the image size to serve as the template area.
[0023] Optionally, the pre-selected search area includes: in the second frame and each subsequent frame, selecting a search area with the position of the target in the previous frame as the center.
[0024] This paper significantly improves tracking accuracy and robustness by introducing multiple parallel interactive attention modules. While maintaining efficient computation, this method enhances the deep information interaction between the template and the search area, effectively improving the tracker's discriminative capabilities. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 This is a flowchart of a twin visual tracking method based on a multi-parallel interactive transformation network provided by an embodiment of the present invention;
[0026] Figure 2 This is a diagram of the overall network structure provided by an embodiment of the present invention;
[0027] Figure 3 This is a structural diagram of the self-attention module within the region provided by an embodiment of the present invention;
[0028] Figure 4 is a test result diagram provided by an embodiment of the present invention;
[0029] Figure 5 This is an example flow chart provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0030] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices consistent with certain aspects of the present disclosure, as detailed in the appended claims.
[0031] As mentioned above, existing Siamese trackers use two independent streams with shared weights to describe the feature representations of the template and search region, but there is no interaction between the two streams. As a result, these trackers can only achieve the final information association through shallow cross-correlation or correlation filters, which ignores the deep interaction between the template and the search region and may reduce the tracker's discriminative ability.
[0032] To address this issue, the present invention proposes a twin visual tracking method based on a multi-parallel interactive transformation network to solve the above-mentioned problem. The present invention is implemented in the following way:
[0033] Please refer to the instruction manual Figure 1 , Figure 1 This is a flowchart of a twin visual tracking method based on a multi-parallel interactive transformation network provided by an embodiment of the present invention. Figure 1 As shown, in one embodiment of the present invention, the execution subject may be a controller, which may include the following steps:
[0034] S100: Building a multi-parallel interactive transformation network model, wherein the multi-parallel interactive transformation network model can track a visual target.
[0035] S200: Training the multi-parallel interactive transformation network model based on a preconfigured data training set.
[0036] S300: Based on a preset verification method, verify the trained multi-parallel interactive transformation network model.
[0037] S400. Deploy the verified multi-parallel interactive transformation network model in a pre-set multi-parallel interactive attention module to enhance the deep information interaction between the template and the search area, thereby effectively responding to changes in the target appearance.
[0038] The present invention significantly improves tracking accuracy and robustness by introducing multiple parallel interactive attention modules. While maintaining efficient computation, this method enhances the deep information interaction between the template and the search area, effectively improving the tracker's ability to distinguish.
[0039] In order to further explain the present invention clearly and completely, based on the above embodiment, the present invention also provides another preferred embodiment, which includes the following steps:
[0040] Step 1: Build a multi-parallel interactive transformation network model. The specific process is as follows:
[0041] like Figure 2 As shown in Figure 1, the framework consists of three parallel branches: template region, transfer branch, and search region. Each branch is specially designed to enhance tracking performance by promoting robust interaction and dynamic information integration. The template branch and the search branch share the same network weights. The core of the multi-parallel interactive transformation network framework consists of multiple (e.g., N m =8) parallel interactive attention modules are connected in series. Each multi-parallel interactive attention module in the network consists of multiple (e.g., 5) key components. For example, the self-attention block f in two regions intra , a template interaction block f TIB , a search interaction block f SIB , and a transport block f trans The input of the i-th multi-parallel interactive attention module is the output of the (i-1)-th module, i.e., the template token Search Token and transfer tokens Among them H t and W t Represent the height and width of the template image, where H s and W s Represent the height and width of the search image respectively, C represents the number of channels of the feature, N TT and D TT denote the number of transfer tokens and the dimension of each transfer token, respectively.
[0042] To describe the basic representation of template and search token, two intra-region self-attention blocks are initially adopted, such as Figure 2As shown in , these blocks have shared weights, where standard Vit is applied to capture the global context information within each region. The computation of the self-attention block within a region can be expressed as:
[0043]
[0044] in and Respectively represent the template area and search area features after multi-head self-attention, and Represent the features of the template area and search area after layer normalization and feedforward network, is a template token, is the search token, f intra is the self-attention block within the region, LN and FFN represent layer normalization and feedforward neural network transfer respectively, and MHSA is the standard multi-head self-attention function. It can be expressed as follows:
[0045]
[0046] Among them, [q intra , k intra , v intra ] represent the query vector, key vector and value vector respectively, dk represents the dimension of the key vector, [q intra , k intra , v intra ]=xU qkv and is the corresponding linear transformation network, U qkv represents the weight matrix, Denotes that the matrix belongs to the real field and has dimension C×3C.
[0047] Step 2: Use the labeled training set to perform end-to-end training on the multi-parallel interactive transformation network model. The specific process is as follows:
[0048] (1) In accordance with Figure 2 Construct a multi-parallel interactive transformation network model and initialize all filters, parameters and weights with random numbers.
[0049] (2) Randomly select N from all training sets train The template sample and the test sample are used as the input of the current mini-batch. The test sample is selected from one or more test sets in the GOT-10k, LaSOT, TNL2K, TrackingNet and other datasets. The target in the template sample is at the center of the image block, while the target in the test sample may be at any position. trainThe samples are used as input to the multi-parallel interactive transformation network, and the loss of the network output is calculated according to the following loss function, and then the network parameters are updated according to the loss:
[0050]
[0051] in and λ iou are the weight coefficients of L1 loss and GIoU loss, B i and are the true bounding box and the predicted bounding box respectively.
[0052] Repeat step (2) after each mini-batch until the pre-set number of training times N is reached batch .
[0053] Step 3: Use the network model trained in step 2 to perform online target tracking. The specific process is as follows:
[0054] Given a sequence of images I i The first frame I1 in (i=1, 2, ..., gate) and the initial rectangular frame B1 of the target are cropped with the target size N centered on the target. t The image block is times larger and the image size is adjusted to 128×128×3 as the template area, and the template feature coding is extracted using a multi-parallel interactive transformation network.
[0055] In the second and subsequent frames, a search region is selected, centered around the target location in the previous frame. A multi-parallel interactive transformation network is used to extract the search region's feature code. This is then combined with the template's feature code through a post-processing module to generate a response map. By finding the location corresponding to the maximum value in the response map, the estimated target location for the current frame is obtained.
[0056] Accordingly, the visual target tracking method based on a multi-parallel interactive transformation network described in the present invention fully utilizes the global information in the target area and the search area for offline training through the inventive steps (1) and (2), thereby improving the model's resolution of the target and the background, and significantly improving the accuracy of visual target tracking through the multi-parallel interactive attention module and the sparse update mechanism.
[0057] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0058] Additionally, the terms "system" and "network" are often used interchangeably. The term "and / or" is simply used to describe a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " generally indicates an "or" relationship between the related objects.
[0059] It should be understood that in the embodiments of the present invention, "B corresponding to A" means that B is associated with A and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B based solely on A; B can also be determined based on A and / or other information.
[0060] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0061] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0062] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.
[0063] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the objectives of the embodiments of the present invention.
[0064] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.
[0065] Through the description of the above embodiments, it will be clear to those skilled in the art that the present invention can be implemented in hardware, firmware, or a combination thereof. When implemented using software, the above functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media includes any medium that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a computer. By way of example and not limitation, computer-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer. In addition, any connection can appropriately become a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wherein disks usually reproduce data magnetically, while discs use lasers to reproduce data optically. The above combinations should also be included in the scope of protection of computer-readable media.
[0066] In short, the above description is only a preferred embodiment of the technical solution of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.
Claims
1. A twin visual tracking method based on multiple parallel interactive transformation networks, characterized in that: include: Building a multi-parallel interactive transformation network model, wherein the multi-parallel interactive transformation network model is capable of tracking a visual target; Training the multi-parallel interactive transformation network model based on a preconfigured data training set; Verifying the trained multi-parallel interactive transformation network model based on a preset verification method; The verified multi-parallel interactive transformation network model is deployed in the pre-set multi-parallel interactive attention module to enhance the deep information interaction between the template and the search area, thereby effectively responding to changes in the target appearance. The multi-parallel interactive transformation network model includes at least a framework part and a core part, wherein the framework part includes at least a template area, a transmission branch and a search area, and the core part includes at least a plurality of multi-parallel interactive attention modules in series, each of which includes at least a self-attention module in the area, a template interaction block, a search interaction block and a transmission block. The calculation process of the self-attention module in the region is expressed by the following formula: ； in, and Respectively represent the template area and search area features after multi-head self-attention, and Represent the features of the template area and search area after layer normalization and feedforward network, is a template token, , is the search token, , is the self-attention block within the region, LN and FFN represent layer normalization and feedforward neural network transfer respectively, MHSA is the standard multi-head self-attention function, The calculation process of MHSA is expressed by the following formula: ； in, Represent the query vector, key vector and value vector respectively, dk represents the dimension of the key vector, and is the corresponding linear transformation network, represents the weight matrix, Denotes that the matrix belongs to the real field and has dimension C×3C.
2. The twin visual tracking method based on multiple parallel interactive transformation networks according to claim 1 is characterized in that: The template region and the search region share the same network weights.
3. The twin visual tracking method based on multiple parallel interactive transformation networks according to claim 1 is characterized in that: Training the multi-parallel interactive transformation network model based on a preconfigured data training set includes: Initialize all filters, parameters and weights with random numbers; Randomly select several template samples and test samples from the preconfigured training data set as the current input; Using the template sample as input to a multi-parallel interactive transformation network, and calculating the loss of the network output; Update the network parameters according to the calculated loss of the network output; Repeat the above steps within the preset cycle.
4. The twin visual tracking method based on multiple parallel interactive transformation networks according to claim 3 is characterized in that: The loss of computing network output is expressed as follows: ； in, and They are The weight coefficient of loss and GIoU loss, and are the true bounding box and the predicted bounding box respectively.
5. The twin visual tracking method based on multiple parallel interactive transformation networks according to claim 1 is characterized in that: The trained multi-parallel interactive transformation network model is verified based on a preset verification method, including: Based on the pre-configured template area, the template area feature code is extracted using the trained multi-parallel interactive transformation network model; Based on the pre-selected search area, the trained multi-parallel interactive transformation network model is used to extract the search area feature code to obtain the response map; The unknown corresponding to the maximum value in the corresponding image is obtained to obtain the target position estimated in the current frame, so as to use the trained multi-parallel interactive transformation network model for online target tracking.
6. The twin visual tracking method based on multiple parallel interactive transformation networks according to claim 5 is characterized in that: The pre-configured template area includes: the first frame in a given sequence of images, and an initial rectangular frame of the target, with the target as the center, cutting out an image block, and adjusting the image size to serve as the template area.
7. The twin visual tracking method based on multiple parallel interactive transformation networks according to claim 6 is characterized in that: The pre-selected search area includes: in the second frame and each subsequent frame, selecting a search area with the position of the target in the previous frame as the center.