Target tracking method based on multi-frequency aggregation block feature fusion network

Abstract

The present invention proposes a target tracking method based on a multi-frequency aggregation block feature fusion network, the method comprising: extracting shallow features of an input image to obtain a shallow feature map; obtaining low-frequency global information through a low-frequency global branch; obtaining high-frequency local information through a high-frequency local branch; obtaining a multi-frequency aggregation attention feature map based on low-frequency global information and low-frequency global information; obtaining a feature map processed by a multi-frequency aggregation block based on the multi-frequency aggregation attention feature map; obtaining a feature map processed by fusion attention based on the feature map processed by the multi-frequency aggregation block; training the model to obtain a trained model; obtaining a classification result based on the trained model and tracking the target. The present invention efficiently extracts information of different frequencies through a multi-frequency aggregation block, and the multi-frequency aggregation block in the shallow layer of the feature fusion network improves the model's attention to low-frequency global features and high-frequency local features, achieving a balance between high and low-frequency information.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a target tracking method based on a multi-frequency aggregate block feature fusion network. Background Art

[0002] Object tracking is one of the core tasks studied in the field of computer vision. It uses the contextual information of the video to model the appearance and motion information of the target, thereby predicting the target motion state and marking the target position. It is widely used in many fields such as video surveillance and security, intelligent transportation, and human-computer interaction. Although target tracking technology has made certain progress, it also faces many challenges in practical applications, such as lighting changes, interference from similar targets, dynamic background changes, object occlusion, motion blur, etc. Therefore, the field of target tracking still has great development prospects.

[0003] In recent years, as Transformer shines in the field of natural language processing, some scholars have tried to introduce Transformer into the field of computer vision, and due to its powerful global modeling ability, Transformer-based trackers have emerged in the field of target tracking. Many researchers have set their sights on Transformer. Transformer has an excellent ability to capture global features (low frequency), which enables the model to consider all positions in the input sequence at the same time. However, due to its advantages in capturing low-frequency information, frequency deviation, and the existence of low-pass filter characteristics, Transformer still faces challenges in capturing high-frequency information. The tracking model based on Transformer can capture the dependencies of any position in the sequence through the self-attention mechanism. However, this global modeling ability also means that Transformer over-captures low-frequency information, resulting in a decrease in the proportion of high-frequency information, making the high- and low-frequency information unbalanced, and thus the ability to capture local edge and texture information is relatively weak. For tracking tasks, the lack of local edge information and texture information will cause the model to have an accuracy impact on the recognition of target instances in different categories. Summary of the invention

[0004] In view of the above situation, the main purpose of the present invention is to propose a target tracking method based on a multi-frequency aggregation block feature fusion network to solve the above technical problems.

[0005] The present invention proposes a target tracking method based on a multi-frequency aggregation block feature fusion network, the method comprising the following steps:

[0006] Step 1: Extract shallow features of the input image through a convolutional neural network to obtain a shallow feature map;

[0007] Step 2: Input the shallow feature map into the multi-frequency aggregation attention module in the multi-frequency aggregation module; in the low-frequency global branch of the multi-frequency aggregation attention module, perform linear transformation and self-attention operations on the shallow feature map in sequence to obtain low-frequency global information;

[0008] In the high-frequency local branch of the multi-frequency aggregation attention module, the shallow feature map is sequentially subjected to multi-window mapping, fusion attention operation, and splicing operation to obtain high-frequency local information;

[0009] The low-frequency global information is concatenated with the high-frequency local information to obtain high- and low-frequency feature maps, and then the high- and low-frequency feature maps are linearly transformed to obtain multi-frequency aggregated attention feature maps;

[0010] Input the multi-frequency aggregated attention feature map into the reverse residual multi-layer perceptron to obtain the feature map processed by the multi-frequency aggregation block;

[0011] Step 3, using the template image and the search area image as input images, repeating steps 1 and 2 to obtain a multi-frequency template feature map and a multi-frequency search area feature map, respectively;

[0012] Step 4: linearly transform the multi-frequency template feature map and the multi-frequency search area feature map to obtain the key vectors of the multi-frequency search area and the template area, the value vectors of the multi-frequency search area and the template area, and the query vector of the multi-frequency search area;

[0013] Performing fusion attention processing on the key vectors of the multi-frequency search area and the template area, the value vectors of the multi-frequency search area and the template area, and the query vector of the multi-frequency search area to obtain a feature map processed by fusion attention;

[0014] Step 5: Combine the feature graph processed by fusion attention and use a large-scale data set to train the model to obtain a trained model;

[0015] Step 6: Adjust the channels of the low-frequency global branch and the high-frequency local branch of the trained model to obtain a channel-adjusted trained model, and then input the shallow feature map into the channel-adjusted trained model, and obtain a new fusion attention-processed feature map after splicing the image block sequence;

[0016] The new feature map processed by fusion attention is input into the classification regression prediction head to obtain the classification result, and the target is tracked according to the classification result.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] 1. The present invention extracts multi-frequency information simultaneously through a dual-branch structure, and extracts more local information in the high-frequency local branch by window division and convolution weight sharing, thereby improving the network's ability to acquire local information;

[0019] 2. The present invention obtains shallow features of the template image and the search area image through the convolution layer in the early stage of the network, and realizes the extraction of high-frequency information and low-frequency information through the multi-frequency aggregation attention of the shallow layer of the network;

[0020] 3. The present invention realizes the fusion and interaction of template image features and search area image features in the deep network fusion attention, which improves the model's ability to extract high- and low-frequency information and global modeling capabilities.

[0021] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flow chart of a target tracking method based on a multi-frequency aggregation block feature fusion network proposed by the present invention;

[0023] Figure 2 It is a schematic diagram of a multi-frequency aggregation module of a target tracking method based on a multi-frequency aggregation block feature fusion network proposed by the present invention;

[0024] Figure 3 This is a structural diagram of the target tracking framework of the target tracking method based on the multi-frequency aggregation block feature fusion network proposed in the present invention. DETAILED DESCRIPTION

[0025] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0026] These and other aspects of the embodiments of the present invention will be apparent with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0027] See also Figure 1 The embodiment of the present invention proposes a target tracking method based on a multi-frequency aggregation block feature fusion network, the method comprising the following steps:

[0028] Step 1: Extract shallow features of the input image through a convolutional neural network to obtain a shallow feature map;

[0029] In this step, the input image includes a template image and a search area image.

[0030] Step 2: Input the shallow feature map into the multi-frequency aggregation attention module in the multi-frequency aggregation module; in the low-frequency global branch of the multi-frequency aggregation attention module, perform linear transformation and self-attention operations on the shallow feature map in sequence to obtain low-frequency global information;

[0031] In the high-frequency local branch of the multi-frequency aggregation attention module, the shallow feature map is sequentially subjected to multi-window mapping, fusion attention operation, and splicing operation to obtain high-frequency local information;

[0032] The low-frequency global information is concatenated with the high-frequency local information to obtain high- and low-frequency feature maps, and then the high- and low-frequency feature maps are linearly transformed to obtain multi-frequency aggregated attention feature maps;

[0033] Input the multi-frequency aggregated attention feature map into the reverse residual multi-layer perceptron to obtain the feature map processed by the multi-frequency aggregation block;

[0034] See also Figure 2 In step 2, in the low-frequency global branch of the multi-frequency aggregation attention module, the shallow feature map is subjected to linear transformation and self-attention operation in turn to obtain low-frequency global information. The relationship between the corresponding process is:

[0035] ;

[0036] in, It means that after linear transformation, represents the query vector obtained after linear transformation, represents the key vector obtained after linear transformation, represents the value vector obtained after linear transformation, represents low-frequency global information, It means that after the self-attention operation, Indicates that after pooling operation, Represents the shallow feature map;

[0037] In the high-frequency local branch of the multi-frequency aggregation attention module, the shallow feature map is sequentially subjected to multi-window mapping, fusion attention operation and splicing operation to obtain high-frequency local information. The relationship between the corresponding process is:

[0038] ;

[0039] in, represents the height of the shallow feature map, represents the width of the shallow feature map, Indicates the window size, Indicates the total number of windows, represents the shallow feature map, represents the query vector obtained after linear transformation, represents the key vector obtained after linear transformation, represents the value vector obtained after linear transformation, represents the query vector obtained after the convolution operation, represents the key vector obtained after the convolution operation, represents the value vector obtained after the convolution operation, Indicates The window features obtained by attention calculation, represents the dimension of the key vector, represents high-frequency local information, All represent the window features obtained through attention calculation. Indicates that it has been processed by a 2*2 convolution operation;

[0040] The low-frequency global information is concatenated with the high-frequency local information to obtain the high- and low-frequency feature maps, and then the high- and low-frequency feature maps are linearly transformed to obtain the multi-frequency aggregated attention feature maps. The relationship between the corresponding processes is:

[0041] ;

[0042] in, represents the high and low frequency feature map, represents the multi-frequency aggregated attention feature map, Indicates that it has been processed by splicing operation;

[0043] The multi-frequency aggregated attention feature map is input into the reverse residual multi-layer perceptron to obtain the feature map processed by the multi-frequency aggregation block. The relationship between the corresponding process is:

[0044] ;

[0045] in, It means that it has been processed by the reverse residual multilayer perceptron. It means that after the processing of the depth-separable convolution of size 1*1, It means that it has been processed by a depth-wise separable convolution of size 3*3.

[0046] Step 3: Repeat steps 1 and 2 with the template image and the search area image as input to obtain a multi-frequency template feature map and a multi-frequency search area feature map, respectively.

[0047] Step 4: linearly transform the multi-frequency template feature map and the multi-frequency search area feature map to obtain the key vectors of the multi-frequency search area and the template area, the value vectors of the multi-frequency search area and the template area, and the query vector of the multi-frequency search area;

[0048] Performing fusion attention processing on the key vectors of the multi-frequency search area and the template area, the value vectors of the multi-frequency search area and the template area, and the query vector of the multi-frequency search area to obtain a feature map processed by fusion attention;

[0049] In step 4, the key vectors of the multi-frequency search area and the template area, the value vectors of the multi-frequency search area and the template area, and the query vector of the multi-frequency search area are subjected to fusion attention processing to obtain a feature map after fusion attention processing. The relationship between the corresponding process is:

[0050] ;

[0051] in, represents the feature map after fusion attention processing, A query vector representing a multi-frequency search region, A key vector representing the multi-frequency search region and the template region, vector of values ​​for the multi-frequency search region and the template region, Represents the transpose symbol.

[0052] Step 5: Combine the feature graph processed by fusion attention and use a large-scale data set to train the model to obtain a trained model;

[0053] In this step, the present invention is on the RTX 3060Ti GPU, where the PyTorch version is 1.13.0. TrackingNet, GOT-10k, LaSOT, and COCO datasets are used for 300 rounds of training at the same time, and the image pairs in the video sequence are randomly sampled. The target box is expanded 4 times and 2 times respectively to obtain the search area image and the template image. The network training uses the AdamW optimizer, and the weight decay is set to 1e-4. When the training reaches 240 rounds, the learning rate is reduced to the initial 10% to make the model converge quickly.

[0054] Step 6: Adjust the channels of the low-frequency global branch and the high-frequency local branch of the trained model to obtain a channel-adjusted trained model, and then input the shallow feature map into the channel-adjusted trained model, and obtain a new fusion attention-processed feature map after splicing the image block sequence;

[0055] Input the new feature map processed by fusion attention into the classification regression prediction head to obtain the classification result, and track the target according to the classification result;

[0056] See also Figure 3 In step 6, the channels of the low-frequency global branch and the high-frequency local branch of the trained model are adjusted to obtain the trained model after channel adjustment. The relationship between the corresponding process is:

[0057] ;

[0058] in, represents the number of channels of the low-frequency global branch, represents the number of channels of the high-frequency local branch, Indicates the overall number of channels;

[0059] The shallow features of the image are input into the trained model after channel adjustment, and the new feature map of fusion attention processing is obtained after splicing the image block sequence. The relationship between the corresponding process is:

[0060] ;

[0061] in, represents the sequence after the flattening of the search area features, represents the sequence after the template region features are flattened, Represents the overall sequence after splicing.

[0062] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0063] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0064] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A target tracking method based on a multi-frequency aggregation block feature fusion network, the method is implemented based on a prediction model, the prediction model consists of a shallow feature extraction module, a multi-frequency aggregation module, a fusion module, a training module and a prediction module, characterized in that: The method comprises the following steps: Step 1: Extract shallow features of the input image through a convolutional neural network to obtain a shallow feature map; Step 2: Input the shallow feature map into the multi-frequency aggregation attention module in the multi-frequency aggregation module; in the low-frequency global branch of the multi-frequency aggregation attention module, perform linear transformation and self-attention operations on the shallow feature map in sequence to obtain low-frequency global information; In the high-frequency local branch of the multi-frequency aggregation attention module, the shallow feature map is sequentially subjected to multi-window mapping, fusion attention operation, and splicing operation to obtain high-frequency local information; The low-frequency global information is concatenated with the high-frequency local information to obtain high- and low-frequency feature maps, and then the high- and low-frequency feature maps are linearly transformed to obtain multi-frequency aggregated attention feature maps; Input the multi-frequency aggregated attention feature map into the reverse residual multi-layer perceptron to obtain the feature map processed by the multi-frequency aggregation block; Step 3: Repeat steps 1 and 2 with the template image and the search area image as input images to obtain a multi-frequency template feature map and a multi-frequency search area feature map respectively; Step 4: linearly transform the multi-frequency template feature map and the multi-frequency search area feature map to obtain the key vectors of the multi-frequency search area and the template area, the value vectors of the multi-frequency search area and the template area, and the query vector of the multi-frequency search area; Performing fusion attention processing on the key vectors of the multi-frequency search area and the template area, the value vectors of the multi-frequency search area and the template area, and the query vector of the multi-frequency search area to obtain a feature map processed by fusion attention; Step 5: Combine the feature graph processed by fusion attention and use a large-scale data set to train the model to obtain a trained model; Step 6: Adjust the channels of the low-frequency global branch and the high-frequency local branch of the trained model to obtain a channel-adjusted trained model, and then input the shallow feature map into the channel-adjusted trained model, and obtain a new fusion attention-processed feature map after splicing the image block sequence; The new feature map processed by fusion attention is input into the classification regression prediction head to obtain the classification result, and the target is tracked according to the classification result.

2. The target tracking method based on multi-frequency aggregation block feature fusion network according to claim 1 is characterized in that: In step 2, in the low-frequency global branch of the multi-frequency aggregation attention module, the shallow feature map is subjected to linear transformation and self-attention operation in sequence to obtain low-frequency global information. The relationship between the corresponding process is: ； in, It means that after linear transformation, represents the query vector obtained after linear transformation, represents the key vector obtained after linear transformation, represents the value vector obtained after linear transformation, represents low-frequency global information, It means that after the self-attention operation, Indicates that after pooling operation, Represents the shallow feature map.

3. The target tracking method based on multi-frequency aggregation block feature fusion network according to claim 2 is characterized in that: In step 2, in the high-frequency local branch of the multi-frequency aggregation attention module, the shallow feature map is sequentially subjected to multi-window mapping, fusion attention operation and splicing operation to obtain high-frequency local information. The relationship between the corresponding process is: ； in, represents the height of the shallow feature map, represents the width of the shallow feature map, Indicates the window size, Indicates the total number of windows, represents the shallow feature map, represents the query vector obtained after linear transformation, represents the key vector obtained after linear transformation, represents the value vector obtained after linear transformation, represents the query vector obtained after the convolution operation, represents the key vector obtained after the convolution operation, represents the value vector obtained after the convolution operation, Indicates The window features calculated by attention, represents the dimension of the key vector, represents high-frequency local information, All represent the window features obtained through attention calculation. Indicates that it has been processed by a 2*2 convolution operation.

4. The target tracking method based on multi-frequency aggregation block feature fusion network according to claim 3 is characterized in that: In step 2, the low-frequency global information and the high-frequency local information are concatenated to obtain high- and low-frequency feature maps, and then the high- and low-frequency feature maps are linearly transformed to obtain multi-frequency aggregated attention feature maps. The corresponding process has the following relationship: ； in, represents the high and low frequency feature map, represents the multi-frequency aggregated attention feature map, Indicates that it has been processed by the splicing operation.

5. The target tracking method based on multi-frequency aggregation block feature fusion network according to claim 4 is characterized in that: In step 2, the multi-frequency aggregated attention feature map is input into the reverse residual multi-layer perceptron to obtain a feature map processed by the multi-frequency aggregation block. The relationship between the corresponding process is: ； in, It means that it has been processed by the reverse residual multilayer perceptron. It means that after the processing of the depth-separable convolution of size 1*1, It means that it has been processed by a depth-wise separable convolution of size 3*3.

6. The target tracking method based on multi-frequency aggregation block feature fusion network according to claim 5 is characterized in that: In step 4, the key vectors of the multi-frequency search area and the template area, the value vectors of the multi-frequency search area and the template area, and the query vector of the multi-frequency search area are subjected to fusion attention processing to obtain a feature map subjected to fusion attention processing. The relationship between the corresponding process is: ； in, represents the feature map after fusion attention processing, A query vector representing a multi-frequency search region, A key vector representing the multi-frequency search region and the template region, vector of values ​​for the multi-frequency search region and the template region, Represents the transpose symbol.

7. The target tracking method based on multi-frequency aggregation block feature fusion network according to claim 6 is characterized in that: In step 6, the channels of the low-frequency global branch and the high-frequency local branch of the trained model are adjusted to obtain a trained model after channel adjustment. The relationship between the corresponding process is: ； in, represents the number of channels of the low-frequency global branch, represents the number of channels of the high-frequency local branch, Indicates the overall number of channels.

8. The target tracking method based on multi-frequency aggregation block feature fusion network according to claim 7 is characterized in that: In step 6, the shallow feature map is input into the trained model after channel adjustment, and a new feature map of fusion attention processing is obtained after splicing the image block sequence. The relationship between the corresponding process is: ； in, represents the sequence after the flattening of the search area features, represents the sequence after the template region features are flattened, Represents the overall sequence after splicing.

Images