A correlation filtering tracking method and system based on swin-transformer multi-scale depth features

Abstract

The invention discloses a correlation filtering tracking method and system based on swin-transformer multi-scale depth features, which belongs to the field of computer vision. The main steps include: data preparation, selecting OTB100 and GOT-10K data sets, and using a target detection algorithm to obtain an initial target frame; then, in the model initialization stage, marking the target to be tracked by the target detection algorithm; then performing feature extraction, using Swin-Transformer to extract multi-scale features, and generating a feature map; in the calculation of related features, an autocorrelation feature template is formed by Fourier transform. In the response map generation stage, the new frame is processed and the response map is calculated, from which the maximum value position is identified to determine the new position of the target, and the template feature is updated according to the new position. Finally, the steps are executed cyclically to continuously track the target, and the target loss problem is handled, and reinitialization is performed when necessary. This process ensures high-precision target tracking. Effectively improves the accuracy, scale adaptability and long-term stability of target tracking and positioning under complex backgrounds.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a correlation filtering tracking method and system based on swin-transformer multi-scale depth features. Background Art

[0002] Visual object tracking is one of the important basic research problems in the field of computer vision. It means that after detecting or specifying a target in the first frame of a video sequence, the target is continuously tracked in subsequent frames. The target is usually calibrated with a rectangular frame to achieve the location and scale estimation of the target. It has a wide range of applications in real life.

[0003] Currently, there are two main types of mainstream video tracking methods: one is a target tracking method based on correlation filtering; the other is a target tracking method based on deep learning to regress the center point and width and height of the target box.

[0004] The correlation filter tracking algorithm mainly extracts traditional image features, mainly based on hog-related features. Hog features are mainly formed by feature processing based on image gradients. They are low-level features of images. Under conditions such as image motion blur, rainy and foggy weather, and no illumination, the target gradient is not obvious, and the target is often lost or mistracked. Since cyclic shift is used to generate cyclic samples and the image is converted from the spatial domain to the frequency domain for feature calculation, this type of method often has a high positioning accuracy for the target tracking frame.

[0005] The target tracking algorithm based on deep learning often regresses the center point and height and width of the target frame to obtain the position of the target in the tracking sequence. Since the center point and height and width of the target are correlated, the deep learning model generally uses relatively independent losses to train the center point and height and width of the target frame, which will often lead to large deviations in the positioning points of the tracking target frame; due to the use of deep learning model adaptive training samples, this type of method often forms a good feature expression for the target tracking frame. Summary of the invention

[0006] The purpose of the present invention is to overcome the problems in the prior art, in which multi-scale depth features are adapted to complex backgrounds but have large positioning deviations, and the correlation filtering tracking framework has accurate target positioning but is difficult to adapt to complex backgrounds, and to provide a correlation filtering tracking method and system based on swin-transformer multi-scale depth features.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] A correlation filtering tracking method based on swin-transformer multi-scale depth features, the method comprising the following steps:

[0009] S1. Data preparation: Dataset selection: OTB100 and GOT-10K are selected as training and testing datasets;

[0010] Image preprocessing: unify the image size to 128x128 (training stage);

[0011] Use the target detection algorithm to obtain a credible target box (testing phase);

[0012] S2. Model initialization: Target detection: The initialization module obtains the initial target frame from the target detection algorithm and marks the target to be tracked;

[0013] S3. Feature extraction: Multi-scale feature extraction module: The Swin-Transformer attention module is used to extract features from the image in the initial target box to obtain feature maps of different scales. The generated feature map sizes include: 56x56x48, 28x28x96, and 14x14x192.

[0014] S4. Correlation feature calculation: Fourier transform: Perform Fourier transform on the extracted feature maps of each scale to obtain the autocorrelation feature template in the Fourier domain;

[0015] S5. Response map generation: perform the same preprocessing and feature extraction on the next frame image; perform cross-correlation calculation on the feature map generated by the new frame and the previous template feature to obtain a response map;

[0016] S6. Target position update: identify the location of the maximum value from the response graph and determine the new position of the target in this frame;

[0017] S7. Template update: Feature update: According to the new position, recalculate the new template features through the same path as the initialization module and update the target template;

[0018] S8. Loop execution: repeat steps S4 to S7 to continuously track the target; target loss processing: if the target is lost, re-use the target detection algorithm to obtain a new target frame and initialize it again.

[0019] Furthermore, the target detection algorithm is a YOLO series target detection algorithm.

[0020] Furthermore, the Swin-Transformer Block attention module is composed of a window-based multi-head self-attention module M-WSA and a sliding window-based multi-head self-attention module SW-MSA connected in series, and the final feature map shape is consistent with the input.

[0021] Furthermore, in step S3, the feature map is generated by downsampling the feature map by a factor of 2 through a Patch Merging operation to obtain feature maps of different scales.

[0022] A correlation filtering tracking system based on swin-transformer multi-scale depth features is provided for tracking and locating targets in complex backgrounds.

[0023] The beneficial effects of the present invention are:

[0024] (1) Combining the advantages of multi-scale deep features and correlation filtering frameworks, that is, multi-scale deep features adapt to complex backgrounds, and correlation filtering tracking frameworks accurately locate targets;

[0025] (2) Effectively improve the accuracy, scale adaptability and long-term stability of target tracking and positioning in complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A flowchart of the method is shown in FIG.

[0027] Figure 2 Schematic diagram of the Swin-Transformer attention module structure;

[0028] Figure 3 Schematic diagram of Swin-Transformer multi-feature extraction structure;

[0029] Figure 4 Diagram of the training framework for multi-scale feature extraction;

[0030] Figure 5 This is a diagram of the correlation filter tracking framework. DETAILED DESCRIPTION

[0031] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0032] See also Figure 1 , provides a correlation filtering tracking method based on swin-transformer multi-scale depth features, the steps include:

[0033] S1. Data preparation: Dataset selection: OTB100 and GOT-10K are selected as training and testing datasets;

[0034] Image preprocessing: unify the image size to 128x128 (training stage);

[0035] Use the target detection algorithm to obtain a credible target box (testing phase);

[0036] S2. Model initialization: Target detection: The initialization module obtains the initial target frame from the target detection algorithm and marks the target to be tracked;

[0037] S3. Feature extraction: Multi-scale feature extraction module: The Swin-Transformer attention module is used to extract features from the image in the initial target box to obtain feature maps of different scales. The generated feature map sizes include: 56x56x48, 28x28x96, and 14x14x192.

[0038] S4. Correlation feature calculation: Fourier transform: Perform Fourier transform on the extracted feature maps of each scale to obtain the autocorrelation feature template in the Fourier domain;

[0039] S5. Response map generation: perform the same preprocessing and feature extraction on the next frame image; perform cross-correlation calculation on the feature map generated by the new frame and the previous template feature to obtain a response map;

[0040] S6. Target position update: identify the location of the maximum value from the response graph and determine the new position of the target in this frame;

[0041] S7. Template update: Feature update: According to the new position, recalculate the new template features through the same path as the initialization module and update the target template;

[0042] S8. Loop execution: repeat steps S4 to S7 to continuously track the target; target loss processing: if the target is lost, re-use the target detection algorithm to obtain a new target frame and initialize it again.

[0043] Embodiment 1:

[0044] S1. Data preparation:

[0045] Dataset selection: OTB100 and GOT-10K are selected as training and testing datasets.

[0046] Image preprocessing: The images in the training phase are resized to 128×128 pixels.

[0047] Object detection: During the testing phase, an object detection algorithm (such as YOLO) is used to obtain credible object boxes.

[0048] S2. Model 𝓕 initialization:

[0049] Target detection: Get the initial target frame through the target detection algorithm. Assume that the initialization frame is:

[0050] ;

[0051] S3. Feature extraction:

[0052] Multi-scale feature extraction: Use Swin-Transformer to extract features from the 128×128 image in the initial target box to obtain feature maps of different scales. The generated feature map sizes are as follows:

[0053] Size 1: 56×56×4856×56×48;

[0054] Size 2: 28×28×9628×28×96;

[0055] Size 3: 14×14×19214×14×192;

[0056] S4. Calculation of relevant features:

[0057] Fourier transform: Perform Fourier transform on the extracted feature maps of each scale to obtain the autocorrelation feature template in the Fourier domain. Assume that a certain scale feature map is:

[0058] ;

[0059] Then perform Fourier transform;

[0060] S5. Response graph generation:

[0061] New frame processing: Extract the target area from the next frame image and generate a feature map , assuming that the size of the feature map after processing is 56×56×48;

[0062] Cross-correlation calculation: Perform cross-correlation calculation with the previous template features to obtain a response map;

[0063] Assume that the maximum position of the cross-correlation is , we calculate:

[0064] ;

[0065] S6. Target position update:

[0066] Identify the maximum value position from the response graph, for example, the maximum value position is:

[0067] ;

[0068] S7. Template Update:

[0069] Feature update: Based on new location The new template features are recomputed through the same path as the initialization module:

[0070] F new = F ( I [ x ′ , y ′ , w 0 , h 0 ] ) ;

[0071] S8. Loop execution:

[0072] Continuous tracking: Repeat steps S4 to S7 to track the target in real time.

[0073] If the target is lost, use the target detection algorithm to re-acquire a new target frame:

[0074] .

[0075] Re-acquire the target position.

[0076] Image target frame data preprocessing: For the training image sequence, since the training image sequence already has a target tracking frame label, the target tracking frame of the training image sequence is expanded outward from the center point to twice the tracking frame, and then resized to 128x128 size; for the test image sequence, the image sequence is often detected through a target detection algorithm (such as the YOLO series) until a credible target is detected, and the target tracking frame preprocessing method consistent with the training image sequence is used.

[0077] See also Figure 2 , Swin-Transformer Block attention module: The Swin-Transformer Block attention module is composed of a window-based multi-head self-attention module M-WSA and a sliding window-based multi-head self-attention module SW-MSA in series. The final feature map shape is consistent with the input. The specific structure is shown in Figure 2 .

[0078] See also Figure 3 , Multi-scale deep feature extraction: The input of feature extraction is obtained by the target tracking box through the above image preprocessing module, and the shape is 128x128x3; through the Patch Partition and Linear Embedding layers, a 56x56x48 feature map is obtained, which means 56x56 tokens are obtained, and the dimension of each token is 48; the 56x56x48 feature map is passed through the above Swin-Transformer Block attention module to obtain a feature map shape consistent with the input shape, that is, 56x56x48. The specific structure is shown in Figure 1; Then, the feature map is downsampled by 2 times through the Patch Merging operation, and the number of channels is doubled, and the size of the feature map is 28x28x96; then the aforementioned Swin-Transformer Block attention module is repeated to extract features, and Patch Merging is used to downsample the feature map to obtain feature maps of different scales. A total of three feature maps of scales are obtained, and the sizes of the feature maps are 56x56x48, 28x28x96, and 14x14x192, respectively. The specific structure is shown in Figure 3 .

[0079] participate Figure 4 , multi-scale deep feature extractor training framework: the training data set consists of OTB100 and GOT-10K, and the labels annotated in each sequence are obtained by the above image preprocessing method to obtain a size of 128x128x3; the forward reasoning process is to randomly extract two images from each sequence in the data set, one is the tracking template and the other is the search template, which are sequentially passed through the above multi-scale deep feature extractor and converted to the Fourier domain to obtain the response map of the corresponding scale; the training label is the Gaussian label at each scale, that is, the label with the center point of the corresponding scale as the two-dimensional Gaussian distribution; the loss function is the sum of the mean square error of the response map and the corresponding two-dimensional Gaussian distribution label at each scale; the model parameters of the multi-scale deep feature extractor are trained by back propagation, and the specific training framework is shown in Figure 4 .

[0080] See also Figure 5 ,The correlation filter tracking framework consists of an initialization module, a detection module and a training ,update module.

[0081] The initialization module obtains the initialization target frame through the target detection algorithm, obtains the input of the feature module through the preprocessing module, and then obtains the feature autocorrelation feature through the feature extractor module and converts it to the Fourier domain; the detection module also obtains the search feature through the preprocessing module and feature extraction module and converts it to the Fourier domain, and cross-correlates it with the template feature obtained by the initialization module to obtain the position of the target in the next frame; the training update module mainly obtains the new template feature from the new position of the target obtained by detection through the same processing path as the initialization module. The above steps are repeated in a common framework for correlation filter tracking. See the specific framework diagram for details. Figure 5 .

[0082] First, the initial target frame of the tracked target is obtained through the detection model; then the target frame is sent to the multi-scale target feature extractor through the image preprocessing module to obtain the features at each scale, and then converted to the Fourier domain through Fourier transform, and the autocorrelation feature templates at each scale are calculated in the Fourier domain;

[0083] The next frame image is at the position of the previous frame image. It is also preprocessed and sent to multi-scale feature extraction to obtain features at each scale, which are converted to the Fourier domain to obtain features at each scale, and then cross-correlation features are obtained with the template features of the previous frame image to obtain response maps at each scale. The largest scale of the response map is selected as the response position of the target in this frame, and the position of the target in this frame is obtained.

[0084] Finally, the template features of the target at each scale are updated according to the position of the target in the frame, and the above process is repeated until the target is lost and a new target tracking frame is obtained through the target detection related algorithm.

[0085] This embodiment combines the advantages and disadvantages of two types of visual target tracking methods, correlation filtering and deep learning, adopts the current mainstream multi-scale depth feature expression based on swin-transformer, and feeds it into the target tracking framework of correlation filtering, which effectively improves the accuracy, scale adaptability and long-term stability of target tracking and positioning in complex backgrounds.

[0086] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A correlation filtering tracking method based on swin-transformer multi-scale depth features, characterized in that: The method steps include: S1. Data preparation: Dataset selection: OTB100 and GOT-10K are selected as training and testing datasets; Image preprocessing: Unify the image size to 128x128; Use the target detection algorithm to obtain a credible target box; S2. Model initialization: Target detection: The initialization module obtains the initial target frame from the target detection algorithm and marks the target to be tracked; S3. Feature extraction: Multi-scale feature extraction module: The Swin-Transformer attention module is used to extract features from the image in the initial target box to obtain feature maps of different scales. The generated feature map sizes include: 56x56x48, 28x28x96, and 14x14x192. S4. Correlation feature calculation: Fourier transform: Perform Fourier transform on the extracted feature maps of each scale to obtain the autocorrelation feature template in the Fourier domain; S5. Response map generation: perform the same preprocessing and feature extraction on the next frame image; perform cross-correlation calculation on the feature map generated by the new frame and the previous template feature to obtain a response map; S6. Target position update: identify the location of the maximum value from the response graph and determine the new position of the target in this frame; S7. Template update: Feature update: According to the new position, recalculate the new template features through the same path as the initialization module and update the target template; S8. Loop execution: repeat steps S4 to S7 to continuously track the target; target loss processing: if the target is lost, re-use the target detection algorithm to obtain a new target frame and initialize it again.

2. According to claim 1, a correlation filtering tracking method based on swin-transformer multi-scale depth features is characterized in that: The target detection algorithm is the YOLO series target detection algorithm.

3. According to claim 1, a correlation filtering tracking method based on swin-transformer multi-scale depth features is characterized in that: The Swin-Transformer Block attention module is composed of a window-based multi-head self-attention module M-WSA and a sliding window-based multi-head self-attention module SW-MSA in series, and the final feature map shape is consistent with the input.

4. According to claim 1, a correlation filtering tracking method based on swin-transformer multi-scale depth features is characterized in that: In step S3, the feature map is generated by downsampling the feature map by a factor of 2 through a Patch Merging operation to obtain feature maps of different scales.

5. A correlation filtering tracking system based on swin-transformer multi-scale depth features, characterized in that: Used to execute the correlation filtering tracking method based on swin-transformer multi-scale depth features described in claims 1 to 4, and used for tracking and positioning targets in complex backgrounds.

Images