RGBT target tracking method based on frequency space enhancement and time adaptation

Abstract

The embodiment of the invention relates to the technical field of computer vision, in particular to an RGBT target tracking method based on frequency space enhancement and time adaptation, which comprises the following steps: obtaining a training sample set which comprises a plurality of video sequences, and each video sequence is composed of a visible light image and a thermal infrared image which are paired; constructing an RGBT target tracking model based on frequency space enhancement and time adaptation, wherein the RGBT target tracking model is composed of a frequency space enhancement network, a cross-modal feature extraction network and an online score prediction network; performing iterative training on the RGBT target tracking model based on the training sample set until convergence to obtain a trained model; and inputting the target video into the trained model to obtain a tracking result of the target video output by the trained model. According to the method, the coping capacity of RGBT target tracking in challenging scenes such as background interference and thermal crossover can be well improved, and the accuracy, efficiency and robustness of RGBT target tracking are effectively improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and in particular to an RGBT target tracking method based on frequency-space enhancement and time adaptation. Background Art

[0002] Visual object tracking is a classic task in computer vision technology. It aims to predict the position and size of an object in subsequent video frames, given an initial target in a given video. Visual object tracking has a wide range of applications in video surveillance, autonomous driving, detection and reconnaissance, and other scenarios. Although many RGB-based single-modal trackers have been proposed in recent years, their performance in complex scenes such as rain, fog, and extreme lighting conditions is often unsatisfactory.

[0003] In contrast, in the field of computer vision, multispectral data can describe image characteristics from different perspectives. With the rapid development of technologies such as big data and multispectral sensors, the difficulty of acquiring multimodal data has been greatly reduced. Effectively utilizing multispectral data and mining the inherent correlations between different modal data can significantly improve the performance of downstream tasks. Visible light is the wavelength most consistent with human vision, with a relatively short wavelength, generally ranging from 400nm to 780nm. Currently, visible light is the most readily available image data, clearly capturing details such as target color and texture, and offering high resolution. However, due to its short wavelength and low penetration, it struggles to maintain high image quality in adverse weather conditions and extreme lighting conditions. This makes target tracking challenging in scenes with extreme lighting conditions, rain, and fog. Thermal infrared, which reflects the distribution of heat sources in images, typically ranges from 3μm to 15μm. Its longer wavelength and strong penetration make its image quality insensitive to adverse weather conditions and lighting variations. Therefore, it can achieve omnidirectional, multi-weather imaging with guaranteed image quality even in adverse weather conditions. Thermal infrared can reveal the location and outline of a target based on the temperature in an image, but it struggles to capture detailed features like texture and color. This leads to issues like thermal crosstalk caused by adjacent objects of similar temperatures during target tracking. Therefore, RGBT tracking technology, which combines the complementary information of RGB and thermal infrared modalities, has emerged. RGBT tracking technology can achieve excellent tracking performance in a variety of challenging scenarios.

[0004] Currently, RGBT tracking methods designed solely using CNN architectures lack global context modeling capabilities, limiting their tracking performance. With the tremendous success of Transformers in computer vision, Transformer-based RGBT tracking methods have gradually emerged. However, these Transformer-based tracking models still rely on CNNs for feature extraction, limiting their global modeling capabilities and failing to fully realize the Transformer's potential.

[0005] Faced with this situation, the RGBT tracking method based on ViT (Vision Transformer) has emerged. This method can simultaneously perform feature extraction and information exchange. Using the ViT backbone network, bimodal feature extraction and inter-modal interaction can be achieved simultaneously. This method enables global context modeling, which is difficult to achieve with traditional CNN methods, and especially improves tracking performance in complex environments. The RGBT tracking method using ViT achieves higher tracking accuracy by combining feature extraction and inter-modal interaction. This makes it more robust under adverse weather conditions and extreme lighting challenges.

[0006] However, the currently proposed ViT-based RGBT tracking methods still suffer from the problem of insufficient interaction between the template and the search image, which results in the ViT backbone network being unable to extract discernible features and thus unable to accurately distinguish between the target and the background. Summary of the Invention

[0007] In order to solve the above technical problems, the embodiments of the present application propose an RGBT target tracking method based on frequency-space enhancement and time adaptation, which can enhance and optimize the collaborative work of different features, enabling the network to focus on the complementary features between different modalities, thereby improving the ability of RGBT target tracking to cope with challenging scenarios such as background interference and thermal crosstalk, thereby improving the accuracy, efficiency and robustness of RGBT target tracking.

[0008] In the first aspect, an embodiment of the present application proposes an RGBT target tracking method based on frequency-space enhancement and time adaptation, the method comprising: obtaining a training sample set; wherein the training sample set contains multiple video sequences, each video sequence is composed of a pair of visible light images and thermal infrared images; constructing an RGBT target tracking model based on frequency-space enhancement and time adaptation; the RGBT target tracking model is composed of a frequency-space enhancement network, a cross-modal feature extraction network and an online score prediction network, and the frequency-space enhancement network is composed of a visible light embedding branch and a thermal infrared embedding branch; based on the training sample set, the RGBT target tracking model is iteratively trained until convergence to obtain a trained model; wherein the RGBT target tracking model The input is specifically the search area and template area of the visible light image and thermal infrared image; the frequency-space enhancement network performs more discriminative feature extraction on the visible light image and thermal infrared image through the frequency attention mechanism and the spatial enhancement mechanism, and obtains the embedding vector of the visible light modality and the embedding vector of the thermal infrared modality; the cross-modal feature extraction network connects the embedding vectors of the two modalities, realizes the feature fusion and interaction between the two modalities to obtain the fused features, and generates the bounding box of the target based on the fused features; the online score prediction network calculates the confidence score of the target, and uses the confidence score to dynamically update the template area to adapt to the changes of the target; the target video is input into the trained model to obtain the tracking result of the target video output by the trained model.

[0009] In some optional embodiments, the search area and template area of the visible light image are input into the visible light embedding branch, and the search area and template area of the thermal infrared image are input into the thermal infrared embedding branch. The visible light embedding branch and the thermal infrared embedding branch each include three embedding layers, and each embedding layer is connected to a frequency space enhancement module; each embedding layer is used to extract features of its own input search area and template area respectively to obtain search features and template features; each frequency space enhancement module consists of a frequency attention unit and a space enhancement unit, and the frequency attention units share weights; the frequency attention unit uses multiple discrete cosine transform frequency components as a channel The frequency attention unit of the search feature and the template feature is obtained and averaged, a frequency attention mask is created to improve the robustness of the frequency attention, and the frequency attention mask is multiplied with the search feature and the template feature to generate an integrated frequency attention interaction between the search feature and the template feature, thereby obtaining the frequency attention interaction feature; the spatial enhancement unit only models the connection between each pixel point of the frequency attention interaction feature and other pixels in the same row or column, thereby enhancing the spatial information of the frequency attention interaction feature, and finally combines it with the search feature and the template feature through the residual connection, and finally outputs the search enhanced feature and the template enhanced feature.

[0010] In some optional embodiments, the input of the cross-modal feature extraction network is the embedding vector of the visible light modality output by the visible light embedding branch, and the embedding vector of the thermal infrared modality output by the thermal infrared embedding branch. The cross-modal feature extraction network is composed of several ViT encoder layers connected sequentially; the cross-modal feature extraction network improves the single-layer fully connected embedding of the original ViT encoder layer into a multi-layer convolutional embedding based on a multi-scale pyramid structure and local inductive bias. The cross-modal feature extraction network uses three layers of convolution in the embedding layer, and applies a local inductive bias similar to spatial attention after each layer of convolution to model the correlation between local pixels, and finally outputs the bounding box of the target.

[0011] In some optional embodiments, the online score prediction network consists of two Transformer decoders and a three-layer perceptron, and the two Transformer decoders are a search token decoder and a template token decoder connected sequentially. The online score prediction network first initializes a learnable score tag, and uses the score tag together with the search area tag as the query, key and value input of the search token decoder, extracts the target information through the score tag encoding, then uses the score tag together with the template tag as the input of the template token decoder, implicitly compares the extracted target with the previous online template target, and finally calculates the confidence score of the target through the three-layer perceptron and the Sigmoid activation function layer. If the calculated confidence score is greater than the currently used online template score, the template area is dynamically updated based on a preset template update interval to adapt to the change of the target.

[0012] In some optional embodiments, the currently used online template score decays as the RGBT target tracking progresses. The decay formula of the online template score is:

[0013]

[0014] in, represents the online template score used when tracking the t-1 frame video, d is the preset attenuation coefficient, Represents the online template score used when tracking the t-th frame of the video.

[0015] In some optional embodiments, the iterative training of the RGBT target tracking model is divided into two stages; in the first stage, the network parameters of the online score prediction network are fixed, and the frequency-space enhancement network and the cross-modal feature extraction network are trained. The training in the first stage uses adaptively weighted Focal classification loss, GIoU loss and L1 regression loss for constraints, so that the RGBT target tracking model can focus on generating bounding boxes of targets with higher confidence and more accuracy; in the second stage, the network parameters of the frequency-space enhancement network and the network parameters of the cross-modal feature extraction network are fixed, and only the NCE loss is used to train the online score prediction network to ensure that the RGBT target tracking model can dynamically select reliable online templates.

[0016] In some optional embodiments, the loss function of the first stage training is expressed by the formula:

[0017] L stage1 =αL Focal +βL Giou +γL L1 ;

[0018] Among them, L Focal , L Giou and L L1 They represent adaptive weighted Focal classification loss, GIoU loss, and L1 regression loss, respectively. α, β, and γ are all preset adaptive weighting coefficients. L stage1 Represents the loss function of the first stage training;

[0019] The loss function of the second stage training is expressed as follows:

[0020]

[0021] Among them, n represents the total number of samples in the second stage training, y i represents the label of the i-th sample, p i represents the predicted probability, L stage2 Represents the loss function of the second stage training.

[0022] This application proposes an RGBT target tracking method based on frequency-space enhancement and time adaptation, which constructs, trains and uses an RGBT target tracking model based on frequency-space enhancement and time adaptation to perform RGBT target tracking. The RGBT target tracking model enhances and optimizes the collaborative work of different features through a backbone network composed of a frequency-space enhancement network and a cross-modal feature extraction network, so that the network can focus on the complementary features between different modalities, thereby improving the ability to cope with challenging scenarios such as background interference and thermal crosstalk. In order to make RGBT target tracking more robust and temporally adaptable, this application introduces an online template update strategy, which achieves precise control through an online score prediction network. The online score prediction network calculates the confidence score of the extracted target, and uses the confidence score to dynamically update the template area to adapt to changes in the target. The introduction of this strategy significantly improves the adaptability of RGBT target tracking, and can achieve accurate tracking even in severe scenarios such as severe occlusion or frequent camera movement.

[0023] In the second aspect, an embodiment of the present application proposes an RGBT target tracking system based on frequency-space enhancement and time adaptation, the system comprising: a sample acquisition module for acquiring a training sample set, wherein the training sample set contains multiple video sequences, each video sequence consisting of a pair of visible light images and thermal infrared images; a model construction module for constructing an RGBT target tracking model based on frequency-space enhancement and time adaptation, the RGBT target tracking model consisting of a frequency-space enhancement network, a cross-modal feature extraction network and an online score prediction network, and the frequency-space enhancement network consists of a visible light embedding branch and a thermal infrared embedding branch; a model training module for iteratively training the RGBT target tracking model based on the training sample set until convergence to obtain a trained model, wherein R The GBT object tracking model takes as input the search and template regions of visible light and thermal infrared images. The frequency-space enhancement network uses a frequency attention mechanism and a spatial enhancement mechanism to extract more discriminative features from these images, generating embedding vectors for the visible light and thermal infrared modalities. The cross-modal feature extraction network connects the embedding vectors of the two modalities, integrating and interacting the features between them to generate fused features. The model then generates a bounding box for the target based on these fused features. The online score prediction network calculates the target's confidence score and uses this confidence score to dynamically update the template region to adapt to changes in the target. The model usage module then feeds the trained model into the target video and outputs the tracking results for the target video.

[0024] In a third aspect, an embodiment of the present application proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute an RGBT target tracking method based on frequency-space enhancement and time adaptation as described in the first aspect above.

[0025] In a fourth aspect, an embodiment of the present application proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement an RGBT target tracking method based on frequency-space enhancement and time adaptation as described in the first aspect above.

[0026] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related technologies, the following is a brief introduction to the drawings required for use in the embodiments of the present application or the description of the related technologies. Obviously, the following drawings are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. The drawings described here are only used to explain the present application and are not used to limit the present application.

[0028] Figure 1 This is a flowchart of an RGBT target tracking method based on frequency-space enhancement and time adaptation provided in one embodiment of the present application;

[0029] Figure 2 1 is a schematic structural diagram of an RGBT target tracking model based on frequency-space enhancement and time adaptation provided in one embodiment of the present application;

[0030] Figure 3 is a schematic structural diagram of a frequency space enhancement module provided in one embodiment of the present application;

[0031] Figure 4 is a schematic diagram of the structure of an online score prediction network provided in one embodiment of the present application;

[0032] Figure 5 is a schematic diagram of a tracking effect provided by an embodiment of the present application;

[0033] Figure 6is a structural diagram of an RGBT target tracking system based on frequency space enhancement and time adaptation provided in another embodiment of the present application;

[0034] Figure 7 It is a structural diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, each embodiment of the present application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will appreciate that in each embodiment of the present application, many technical details are provided to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be implemented. The division of the following embodiments is for convenience of description and should not constitute any limitation on the specific implementation of the present application. The various embodiments can be combined with each other and referenced to each other under the premise of no contradiction.

[0036] One embodiment of the present application proposes an RGBT target tracking method based on frequency-space enhancement and time adaptation, which is applied to an electronic device, where the electronic device can be a terminal or a server. This embodiment and the following embodiments are all described using the server as an example. The following is a detailed description of the implementation details of the RGBT target tracking method based on frequency-space enhancement and time adaptation proposed in this embodiment. The following content is only for the convenience of understanding and is not required for the implementation of this solution.

[0037] The specific process of the RGBT target tracking method based on frequency space enhancement and time adaptation proposed in this embodiment can be as follows: Figure 1 Shown, including:

[0038] Step 101: Obtain a training sample set, where the training sample set includes multiple video sequences, each of which consists of a pair of visible light images and thermal infrared images.

[0039] In the implementation, the server first needs to obtain a training sample set, which contains multiple video sequences, each consisting of a pair of visible light and thermal infrared images. In addition to the training sample set, the server also needs to obtain a test sample set, which is used to subsequently test the performance of the trained RGBT object tracking model.

[0040] In one example, the server can download the current mainstream public RGBT target tracking dataset through the Internet as a training sample set and a test sample set of the RGBT target tracking model.

[0041] Step 102: construct an RGBT target tracking model based on frequency-space enhancement and time adaptation. The RGBT target tracking model consists of a frequency-space enhancement network, a cross-modal feature extraction network, and an online score prediction network.

[0042] In the specific implementation, after the server obtains the training sample set, it can build an RGBT target tracking model based on frequency-space enhancement and time adaptation. The RGBT target tracking model consists of a frequency-space enhancement network, a cross-modal feature extraction network, and an online score prediction network. The frequency-space enhancement network consists of a visible light embedding branch and a thermal infrared embedding branch. The specific structure of the RGBT target tracking model built by the server can be as follows: Figure 2 shown.

[0043] The inputs to the RGBT object tracking model are the search and template regions of the visible light image, as well as the search and template regions of the thermal infrared image. The visible light image search and template regions are fed into the visible light embedding branch of the frequency-space enhancement network, while the thermal infrared image search and template regions are fed into the thermal infrared embedding branch of the frequency-space enhancement network. The outputs of the visible light and thermal infrared embedding branches serve as input to the cross-modal feature extraction network, which in turn serves as input to the online score prediction network. The output of the online score prediction network is then fed back into the frequency-space enhancement network.

[0044] Step 103: Iteratively train the RGBT target tracking model based on the training sample set until convergence, to obtain a trained model.

[0045] In the specific implementation, after the server completes the construction of the RGBT target tracking model and the training sample set, it can iteratively train the RGBT target tracking model based on the training sample set until convergence, and obtain a trained model. During the training process, the frequency-space enhancement network uses the frequency attention mechanism and the spatial enhancement mechanism to extract more discriminative features from visible light images and thermal infrared images, and obtains the embedding vector of the visible light modality and the embedding vector of the thermal infrared modality. The cross-modal feature extraction network connects the embedding vectors of the two modalities, realizes the feature fusion and interaction between the two modalities to obtain the fused features, and generates the bounding box of the target based on the fused features. The online score prediction network calculates the confidence score of the target and uses the confidence score to dynamically update the template area to adapt to changes in the target (mainly changes in the target shape).

[0046] In one example, the search area and template area of the visible light image are input into the visible light embedding branch, and the search area and template area of the thermal infrared image are input into the thermal infrared embedding branch. Both the visible light embedding branch and the thermal infrared embedding branch contain three embedding layers, and each embedding layer is connected to a frequency space enhancement module. Each embedding layer is used to extract features of its own input search area and template area respectively to obtain search features and template features. Each frequency space enhancement module consists of two parts: a frequency attention unit and a spatial enhancement unit, and each frequency attention unit shares weights. The specific structure of the frequency space enhancement module can be as follows Figure 3 shown.

[0047] For each frequency-spatial enhancement module, the frequency attention unit uses multiple discrete cosine transform frequency components as channel coefficients to obtain the frequency attention units of the search feature and template feature and average them together. This creates a frequency attention mask to improve the robustness of the frequency attention. The frequency attention mask is then multiplied with the search feature and template feature to generate an integrated frequency attention interaction between the search feature and the template feature, thereby obtaining a frequency attention interaction feature. For each pixel in the frequency attention interaction feature, the spatial enhancement unit only models the connection between it and other pixels in the same row or column, thereby enhancing the spatial information of the frequency attention interaction feature. Finally, it is combined with the search feature and template feature through a residual connection to ultimately output the search enhanced feature and the template enhanced feature. Although the presence of the spatial enhancement unit slightly increases the number of parameters in the RGBT object tracking model, it can enhance the spatial information of the features.

[0048] It can be understood that since the absolute sizes of the search area and the template area are different, using the spatial enhancement unit to enhance their spatial information respectively, and finally combining the search features and template features through the residual connection can well strengthen the correlation between them.

[0049] In one example, the input to the cross-modal feature extraction network is the embedding vector of the visible light modality output by the visible light embedding branch, and the embedding vector of the thermal infrared modality output by the thermal infrared embedding branch. The cross-modal feature extraction network consists of several sequentially connected ViT encoder layers. The cross-modal feature extraction network improves the single-layer fully connected embedding of the original ViT encoder layer into a multi-layer convolutional embedding based on a multi-scale pyramid structure and local inductive bias. The cross-modal feature extraction network uses three layers of convolution in the embedding layer and applies a local inductive bias similar to spatial attention after each convolution layer to model the correlation between local pixels. Finally, a regression head is used to output the bounding box of the target. The improvement of the ViT encoder layer by the cross-modal feature extraction network significantly enhances the feature extraction capability of the RGBT target tracking model, thereby improving the accuracy of RGBT target tracking.

[0050] In one example, the number of ViT encoder layers in the cross-modal feature extraction network is 11.

[0051] In an example, the specific structure of the online score prediction network can be as follows Figure 4 As shown in the figure, the online score prediction network consists of two Transformer decoders and a three-layer perceptron. The two Transformer decoders are sequentially connected to the search token decoder and the template token decoder. The online score prediction network first initializes a learnable score token and uses the score token and the search region token as the query, key, and value inputs of the search token decoder. The target information is extracted through the score token encoding. Then, the score token and the template token are used as the input of the template token decoder. The extracted target is implicitly compared with the previous online template target. Finally, the target confidence score is calculated using a three-layer perceptron and a sigmoid activation function layer. If the calculated confidence score is greater than the current online template score, the template region is dynamically updated based on the preset template update interval to adapt to the target change.

[0052] In an example, when the RGBT target tracking model is actually used, the online template score currently used decays as the RGBT target tracking progresses. The decay formula of the online template score is:

[0053]

[0054] in, represents the online template score used when tracking the t-1 frame video, d is the preset attenuation coefficient, Represents the online template score used when tracking the t-th frame of the video.

[0055] That is to say, for each frame of video tracked, the online score prediction network will calculate the confidence score and compare it with the attenuated online template score. If the current confidence score is higher than the currently used online template score, the current target and the corresponding confidence score will be recorded, but will only be updated at the preset template update interval.

[0056] In one example, the server's iterative training of the RGBT object tracking model is divided into two main phases. In the first phase, the server fixes the network parameters of the online score prediction network and trains the frequency-space enhancement network and the cross-modal feature extraction network. The first phase of training uses adaptively weighted Focal classification loss, GIoU loss, and L1 regression loss for constraints, allowing the RGBT object tracking model to focus on generating bounding boxes with higher confidence and more accurate targets. In the second phase, the server fixes the network parameters of the frequency-space enhancement network and the cross-modal feature extraction network, and only trains the online score prediction network using the NCE loss to ensure that the RGBT object tracking model can dynamically select reliable online templates.

[0057] In one example, the loss function for the first stage of training is expressed as:

[0058] L stage1 =αL Focal +βL Giou +γL L1 ;

[0059] Among them, L Focal , L Giou and L L1 They represent adaptive weighted Focal classification loss, GIoU loss, and L1 regression loss, respectively. α, β, and γ are all preset adaptive weighting coefficients. L stage1 Represents the loss function of the first stage training.

[0060] In one example, the loss function for the second stage of training is formulated as:

[0061]

[0062] Among them, n represents the total number of samples in the second stage training, y i represents the label of the i-th sample, p i represents the predicted probability, L stage2 Represents the loss function of the second stage training.

[0063] Step 104: input the target video into the trained model to obtain the tracking result of the target video output by the trained model.

[0064] In the specific implementation, after obtaining the trained RGBT target tracking model, the server can deploy the trained RGBT target tracking model in the scenario where it is needed. When there is a need for RGBT target tracking, the target video (the video that requires RGBT target tracking) is input into the trained RGBT target tracking model, and the tracking result of the target in the target video output by the trained RGBT target tracking model is obtained.

[0065] This embodiment proposes an RGBT target tracking method based on frequency-space enhancement and temporal adaptation. It constructs, trains, and uses an RGBT target tracking model based on frequency-space enhancement and temporal adaptation to perform RGBT target tracking. The RGBT target tracking model enhances and optimizes the collaborative work of different features through a backbone network consisting of a frequency-space enhancement network and a cross-modal feature extraction network, enabling the network to focus on complementary features between different modalities, thereby improving its ability to cope with challenging scenarios such as background interference and thermal crosstalk. To make RGBT target tracking more robust and temporally adaptable, this embodiment introduces an online template update strategy that achieves precise control through an online score prediction network. The online score prediction network calculates the confidence score of the extracted target and uses the confidence score to dynamically update the template area to adapt to changes in the target. The introduction of this strategy significantly improves the adaptability of RGBT target tracking, enabling accurate tracking even in challenging scenarios such as severe occlusion or frequent camera movement.

[0066] The steps of the various methods described above are divided for clarity of description only. They can be combined into a single step, or some steps can be broken down into multiple steps. As long as they share the same logical relationships, they are all within the scope of protection of this application. Adding minor modifications or introducing minor design changes to the algorithm or process, but not changing the core design of the algorithm or process, is also within the scope of protection of this application.

[0067] In one embodiment, we use the following simulation experiments to demonstrate the superiority of the RGBT target tracking method based on frequency space enhancement and time adaptation (hereinafter referred to as FETA) proposed in this application. We visualize the tracking results of FETA in four challenging scenarios, such as Figure 5 shown. Figure 5 The tracking results in part a show that FETA can still accurately track the target even in severely occluded scenes. Figure 5 Part c contains multiple challenges, such as low illumination, partial occlusion, and low resolution, and the prediction results of FETA are very close to the true values. Figure 5In parts b and d, FETA maintains its robustness despite deformation, thermal crosstalk, background interference, and camera motion. These experimental results demonstrate that FETA closely integrates search features and template features, helping the network extract discriminative features. Furthermore, the online score prediction network dynamically updates the online template by combining information from previous and next frames, giving FETA temporal adaptability and high efficiency in a variety of challenging scenarios.

[0068] Another embodiment of the present application proposes an RGBT target tracking system based on frequency-space enhancement and time adaptation. The following is a detailed description of the RGBT target tracking system based on frequency-space enhancement and time adaptation proposed in this embodiment. The following content is only for the convenience of understanding the implementation details provided and is not necessary for the implementation of this embodiment. Figure 6 20 is a schematic structural diagram of an RGBT target tracking system based on frequency-space enhancement and time adaptation proposed in this embodiment, including: a sample acquisition module 201, a model construction module 202, a model training module 203 and a model use module 204.

[0069] The sample acquisition module 201 is used to acquire a training sample set, wherein the training sample set includes multiple video sequences, and each video sequence consists of a pair of visible light images and thermal infrared images.

[0070] The model construction module 202 is used to construct an RGBT target tracking model based on frequency-space enhancement and time adaptation. The RGBT target tracking model consists of a frequency-space enhancement network, a cross-modal feature extraction network, and an online score prediction network. The frequency-space enhancement network consists of a visible light embedding branch and a thermal infrared embedding branch.

[0071] The model training module 203 is used to iteratively train the RGBT target tracking model based on the training sample set until convergence to obtain a trained model, wherein the input of the RGBT target tracking model is specifically the search area and template area of the visible light image and the thermal infrared image. The frequency-space enhancement network uses the frequency attention mechanism and the spatial enhancement mechanism to perform more discriminative feature extraction on the visible light image and the thermal infrared image to obtain the embedding vector of the visible light modality and the embedding vector of the thermal infrared modality. The cross-modal feature extraction network connects the embedding vectors of the two modalities to realize feature fusion and interaction between the two modalities to obtain fused features, and generates the bounding box of the target based on the fused features. The online score prediction network calculates the confidence score of the target and uses the confidence score to dynamically update the template area to adapt to changes in the target.

[0072] The model using module 204 is used to input the target video into the trained model to obtain the tracking result of the target video output by the trained model.

[0073] It is not difficult to find that this embodiment is a system embodiment corresponding to the above-mentioned method embodiment. This embodiment can be implemented in conjunction with the above-mentioned method embodiment. The relevant technical details and technical effects mentioned in the above-mentioned method embodiment are still valid in this embodiment. In order to reduce repetition, they will not be repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above-mentioned method embodiment.

[0074] It is worth mentioning that all modules involved in this embodiment are logical modules. In actual applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, to highlight the innovation of this application, this embodiment does not include units that are not closely related to solving the technical problem proposed by this application. However, this does not mean that other units do not exist in this embodiment.

[0075] Another embodiment of the present application provides an electronic device, the specific structure of which is as follows: Figure 7 As shown, it includes: at least one processor 301; and a memory 302 communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions that can be executed by the at least one processor 301, and the instructions are executed by the at least one processor 301 to enable the at least one processor 301 to perform an RGBT target tracking method based on frequency space enhancement and time adaptation as described in the above method embodiment.

[0076] The memory and processor are connected using a bus, which includes any number of interconnected buses and bridges. The bus connects various circuits of one or more processors and memories. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and therefore will not be described further in this article. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. Data processed by the processor is transmitted on a wireless medium via an antenna. Furthermore, the antenna also receives data and transmits it to the processor.

[0077] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0078] Another embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement an RGBT target tracking method based on frequency-space enhancement and time adaptation as described in the above method embodiment.

[0079] That is, those skilled in the art will understand that all or part of the steps in the above-described embodiments can be implemented by instructing the relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (such as a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present application. Storage media include: U disk, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk, etc., various media that can store program code.

[0080] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and that in actual applications, various modifications may be made to the embodiments in form and detail without departing from the spirit and scope of the present application. Those skilled in the art will appreciate that improvements and modifications may be made without departing from the principles of the present application, and such improvements and modifications are also considered to be within the scope of protection of the present application.

Claims

1. A RGBT target tracking method based on frequency space enhancement and time adaptation, characterized in that: The method comprises: Obtain a training sample set; wherein the training sample set includes multiple video sequences, each video sequence consisting of a pair of visible light images and thermal infrared images; Build an RGBT target tracking model based on frequency-space enhancement and temporal adaptation. The RGBT target tracking model consists of a frequency-space enhancement network, a cross-modal feature extraction network, and an online score prediction network. The frequency-space enhancement network consists of a visible light embedding branch and a thermal infrared embedding branch. The RGBT target tracking model is iteratively trained to convergence based on the training sample set to obtain a trained model. The input of the RGBT target tracking model is specifically the search area and template area of the visible light image and thermal infrared image. The frequency-space enhancement network uses the frequency attention mechanism and spatial enhancement mechanism to extract more discriminative features from the visible light image and thermal infrared image, obtaining the embedding vector of the visible light modality and the embedding vector of the thermal infrared modality. The cross-modal feature extraction network connects the embedding vectors of the two modalities, realizes the feature fusion and interaction between the two modalities to obtain the fused features, and generates the bounding box of the target based on the fused features. The online score prediction network calculates the confidence score of the target and uses the confidence score to dynamically update the template area to adapt to changes in the target. The target video is input into the trained model to obtain the tracking result of the target video output by the trained model.

2. The RGBT target tracking method based on frequency space enhancement and time adaptation according to claim 1 is characterized in that: The search area and template area of the visible light image are input into the visible light embedding branch, and the search area and template area of the thermal infrared image are input into the thermal infrared embedding branch. Both the visible light embedding branch and the thermal infrared embedding branch contain three embedding layers, each of which is connected to a frequency-space enhancement module. Each embedding layer is used to extract features from its own input search area and template area respectively to obtain search features and template features; Each frequency-space enhancement module consists of two parts: a frequency attention unit and a spatial enhancement unit, and the frequency attention units share weights; The frequency attention unit uses multiple discrete cosine transform frequency components as channel coefficients, obtains the frequency attention units of the search feature and template feature and combines them on average, creates a frequency attention mask to improve the robustness of the frequency attention, multiplies the frequency attention mask with the search feature and template feature, and generates an integrated frequency attention interaction between the search feature and the template feature, thereby obtaining the frequency attention interaction feature; For each pixel point of the frequency attention interaction feature, the spatial enhancement unit only models the connection between it and other pixels in the same row or column, thereby enhancing the spatial information of the frequency attention interaction feature. Finally, it is combined with the search feature and template feature through residual connection to finally output the search enhancement feature and template enhancement feature.

3. The RGBT target tracking method based on frequency space enhancement and time adaptation according to claim 2 is characterized in that: The input of the cross-modal feature extraction network is the embedding vector of the visible light modality output by the visible light embedding branch, and the embedding vector of the thermal infrared modality output by the thermal infrared embedding branch. The cross-modal feature extraction network consists of several ViT encoder layers connected sequentially; The cross-modal feature extraction network improves the single-layer fully connected embedding of the original ViT encoder layer into a multi-layer convolutional embedding based on a multi-scale pyramid structure and local inductive bias. The cross-modal feature extraction network uses three layers of convolution in the embedding layer and applies a local inductive bias similar to spatial attention after each convolution layer to model the correlation between local pixels, and finally outputs the bounding box of the target.

4. The RGBT target tracking method based on frequency space enhancement and time adaptation according to claim 3 is characterized in that: The online score prediction network consists of two Transformer decoders and a three-layer perceptron. The two Transformer decoders are a search token decoder and a template token decoder connected sequentially. The online score prediction network first initializes a learnable score marker and uses the score marker together with the search area marker as the query, key and value input of the search token decoder. The target information is extracted through the score marker encoding. Then, the score marker and the template marker are used as the input of the template token decoder. The extracted target is implicitly compared with the previous online template target. Finally, the confidence score of the target is calculated through a three-layer perceptron and a Sigmoid activation function layer. If the calculated confidence score is greater than the current online template score, the template area is dynamically updated based on the preset template update interval to adapt to the change of the target.

5. The RGBT target tracking method based on frequency space enhancement and time adaptation according to claim 4 is characterized in that: The currently used online template score decays as RGBT target tracking progresses. The decay formula of the online template score is: in, represents the online template score used when tracking the t-1 frame video, d is the preset attenuation coefficient, Represents the online template score used when tracking the t-th frame of the video.

6. The RGBT target tracking method based on frequency space enhancement and time adaptation according to any one of claims 1 to 5, characterized in that: The iterative training of the RGBT object tracking model is divided into two stages; In the first stage, the network parameters of the online score prediction network are fixed, and the frequency-space enhancement network and the cross-modal feature extraction network are trained. The first stage training uses adaptively weighted Focal classification loss, GIoU loss, and L1 regression loss for constraints, so that the RGBT target tracking model can focus on generating bounding boxes with higher confidence and more accurate targets; In the second stage, the network parameters of the frequency-space enhancement network and the cross-modal feature extraction network are fixed, and only the NCE loss is used to train the online score prediction network to ensure that the RGBT object tracking model can dynamically select a reliable online template.

7. The RGBT target tracking method based on frequency space enhancement and time adaptation according to claim 6, characterized in that: The loss function of the first stage training is expressed as: L stage1 =αL Focal +βL Giou +γL L1 ; Among them, L Focal 、L Giou and L L1 They represent adaptive weighted Focal classification loss, GIoU loss, and L1 regression loss, respectively. α, β, and γ are all preset adaptive weighting coefficients. L stage1 Represents the loss function of the first stage training; The loss function of the second stage training is expressed as follows: Among them, n represents the total number of samples in the second stage training, y i represents the label of the i-th sample, p i represents the predicted probability, L stage2 Represents the loss function of the second stage training.

8. An RGBT target tracking system based on frequency space enhancement and time adaptation, characterized in that: The system comprises: A sample acquisition module is used to acquire a training sample set, wherein the training sample set includes multiple video sequences, each of which is composed of a pair of visible light images and thermal infrared images; The model building module is used to build an RGBT target tracking model based on frequency-space enhancement and temporal adaptation. The RGBT target tracking model consists of a frequency-space enhancement network, a cross-modal feature extraction network, and an online score prediction network. The frequency-space enhancement network consists of a visible light embedding branch and a thermal infrared embedding branch. The model training module is used to iteratively train the RGBT target tracking model based on the training sample set until convergence, thereby obtaining a trained model. The input of the RGBT target tracking model is specifically the search area and template area of the visible light image and the thermal infrared image. The frequency-space enhancement network uses the frequency attention mechanism and the spatial enhancement mechanism to extract more discriminative features from the visible light image and the thermal infrared image, obtaining the embedding vector of the visible light modality and the embedding vector of the thermal infrared modality. The cross-modal feature extraction network connects the embedding vectors of the two modalities, realizes the feature fusion and interaction between the two modalities to obtain the fused features, and generates the bounding box of the target based on the fused features. The online score prediction network calculates the confidence score of the target and uses the confidence score to dynamically update the template area to adapt to the changes of the target. The model usage module is used to input the target video into the trained model and obtain the tracking result of the target video output by the trained model.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the RGBT target tracking method based on frequency space enhancement and time adaptation according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it can implement the RGBT target tracking method based on frequency space enhancement and time adaptation according to any one of claims 1 to 7.

Images